ExploreYourMind: Software for Harmonic Combination of Video and Music

Overview

For bloggers, video editors and composers, choice of music to video and video to music, which harmoniously combines and successfully share author's idea, is a complex and lengthy process. Because it is not possible to feel many of the detailed characteristics of video or audio yourself, to look at million of different combinations and to compare each of them individually. Therefore, I created software which uses modern methods of machine learning and data analysis, it analyzes video, obtains detailed characteristics (such as objects and actions on video, activity and peaks, etc.), and recommends necessary characteristics for a musical composition (such as genre, mood, key, instrumentalness, activity and peaks, etc.). Then system finds music in the database with such parameters and harmoniously combines it with a video. Also, software performs reverse function: when receiving music, it recommends the most appropriate video - existing or created by artificial intelligence on its own. Since this study was the first to perform this task, I had to assemble own dataset. Therefore, by combining neural networks of deep learning, I created system for detailed video and music analysis. System analyzed many of the most successful videos and movies and saved found characteristics of video and music into dataset. After training on this dataset, I compared models of machine learning with random generation of parameters. The last has 1346.17 mean absolute error on test dataset, and trained neural network - 8.74. It can be concluded that model has found patterns in the data and used them for harmonious combination of new data. This software will be real help for bloggers, video editors and composers, helping them quickly combine music to video and vice versa.

Awards (3)

• Fourth Award of $500 $500
• Innopolis University : Full tuition scholarships for the Bachelor program in Computer Science
• National Center Junior Academy of Sciences of Ukraine: UN Sustainable Development Goal Award $ 500.00 $500

Competition history

• ISEF 2019 Robotics and Intelligent Machines · Entry ROBO047

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.