Inspiration
Erica, one of our group members, played the violin in middle and high school. However, after arriving at A&M, as time passed and she adjusted to college life, she became immersed in the daily grind of coursework, extracurriculars, and commuting between classes. Soon, Erica found herself getting rusty at the instrument that was her passion for over 4 years, and disheartened because she had no way of improving, she was forced to stop playing almost completely.
Such is a dilemma college students often face. The hustle of college, corroborated with barriers to re-entry into the music field such as lack of lessons and expensive online tools for learning music, not only bars individuals trying to learn music for the first time, but even those attempting to get back into their habits of days past. That's what inspired us to create an app to solve this problem - a way for people to easily get back into playing music, in a fun yet informative manner.
And this app also overcomes an inability to not only read but also compare your music to that of sheet music, so you never have to guess a sequence of musical notes again!
What it does
Through this application, the user is able to identify the sequence of notes on a sheet of sheet music, by uploading the sheet music and highlighting individual notes. Through this feature, whether if the user is a novice first entering into music or an experienced player brushing up on their skills, the user is able to easily identify the individual notes they need to play on their instrument.
Secondly, this application also allows the user to upload their own music played on their instrument, and, comparing it with the sheet music the user is using as reference and has uploaded, scores the user's performance. The scoring criteria is based on established musical benchmarks, as following:
- 50% Pitch Accuracy: Identifying which note to practice is crucial to understanding musical patterns
- 30% Coverage: Considers all notes as a whole, accounting for omissions of difficult notes marked against in pitch accuracy
- 20% Rhythm: Incorporates musical fundamentals such as tempo, leaving room for the user's own musical interpretation.
Thirdly, in order to make the application more interactive for the user, thus making it more interesting and attractive and increase musical practice, we are integrating musical detection in the form of a game. In this game, the score is based on how close the user is to the sheet music; in a sequence of challenges, the user attempts to play notes, either in normal tempo or in increasingly rapid succession.
Fourthly, music players oftentimes question whether their form or technique is proper, and online resources are far too disorganized and spread out. Thus, to fix this issue, we incorporated a gesture detection feature where, based on their instrument, the user can see their technique score when playing a certain song, with feedback to help them get better.
Finally, to allow the user to be able to experience an end-to-end learning music experience, they are able to use a dynamic tuner and metronome, allowing them to test their instrument and tempo prior to playing and improving their music.
How we built it
Our basic implementation took user audio and fit the recording to the user's uploaded sheet music. Then, a machine learning model trained on IMSLP publicly available music data scored the user's music recording, based on the scoring weights provided above, and provided the score and details regarding the score. More specifically, our model uses CREPE (convolutional representation for pitch estimation), a neural pitch detector, which does the following;
- Reads the MusicXML file and IDs the different notes on the sheet music.
- Estimates pitch and confidence and, using the model, groups multiple occurrences per second by pitch changes, identifying individual notes played by the user.
- Compares played to existing notes, scoring on the basis of various factors, and accounting for skipped or extra notes.
For ambiguous notes, CREPE uses surrounding notes to identify the played note. In cases of extreme ambiguity, the platform flags the note and excludes it from the general score.
We then incorporated this base machine learning model and scoring weights into a game format, where the user's attempts at playing different notes as the game presents them allows the user to, through interactive means, better understand how to play their instrument.
Finally, the sheet music comparison was extrapolated to analysis of the sheet music itself. Reading the sheet music using a note dictionary, it identifies the individual notes, alleviating the difficulty of learning sheet music for beginners venturing into playing an instrument.
Challenges we ran into
Getting the app to hear a guitar clearly: A plucked note has a sharp attack and a fading tail. Two plucks of the same string can have the same pitch but still be two distinct notes. We had to use attack information as well as pitch to tell them apart.
Keeping our place after a mistake: One skipped note could make every note after it look wrong. Allowing gaps in the alignment helped the app recover and recognize when the musician was back on track.
Making the game respect interpretation: A violinist might stretch the end of a phrase on purpose. Using nearby notes to estimate the local beat helped us distinguish that choice from an isolated late entrance.
Knowing when to withhold feedback: A quiet recording or an obscured fingertip gives the app limited evidence. We had to make “too little information” a valid result so that a tracking failure wouldn’t become criticism of the musician.
Accomplishments that we're proud of
One of our favorite tests involved two versions of the same four-measure melody. In one, we deliberately slowed the ending. In another, we replaced the first A in measure two with a B-flat. The clean and slowed synthetic takes both scored 100. The wrong-note take scored 96.54, and Accelerando located the substitution within seven milliseconds of its known onset. Those are results from a controlled demo, but they capture a distinction we cared about from the beginning: a player can take their time and still play the right notes.
We’re especially proud of what happens after the score appears. A 96.54 alone gives Erica very little to work with. Being able to open measure two, hear the B-flat, and try the entrance again gives her a reason to pick up the violin for one more take.
What we learned
We learned that a pitch estimate and a practice instruction are very different things. A model can identify a B-flat, but it takes the score, the surrounding notes, and a reliable match to explain why that B-flat matters. We also learned how much judgment goes into a single number. Our 50/30/20 weighting reflects what we chose to prioritize in this prototype. Comparing those choices with feedback from music teachers is an important next step.
What's next for Accelerando
First, we want to put Accelerando in front of musicians, starting with players like Erica who are coming back after time away. We want to see whether someone can find a flagged passage without our help, understand the feedback, and use it on a second take.
We also want to compare our grades with teachers’ assessments across more instruments, voices, and recording conditions. For webcam coaching, we’d like to develop teacher-guided exercises for specific techniques, such as returning to a violin fingering. And because many musicians already own the piece they want to revisit on paper, we’d love to support scanned sheet music alongside MusicXML. That would let someone start with the music already sitting on their stand.