Paul Taele and Laura Barreto - Innovative Applications of Artificial Intelligence Conference

On Tuesday, January 27, two members of the Sketch Recognition Lab, directed by Dr. Tracy Hammond, PhD student Paul Taele and an undergraduate sophomore Laura Barreto presented their paper, “Maestoso: An Intelligent Educational Sketching Tool for Learning Music Theory” authored by Taele, Barreto, and Hammond, at the 2015 Innovative Applications of Artificial Intelligence Conference (IAAI) in Austin, Texas. Taele and Barreto also received travel support from the Association for the Advancement of Artificial Intelligence (AAAI). 

 (Above: Laura Barreto presenting Maestoso)

Maestoso is an artificial intelligence program that teaches students the fundamentals of writing music and music theory via composition and sketching as well as provided lessons. Along with quizzes and lesson practice, Maestoso also gives feedback and critiques on a student's performance when writing music via a tablet connected to a computer. Maestoso impressively recognizes students hand drawn solutions and automatically corrects them providing intuitive feedback, recognizing subtle differences.

The love of music is a commonality shared by people across the globe – including musicians who practice and compose and non-musicians who enjoy listening to music. It connects dancers to power-walkers, bands to businesses, performers to listeners, and everyone in between. Maestoso is a program that is beneficial to a vast and diverse audience, and is especially exciting for aspiring musicians and people who want to learn music basics. 

Although aspiring musicians can more clearly see the practicality of understanding and mastering music theory, non-musicians may not see the benefits that come from it. As written in Taele and Barreto’s paper, the advantages gained by musicians who have mastered music theory include more accurately writing and performing musical pieces and being able to more fully express their ideas of and from the craft. However, for non-musicians, mastery provides greater critical thinking, having greater appreciation for the music content they consume, and the ability to improve math skills through a fun and real-life example.

As it is with well written musical pieces, Paul Taele and Laura Barreto’s presentation consisted of a delightful mix of tempos and beats, measures and pitches.  Barreto conducted her introductory and motivational section with a calm professionalism intermingled with humor that captured the attention of the audience. Her transition into the remainder of her first part of the speech – which covered the advantages and disadvantages of conventional ways of teaching music fundamentals and theory as well as betting to introduce Maestoso – informed and entertained.

Traditionally, new students to music are trained and taught under an expert instructor, who in turn teaches and tests them face-to-face to gain understanding of the results of learning and then provides feedback. This tactic, like all in-class courses, becomes less advantageous when there is only one professor with a large class, or if students cannot effectively get in touch with the instructor outside of class. Both issues cause a lack of personalized time and teaching with students. Furthermore classic textbook and workbook exercises rely on memorization and reiteration with no interactive feedback.

In this technology age, many students turn to innovation in mobile and online learning sites to advance their knowledge of subjects and solve problems. In regard to music theory, although computer-instructors are efficient, they function under the assumption that the student already has a grasp on the concepts and already has years of music training. It is not geared at beginners or hobby-learners.

It was at this point in the presentation that Taele began his cavantina - his solo speech describing the algorithms and study results.

(Paul Taele [right])

He began with an opening bow to Barreto by saying, “This project wasn’t possible if it weren’t for the help of Laura. And she is a sophomore in college. It just shows how much of an age gap there is between the both of us, so I wanted to let everyone know that I am very impressed and she is probably one of the best mentees I have ever had.”

Taele then began his speech on the history and design of Maestoso. Musical notes and symbols were broken down into two categories – geometrically simple and geometrically complex – based on what was observed in background research on how students drew notes on regular written quizzes.

Taele and Barreto quickly recognized two challenges arising in the creation and design of Maestoso. First, they would need to use multiple technological recognizers that were created over a period of many years to accurately read music symbols. Music notation is diverse and complex for systems to recognize due to the multiple sections and parts that make up one drawing of a symbol. Second, they had to account for notes that were ‘scribble capable’ – that is, notes and symbols which need to have spaces filled in within the outline.

Through geometric recognition and image matching, Maestoso is able to track the sketches that students input into its system via a drawing tablet, read the musical notes and commands, and then give automatic feedback. Just like a human professor who gives written exams to a small student body or on an individual basis.

Taele also explained that teachers can easily use Maestoso to aid in instruction by creating their own quizzes and routines for students to complete.

Looking toward the future, Taele and Barreto hope to collaborate with music instructors at the secondary and primary school level and integrate Maestoso into a classroom setting. They also want to expand the system to cover more advanced lessons, for more advanced students.

With the presentation concluded, Taele and Barreto addressed questions from a curious and eager audience. As they discussed the present constraints, grammar guides that aid in recognition, and future goals of Maestoso, the audience’s interest toward such an amazing program grew to a crescendo. 

One audience member asked when he was planning to finish his PhD, to which Paul proudly replied in humorous defiance "Well, this is my eighth year, I think that means I am about halfway done", to which everyone exploded in roaring laughter.

With sights and hopes on the future, Taele and Barreto’s presentation reached its coda and the conductors gave their gratitude and, literal, humorous bows to the audience as they concluded.

(Laura participating in an artificial intelligience trivia contest later in the conference.)

When Dr. Hammond was asked about her perspective on the work and presentation, she stated: 

"Laura came to the Sketch Recognition Lab during the summer of 2014 as part of the NSF DREU project. At this time she was just finishing up her first year at Vassar College, and she had only taken a single course in programming, which was not a language commonly used in the lab. But Laura came very highly rated, so we took a chance, and we were thrilled. Laura picked her own project and ran with it it a breakneck speed, finding music experts, working with legacy Mechanix code to learn which algorithms would be appropriate for this new domain. I am so impressed with what she was able to accomplish at such an early stage in her career and look forward to her future accomplishment."

... And Paul? ...

Dr. Hammond smiles and says, "Paul is just amazing, absolutely amazing.... and I promise he will graduate in fewer than 16 years :-p." 

"And, I am really proud Paul used the subjunctive case correctly, that is not so common in this day and age :-)"

(TAMU folks got together at the AAAI conference for a photo.)