distributed research experience for undergraduates

Katie Wolf - DREU Experience 2010

University of Minnesota - University of Southern California

week 10
week 9
week 8
week 7
week 6
week 5
week 4
week 3
week 2
week 1

weekly log

Here is where I will be keeping track of my summer research experience. The links to the left will bring you to the page for that week.

follow-up 8/17/10

After reading through the log a month later, it is very interesting to see how much I have learned (and the mistakes that I made) since that first week on the project. After attending the International Society for Music Information Retrieval Conference (ISMIR) it was quite amazing to see and meet some of the people that were making these new advances that I was reading all about in the papers. Even more, it was great to be able to understand a lot of what they were doing and also to see the other subareas of Music Information Retrieval. Here is the final version of the poster that I presented at the late breaking-demo session of the conference: Evaluation of Performance-to-Score MIDI Alignment of Piano Duets

A full review of my days at the conference will also be posted soon. From the whole experience of being introduced to the field and just starting by reading papers that were out there, to actually attending the conference and meeting all the people behind the works, it has been pretty amazing. I am excited to have found a direction that I would like to take in my graduate studies.

week 10: 7/19/10 - 7/25/10

I spent the last week working more on the adjustments of the annotations and analyzing some of the initial results from the highest alignment accuracies for each separate performance. I found that there are many things that affect the accuracy of the alignment for our experiments: the overall difficulty of the piece, the amount of added auditory delay, the order in which the performer's played the pieces (the players' learned knowledge of how to compensate for the coordination). For future work it would be good to have the players first play a practiced piece with a single delay several times to see how the accuracy is affected by the number of times a piece is played. In some of the initial results it was hard to tell whether an alignment did better because the amount of delay was less or if it was because it was after the player's had tried to play it with delay previously and had started to find a method for compensating for the delay.

Even though my stay in California is over, my work definitely is not! I have to prepare my late breaking-demo poster for the ISMIR conference, finish my final report, complete my findings on the project, and write at least one paper for submission to a conference next year. Hopefully I can get a lot of that done before the semester begins in September, but there are always other things going on as well. The semester is also keeping me busy with working on my honors thesis at the University of Minnesota, working on the graduate school applications and the application for the NSF Graduate Research Fellowship , getting ready for presenting at the Grace Hopper Celebration of Women in Computing Conference and my study abroad in the spring, and then the normal stress of classes (which should not be too bad this semester).

Weekend:

I spent the last weekend in California relaxing with my roommates and packing, and then made it safe and sound back to Minnesota and my new apartment.

week 9: 7/12/10 - 7/18/10

Continued to align the files with different parameters and run the error analysis on them. Also began formatting the graphs that will be used to display that data. With only one week after this one, I needed to start getting everything together for backup on the MuCoaCo hard drive. So I went through the matlab code that I had written and did the commenting that I should have done when I wrote it. I guess it is better to do it late than not at all.

After last summer, I tried to write the final report while I was taking classes during the semester, and it was very hard to find the time to get it finished up. So this summer, I plan to write the final report before school starts. First I will work at getting my presentation ready for the ISMIR conference, and then use that as an outline for the paper.

Weekend:

Continued to run some of the alignments over the weekend, while watching marathons of Criminal Minds, a TV show I seemed to have gotten my roommates hooked on. I also helped Tanvi, one of my roommates, move some of her stuff into her new apartment. It got me really excited to move into my new place when I get back to Minnesota.

week 8: 7/6/10 - 7/11/10

I was able to get the midi-to-midi annotations completed for all of parts, and tested out ways to convert from midi to wav so that I can test the wav-to-wav alignment with the same pieces as the midi-to-midi. The rest of the week was spent putting together everything I have done in the project so far into an outline so that I could compose a submission to the late breaking/demo session at the ISMIR conference coming up in August. I had already decided that I was going, and when the deadline was extended, Professor Chew and I decided that it would be worth it to write something.

One of the hardest parts that I have found about research is putting it all together at the end. Even though you have all of the knowledge of what you did, putting it into words in an organized fashion is always tricky. Once you have written something it is always easier to write more, but that first time is always the hardest.

The deadline for our submission was Friday, and we got it all submitted! Here is our proposal: Evaluation of Performance-to-Score MIDI Alignment of Piano Duets

Weekend:

My sister came out to California from Minnesota to visit me for the weekend. She has never been out here and never seen the ocean, so we planned a ton of things to do. Every second Thursday of the month there is an Art Walk that goes on in downtown LA. So, my sister being an art major, we decided to check that out. After eating at this awesome Ramen place, we walked around for a lot of the night checking out the various galleries that had stuff on display. On Friday, my sister walked around USC and took a ton of pictures of the area while I finished up the submission to the late breaking/demo session, and we went to a pretty good Indian place for dinner. Saturday was Universal Studios! We went on all the cool rides and to the interesting shows. I got to experience 3D movies for the first time, and also 4D (your seat moves and stuff gets squirted at you)! It was a pretty intense day, followed up by going to the new Twilight movie - Eclipse in IMAX, and Yogurtland! Sunday was all about the beach! We went to Santa Monica and rented bikes for the day. Venice Beach was pretty cool, we walked along and checked out all of the cool shops and definitely spent too much money! But that is okay, since this was really the only weekend where I really went all out and got to cross stuff off of my list of stuff I wanted to do while in California. I only have 2 more weeks left, so I better make the most of it!

week 7: 6/28/10 - 7/5/10

Studied the results of the final few movements with the annotation evaluation method. I also created graphs to represent the data so that when the pieces are evaluated graphs are automatically saved off to be used at a later time. Matlab graphs are always tricky to work with and I have really been learning a lot on how to do the plotting. After setting up all of the pieces to run the tests on (some of them needed to be trimmed and the prima and seconda tracks needed to be separated into their individual midi files), I spent the rest of the week running the different alignment programs on the DIP files (having different delays and the different methods they used) with the different parameters for the interval size (hop size) used for the Dynamic Time Warping (DTW) algorithm.

Since we need to run DTW algorithm with each of the different parameters for the hopsize, it takes up a lot of time and memory. Even though I have run the tests to get the alignment, I did not run the program to analyze the results and compute how well the aligned annotations were. That will be the next step. However, since I only have two weeks left (yikes!), I need to really push it in gear and focus on figuring out a way to evaluate the DIP project files where we are looking at the different perspectives of the delay and looking the collaboration between the prima and seconda parts.

The 11th ISMIR (International Society for Music Information Retrieval) Conference this year is being held in Utrecht, Netherlands from August 9th-13th, and after Professor Chew mentioned that they had opened up a student travel award, we decided that it would be a good conference to use the DREU travel stipend with. So over the week I began to plan my trip and figure out the arrangements for that. It will be my first time going abroad and I am very nervous and excited! Since it is a part of DREU, and it is coming up so very fast, I will keep a log of the things that I will be participating in and the places that I will be going here as well. I think one of the things I am most excited for is being able to sit in on sessions and meet the people that have written the papers I have been reading all summer. Going to the conference will help me connect with these people and figure out what I am interested in. Also, it will help me get a feel for what these conferences are like, so that when I (hopefully) submit something to the conference I will know what to expect.

Weekend:

I spent the fourth of July weekend flying up to San Francisco where my boyfriend is taking an intensive Ancient Greek course for 10 weeks. It worked out really well that we both ended up in California for the summer and it has been really nice to be in the same time zone at least. So I flew up on Thursday night and while Matt was in class on Friday I worked at his house and that night we cooked together. Saturday we went to golden gate park in San Fran and went geocaching (a very fun past time involving a GPS and (sort of like) a treasure hunt!). For the fourth of July we kept going back and forth on what we should do, and it ended up being really breezy so we ended up just doing some geocaching around Berkeley and then cooking together again. Overall it was a pretty amazing weekend and after flying back Tuesday morning, I was ready to get back to work and really focus for the next couple of weeks.

week 6: 6/21/10 - 6/27/10

This week I spent time on categorizing the notes into different classes and going through the errors that were not categorized to see where the annotations were having troubles. I found out that some of the notes in the midi files were incorrect, so I then had to go through the very time consuming process of making sure the notes in the midi were all correct. The final note categorization that we came up with was:

B1 = constant harmony notes- repeated rhythms of the same notes
B2 = boundary notes - first and last notes of 'sections' - separated by more than 1 measure
B3 = Ornamented Notes:

B3A = trills
B3B = grace notes (diff kinds)
B3C = Runs

B4 = Target Notes beginning and after Ornamented

B4A = After Trills
B4B = After Grace Notes
B4C = After Runs
B4D = Beginning of Runs

B5 = weak bass notes - only left hand playing

When we analyze the errors we analyze how well the annotation does while also excluding each one of these categories and then excluding all of them. This helps us get a better idea of where all of the errors are occurring.

Weekend:

On a whim my roommates woke me up at 4 in the morning on Saturday and asked if I wanted to take a road trip with them to Las Vegas. After a 3 to 4 hour drive we were there and spent the day walking and looking at all of the sites. I had never been to Vegas before and it was pretty cool to see. It made me really want to watch Ocean's 11. We had to head back Saturday night, so I didn't get to experience the night life, but overall it was pretty fun! Then Sunday I spent the day recovering from the heat that was Las Vegas and sleeping half of the day and cooking and baking the other half.

week 5: 6/14/10 - 6/20/10

I started the week creating the code to automatically calculate errors for the different annotation techniques. We decided to use a method similar to what Meron and Hirose use in their paper on automatic alignment of a musical score to performed music. After getting the annotation, we use the note number (frequency) of each of the annotated notes and an error measure that represents the window of time to search around the annotated notes for the real onset for that particular note. If it finds the same note number within that window, then the note is correct, and if it does not than it is an 'error'.

We decided to also bring in the categorization techniques of Sapp, M�ller, and Grosche used in their paper on beat detection. When we first started working with the annotation, we noticed that some types of notes introduced a lot of error, and we thought to remove them from the annotation. However, by creating categories for notes where errors tend to occur we are able to keep the notes in and still see how well annotation techniques work while excluding certain categories of notes.

After getting the automatic error calculation working with the initial note categorization (starting with only the prelude prima of Poulenc's Sonata for Four Hands), I reviewed the errors that were occurring that were not in any category to see if any other category needed to be created. I found that in some places the MIDI files were off, and there were some places where not just the grace note, but the note that comes after the grace note should also be considered ornamented. I also noticed that one of the categories that we had initially created, may not be needed. The rest of the week was spent going through the score MIDI files with the score beside me and making sure that the notes were correct.

Note: I am in the process of adding a section for papers that I have read very soon.

Weekend:

I had a really fun weekend starting with watching a movie with the roommates Friday night and then going to with my roommate and her family to Santa Monica to see the pier and beach Saturday morning. It was an incredibly windy day which made the air really cold so we didn't swim, but we did walk along the water.

Then we headed to Orange County to go to the Vietnamese Market so that my roommate could pick up food for her family who were going to celebrate Father's Day the next day by doing a picnic. I got to try all kinds of new food and it was all really delicious! And then we went to Pasadena for a live music festival which turned out to be us walking all around old town in search of live music finally ending at a live jazz place where we were all severely under dressed but got a table anyway and had a nice time listening to the music. Sunday was spent a lot like my typical weekend, sitting in the sun and reading my book.

week 4: 6/6/10 - 6/13/10

During the beginning of the week I worked on creating an adjustment algorithm that would take the onsets extracted directly from the midi with expression and use them as a guide to adjust the generated onsets taken from the Match program to create a more accurate alignment. There were several ways this could be done: taking the average time of the expression onsets within a time window around the generated onset, taking the first expression onset within a time window around the generated onset, taking the onset with the most velocity, or taking the mean of the onsets, etc. I tried a variety of different ways, and they all seemed to make a positive impact on the alignment. However, it was hard to judge to what extent it made a difference. By listening you really couldn't tell the difference between the way one method worked over another.

I then began researching different methods for calculating the errors in an alignment. I found several papers that will be of use to us on our project. Finally, I found a paper in which the researchers did midi-to-midi alignment (Meron and Hirose), so I spend the rest of the week trying to reproduce the methods that they used to do their alignment. Their method for calculating errors was also helpful in giving us one way that we will be able to measure how well the alignment works. Another paper (that used midi-to-audio alignment) gives classes to each of the beats (they use the word beats - but they are technically working with onsets) and use that as a way to associate the errors in the evaluation of the methods. The different categories they use are:

Non-event beats B1: Beats that do not coincide with any note events
Boundary beats B2: Beats of the first measure and last measure of the piece
Ornamented beats B3: Beats that coincide with ornaments such as trills, grace notes, or arpeggios
Weak bass beats B4: Beats where only the left hand is played
Constant harmony beats B5: Beats that correspond to consecutive repetitions of the same chord

For our project, we will do something similar to give an idea of where the errors occur relative to what is happening in the music.

For my website, I will create a page where I list the papers that I have been reading over the summer. I will also be working on getting my homepage up and running so that I can gather together all of the stuff that I have been working on outside of my summer research.

The rest of the week was spent comparing our midi-to-midi alignment with that of Meron and Hirose. I ran into a few problems in recreating their method, and I am not sure if it is what I am doing in the program, or if it is that the program does not work well on this piece of music. Also, for the midi-to-midi Match that I had first created, I did not give any weight to how the notes were held out, and only gave a weight to the onset itself. After figuring this out, I began to work with different ways to express the held out notes, but nothing seemed to work as well as only using the onsets.

Thursday Night: I went with Yawen, the other undergraduate that works in the lab, to an amazing Mexican Restaurant (I don't think I have ever really had authentic Mexican food before - it was delicious!!!). Then we went to this crazy zombie improvised musical called Zombeince. It was at Theater Asylum, a fairly small theater that had only around 30 seats, but it was pretty funny. Afterwards, we went to this pretty famous cake place (where the stars go) called Sweet Lady Jane and took our cake to go. I devoured the entire piece (a triple berry white cake with whipped frosting - amazing!) in one sitting. After I had one bite I just couldn't stop and eventually it was gone... I will have to go back!

Weekend: The weekend was really chill, with nothing really planned. After sleeping in, I spent most of the day walking around the USC campus and sitting in the sun and reading my book. I got to have long conversations with my friends and family and spend time doing my own thing. Saturday night I hung out with my new roommate (just moved in to make it 5 girls in our apartment!) making homemade veggie pizza, having a glass of wine, and watching Shutter Island, which was a kind of weird movie. Then on Sunday, more walking and reading in the sun. I got a pretty nice tan and read through half of the book that I just got. Didn't really get anything done that I needed to, so my goal for this week is to be productive outside of doing the research in the lab. Starting with working on my homepage with getting all of my research stuff posted and a final version of my CV. Then onto making a list of graduate school fellowships to apply to with due dates and requirements for each. If I can get those things done this week, it will be a huge accomplishment!

week 3: 6/1/10 - 6/6/10

I started out the week finishing up applying the techniques that we had come up with to the Rustique and the Final parts of the Poulenc piece, and I came up with a scheme for which methods worked for which parts (though some of the pieces still did not have a great method). I also tried altering different parameters and timing in the pieces to see how the annotation outcomes would result. Nothing seemed to make a huge difference. Professor Chew gave me the task of researching to see if any work had been done with MIDI to Score alignment, and I found some interesting papers, but none that were what we were really looking for.

I spent the rest of the week tweaking Brain's MATCH program to work at aligning two midi files (rather than the wav which we had to first synthesize the midi in order to get the wav and do the processing). It took most of Thursday, but once it was done it seemed to work really well, and gave us another technique to analyze in order to finalize our method.

One thing that came to our attention was that the generated onsets from either of the MATCH programs (wav to wav and midi to midi) were offset from the position that they should be in. This is because the program snaps the annotation to the closest in a sampling of times taken every 20 ms. This could be trouble when we are looking to give a measure of the incorrect notes, since it is difficult to define the measure of error of what is an incorrect note. This will also be more important in the analysis of the collaboration later. So if we can reduce the sampling times so that the annotations are more exact by taking the samplings every 10 ms instead of 20 ms, we would be able to get a better method. However, the more detail we want the more space and memory we take up and every 10 ms was too many. It did work for 15 ms and the results were better, but with pieces of different lengths, I was unsure that it would work for the other pieces as well.

Professor Chew suggested that after the annotation is found with either the 20 ms or 15 ms sampling, we could look at the 20 ms of surrounding onsets taken from the exact onsets of the expressive piece (that were found with the MIDI toolbox), and make an adjustment so that the onset snaps to the first occurring note of that range or the average of the onsets in the range. Then we could decide which of these two methods work better. I began to work on code for the annotation adjustment on Friday.

Weekend: My roommates decided to drive up to San Francisco for the weekend and they offered to let me hitch a ride so I could see my boyfriend who just got to Berkeley Friday night for a summer session that starts on Monday. It was the most amazing weekend I have had in such a long time! We got into San Jose pretty late (around 10:00pm) and after some schedule changes to our itinerary and a mix up of when the CalTrain arrived at what station, I made it on my way to meet him at the airport, and we headed on our way up to Berkeley using the BART. I am very jealous of my boyfriend who gets to spend the summer in an area with such a nice public transportation system. Even though he probably won't use it much (he will be so busy with his accelerated Ancient Greek program - he is doing a year's worth of the language in 10 weeks!).

We spent Saturday walking around Berkeley and getting him his books and other things that he needed around his new place. Before I made it up there on Friday I had found out that the Minnesota Twins were playing the Oakland Athletics (One of our favorite things to do in MN is go to Twins games). So I had bought tickets for the game on Saturday night which was a ton of fun since it was a good game and the Twins won!

Then on Sunday we spent all day exploring San Francisco and the waterfront area. We had such an amazing time! We went Geocaching around the city and while we only found 6 caches we had a pretty awesome time and got a nice tan.

Going back to LA was pretty hard, but I am really surprised by how fast the summer is going. I have a few more things on my list of what I want to do before the summer is over. Here are some of them:

go to the beach (any beach! on a really nice day)
maybe take a trip to Las Vegas
take GRE (this is more of a personal goal rather than one I am looking forward to - but I better schedule it soon so I start studying!)
read more books (so far I have made it through 3 - which is pretty good since it has only been 3 weeks - a book a week isn't too bad of a pace)
finish my crochet blanket (I started it before I left MN and it is already almost done - I might even start another one!)

week 2: 5/24/10 - 5/31/10

Brian Highfill, one of the students that took a course this last spring with Professor Chew, worked on a project that aligned wav files by recreating the MATCH algorithm in matlab. Chandra and I spent Monday working on a way to convert the beats from one recording to another recording of the same piece using Brain's MATCH matlab code. Using dynamic programming his program took in two wav files and extracted their features and found a min-cost path through the cost matrix of the alignment of the two files. We added to Brian's code so that the program could take in an annotation text file that contained the beat annotations and would output the annotations for the target recording. We used two audio recordings of guitar instead of piano, so we had a few more things to test to see if the computed annotations are even good enough to only have to make a few corrections in order to make it work. Throughout the week I worked at testing the program to align MIDI piano files (here I looked at synthesized midi files that were first converting to WAV since we could not find an alignment program for two MIDI files) and it worked fairly well, so we were ready to apply the algorithm to the recordings that we intended to analyze for the project and begin to finalize our method.

The recordings for the DIP project that we are analyzing are of Poulenc's Sonata for Four Hands performed by the Tosheff Piano Duo. An initial tempo analyzation of these recordings were calculated in [http://smcnetwork.org/node/1052], and our goal is to be able to calculate how well the two performers in the duo play together when different latencies are added to the performance.

Once we had the recordings of the duo, we started to test the annotation software on them. First an annotation of the expressionless midi file (sometimes called the score) was needed so that we can have a very accurate annotation of the beats. Since midi's have discrete notes, with an expressionless version the manual annotations (going through by hand and labeling the beats) from the midi files will be more accurate than trying to annotate the synthesized version. Once we have an annotation of the emotionless piece we can then use the annotations and the midi file (synthesized in wav format) along with a wav file from one of the pieces to use the match program and get the annotations for the expressive piece.

After starting to test this auto annotation matlab code on the Poulenc piece, we started to have trouble in the silent parts of the piece. This is because with no tonal information, the algorithm in MATCH does not have any information on where to put the beat. Professor Chew suggested not annotating those parts on the manual version. We had this same issue with Sustained notes (notes that are held out) so we decided to obtain only the onsets of the notes. Professor Chew also suggested using a matlab toolbox that had been created by a group in Finland for getting midi information, including the note onsets . The site for this toolbox download is here. For both the expression-less piece and the expression-full piece the MIDI matlab toolbox worked well to extract the onset data. However since the expression-full piece dealt with real time notes, when a chord is played the notes do not hit exactly at the same time. In order to get that timing of the chords, an average of the onset times of the individual notes needs to be taken. This gets fairly difficult when the margin of time between when the notes in a chord are played may be similar to the time between sequential notes. Using the output annotations from the time warp algorithm from the expressionless piece to the expression-full piece, we are able to guide where the averages should be.

We also encountered a problem with trilling (rapid alternation in sequential notes). Since there is no coordination involved with trilling, even when two players are doing it simultaneously, it is easier to remove all but the first note of the trilling. This means for all of the recordings, I will have to go through by hand and remove the trilling before any further processing can be done.

By the end of the week, I got through matching the onsets from the score to the recordings for both the prima and seconda (the first and second part) of the Prelude. Next week, I will continue using those methods with the second movement, Rustique, and the third, Final. Once the method is finalized for each of the movements, I will begin analyzing the coordination between the prima and seconda parts in each of the different recordings with varying latencies.

Weekend: The weekend was a long weekend with Memorial Day on Monday, so I spent Saturday afternoon walking around campus and sitting out in the beautiful weather reading my book. Then on Sunday I had a really great time going with my roommate and her boyfriend back to her hometown in Palmdale to visit her family. Her parents had a bunch of relatives and friends over, and we celebrated birthdays and ate lots of very good food. It made me miss my family a bit, but from what I learned last summer doing this program, the 10 weeks will fly by.

week 1: 5/17/10 - 5/23/10

I spent my last week in Minnesota packing and waiting for my grades to come back from the semester. For housing in LA, I had found a place a while before through the Marketplace on Facebook in an apartment not far from campus for a pretty decent price. So everything was arranged for my stay in Cali. I really was not sure what to expect when I arrived.

Like usual, I ended up packing way too much, but my flights from MSP to LAX (layover in Denver) went smoothly, and I was able to get settled in. The first few days were rainy (apparently unusual for LA) and were actually colder than Minnesota. Monday, I got to meet the people I will be working with over the summer: my mentor: Professor Chew, another undergraduate student: Yawen, and a master's student who will be here for only a couple weeks: Chandra.

I spent the week getting familiar with the lab and tools I will be working with over the summer. One program that I started to work with is called Sonic Visualiser. It is an application for viewing and analyzing the contents of music audio files. Another program which I started to play around with is called MATCH (Music Alignment Tool CHest) which deals with the alignment of audio files using the on-line time warping (OLTW) algorithm. I read over this paper on Match: MATCH: A Music Alignment Tool Chest, Simon Dixon and Gerhard Widmer, ISMIR 2005, 6th International Conference on Music Information Retrieval, 11-15 September 2005, London, England. It gave me an idea of what the program does and the algorithm involved.

By the end of the week, it was decided that I will be working the Distributed Immersive Performance (DIP) Project that explores remote and synchronous musical collaboration. Using the audio recordings that were taken previously, I will do additional expressive musical performance analysis.

updated Dcember 2010.