Week 1: My Arrival
I arrived in Greenville, SC in the morning and managed to get a Lyft and make my way to Clemson University by around noon! I arrived at Calhoun Courts, the summer housing only to discover that I wasn't getting any linens. Off to a great start, considering the nearest place to buy linens is the big Walmart over 5 miles away.
I met with Dr. Jörg for the first time and was introduced to the other students working in the lab.
I found the nearest Panda Express and Chipotle, and was so happy since I was already missing home. (Clemson is a small town in a more rural area than I am used to). I had no linens so I slept on the plain mattress and used some clothes as a pillow. 'Twas cold, but not too unpleasant. The next day, Lorraine, a PhD student also working in my lab offered to take me to Walmart, and I was ecstatic. I bought the minimum things I needed to live: kitchen ware, some food, LINENS!, and a pillow.
The rest of the week went pretty smoothly. I started working on a script to gather data that I wanted from the Maya files in the data set Dr. Jörg wanted me to work with. I wrote a script in Python that collected the rotation of all the joints and the translation of the hips in each frame. I then put these numbers into matrices and saved them into files. After I collected all the data from Maya, I wrote a Python script that read from these files, created a large NumPy matrix, and from there split the data into "X" and "Y" matrices to get ready for machine learning.
The focus of this week was to create a pipeline in which I could plot data from machine learning algorithms. I started by using the whole X matrix to generate a K Means predicted plot and a ground truth plot using PyPlot from Math Plot Library. Since I used PCA to reduce 183 features (dimensions of the matrix) down to 2 dimensions so I could plot the data on a 2D graph, I was unsure of what the data meant. (And at this point I didn't really care since I was just generating the pipeline). I got a script running that could run K Means Clustering and generate prediction and ground truth plots using sklearn! *How exciting*
June 19-23, 2017
Week 2
I bought a bike over the weekend! This campus now doesn't seem so big and isolated...
Dr. Jörg was in Seattle for the first couple days, so I continued work as usual. On Monday I split my X data into training and testing sets in order to attempt actual machine learning. After attempting linear and lasso regressions (the former with terrible root mean squared error and the latter not even converging), I concluded that the data set was definitely not linear. So I began looking into more methods other than K Means Clustering to learn.
I attempted K Nearest Neighbors Classification, Isometric Mapping used to reduce features on nonlinear data sets (this didn't work as well as PCA), and K Means Clustering again. I got bad accuracy--lower than 50%--so I started thinking about feature selection. Dr. Jörg and I had discussed last week that the left arm, legs, and feet likely did not contribute much to the deciding of mich hand gesture to choose since those parts hardly moved. I removed these features from the data set and achieved slightly better results, but still pretty dismal.
Next, we will attempt to generate new features from these collected features and others such as velocity of the body parts and the distance of the hand to the body and the fingers to the hand for example. These features could help with the clustering, considering the gestures have a lot of the rotations and motions in common. This explains why K Means has a hard time clustering.
Since we want to use more general features that describe the whole gesture rather than each frame, we want more wholistic features. Dr. Jörg and I discussed many potential new features we could use such as velocity of the right hand, angle of the elbow at the gesture hold point, acceleration, maximum bounding box volume, and others. Next week I'll work on retrieving the new data from Maya and write the script that will calculate all of that.
June 26-30, 2017
Week 3
I met a few of the other REU students also studying here at Clemson this summer! We had a fun weekend together: we went to Y beach one day, hung out and played Cards Against Humanity later that night, went to go see Wonder Woman at the theater in Seneca, and had a 4th of July BBQ and had fun with sparklers and some fireworks.
I went back to work on the 5th, and worked on organizing my script that reads from my text files of data and creates a matrix that I can then use to do machine learning.
I'm postponing collecting the data again from Maya since I want to make sure I get everything I can in one go and don't have to do data collection again (it takes so long). Organizing my script that creates the data matrix requires a lot of thought so that I don't get lost in my code and so that my machine learning preprocessing can be efficient when I reach that point.
I eventually collected all the rotations and translations I needed and started working on my big script that would go through this data and create a matrix that I would use to do machine learning.
July 3-7, 2017
Week 4
This past weekened was pretty fun! I went to the Smoking Pig BBQ with some REU gals--the food was amazing! I got the chopped pork with sweet potato fries and baked beans, and I tried sweet tea for the first time. I was pretty impressed. I went to the Tiger Town Tavern that night for the first time, and it was packed and super exciting. On Saturday, we went cliff jumping at this beautiful beach on the lake. The sand was sparkling with gold–-muscovite we later learned.
This week I finished the script that computes all of the 18 features I would be using to do machine learning on the 64 clips I have of 8 different gesture types.
I then started doing machine learning on the data! I split the data into training and testing sets. I first tried linear regression and lasso, but got not great results. I tried k means clustering, but that didn't work either. I believe the data points are just so similar that they overlap a bit too much. I also tried K nearest neighbors, but that also didn't work since it's a similar idea to k means.
I then tried decision tree classification and got better results, and eventually I attempted random forest and got great results! It spit out an accuracy score of 81%, which is great. I then got a martix of feature importances. My next steps will be to work with those important features to see if we can achieve the same results with fewer features.
I was also given the task of leading the project to make the lab we work in more homey and welcoming. Researching mentality behind that shall be interesting next week.
July 10-14, 2017
Week 5
This weekend we went to Greenville on Saturday. We went shopping at the mall and explored main street. We had a really yummy dinner at Tupelo Honey Cafe!
At work, we visited some labs in the McAdams building since Sophie wants Erin and I to help her redecorate her lab. We reached a few conclusions about the decor and feeling of homey-ness after those visits: carpet makes labs feel more welcoming! Also, when there are personal items at desks like drawings and even memes, the lab just feels more lived in.
In regards to my machine learning research, I finally reached a conclusion with the data set I had been working with. I created a confusion matrix and outputted the classification report to assist in analysis. I analyzed the data more and discovered which features were most important. I then added a feature for the up and down motion of the shoulders since "shrug" was being wrongly classified every time. This helped the accuracy reach 87% when using all of the features. Also, using just our top 5 features, we achieved an accuracy of 81% when we did leave one out. The feature I added just then was one of our top features, likely since it helps classify "shrug" well.
Next week I will begin work on prediction rather than classification on a different dataset: conversation. Now I will be working with a long stream of body motions and I will not have categories to classify the motions into. I will need to generate an error metric that tells me how different the closest match is for a given data point.
July 17-21, 2017
Week 6
This weekend it rained and I needed a break so I didn't do much. I went out on Saturday with a couple of my friends though! We had a good time dancing. On Sunday, I finished reading Doctor Sleep by Stephen King, which was excellent, and I just took the day to chill.
I started and finished working on the code that segments the new data set into gestures that I can use to create features. I also finished compiling all of my findings about the lab tour data I collected. I crafted conclusions based on the data I collected and from outside sources.
I took the Friday off to drive to Nashville with some of my fellow REU friends here at Clemson.
July 24-27, 2017
Week 7
Nashville this past weekend was a lot of fun! We went to a couple museums, listened to lots of great music, and ate at a lot of vegan and gluten free spots. My heart, ears, and stomach were very happy (:
This week Sophie is at SIGGRAPH and I'm still working on my work as usual. Since I finished working on translating the segmentation code to python, this week I've been focused on choosing a threshold for the right hand's velocity that works for all of the files. I'm still having trouble with one of the files since it has different settings than the other 4. But I do have working code that does leave-one-out on the gestures and computes the cosine similarities! I now have to figure out how to compare the gestures' finger motions if they are similar gestures. I started out using PCA to compile each of the gesture's finger motion matrices into a one dimensional matrix, and then I did the cosine similarity of that. Next week I will use Euclidian distance instead and compare the results.
July 31-August 4, 2017
Week 8
This past weekend I said goodbye to my other REU friends and moved out of Calhoun Courts and into the new place I found for Erin and me. Katie, the one who is letting us crash is such a sweetheart. Erin and I went to dinner at Monterrey's Mexican on Sunday night where I had some great fish tacos. I've also been practicing a lot for the GRE - part of it is because I need to and the other part is I don't have a lot to do anyways.
This week I plan to continue working on the conversation dataset. I implemented Euclidian distance to use in comparing the gestures and their corresponding finger motions. I use the fingers' local rotations to compare the similarities since global translation doesn't tell us anything. I ended up choosing a threshold of 10 for the time being. It seems to work well enough for now. I generated plots comparing the gesture and finger distances. However, since Euclidean distance is not necessarily the best metric since the distances are squared and places great emphasis on the features with large dissimilarities, I'll begin looking at other metrics.
I normalized the gesture matrices since some of the features had such a larger range than the other features. It helped the graph look nicer. I also used only the 5 important features in the comparisons and found similar results to when using all of the features, so the machine learning I did before using Random Forest is looking to have made an impact.
Before, I was comparing the left out gesture to all other gestures and then picking the most similar gesture, then getting the corresponding finger motions. Rather than pick the most simialr gesture, I then made it so that I would get all of the comparisons, and out of the top 5 most similar, I would pick the gesture that had the most similar finger motions to the one left out. This would show that you could pick a finger motions from a specified range since finger motions are rather similar often. This proved to work: we found even more similar finger motions when given a range rahter than just choosing the most similar gesture.
I also started writing a paper and asking Sophie about potentially publishing the paper for this research in a conference, and she was rather optimistic about it! So I might be publishing a paper! (:
August 7-11, 2017
Week 9
This past weekend Erin left for home, so I'm the last one of the summer students. Blythe, Katie's friend (who lets me carpool with her to work every morning), offered to take me with her to Trader Joe's in Greenville! I was ecstatic. I miss TJ's so much! We ate lunch at Zoe's Kitchen, which was yummy but also overpriced for the food I received. The next day I also managed to sell both of our bikes to an interested party. The day I go home is getting closer and closer!
At the beginning of the week I fixed the bugs in my code from last week and successfully modelled a hand using pyplot and the fingers' world translations. After modeling it, I figured out how to animate it using the animation tools from matplotlib. I now can see the gestures that are being compared! So exciting! After I did this I got more motivated to make progress in my research - it's crazy how something seemingly simple can motivate so much!
I learned how to create a separate file for the animation code that my main script can call from since the animation code is so long. I then figured out how to animate two gestures simulateously so I can compare them! It's especially cool since they're rainbow by default. (: I then wrote an algorithm that scales two matrices to the same length so that I can effectively compare two gestures of different lengths.
August 14-18, 2017
Week 10
This past weekend was a long weekend due to the total eclipse on Monday! Campus was packed, but it was super fun being around a bunch of people all gawking at the sun/moon. In the morning I took Katie's younger cousins to the planetarium in the physics building - that was really cool. We saw what the eclipse would make our surroundings look like and the stars we could maybe see (we ended up seeing Venus during the eclipse) and we rode a "space rollercoaster" which was incredible disorienting but cool nevertheless. The totality was incredible. I'm so lucky to have been here in Clemson during this event! After the eclipse, we headed back to the apartment and then headed to Uptown Art to paint a moon! I had a lot of fun despite the massive headache I was sporting.
Back at work the next days I incorporated the scaling code I created into my main code. I then implemented Root Mean Square Error (RMSE) as another distance metric to use in comparing the gestures and finger motions. I achieved similar results to Euclidean distance, which is to be expected, so it was a success!
Next, I will work on animating select gestures so that we can visualize and understand the generated graphs and comparisons.
August 21-25, 2017