Week 1 at Clemson University was quite interesting to say the least. I arrived on Saturday May 18th to an empty apartment with all my things. After spending the weekend generally exploring, I headed to McAdams Hall early Monday morning. After meeting some of the administrative staff and getting a keycard to the labs, I met Dr. Joerg and Dr. Daily. Both of my Faculty Mentors seemed excited to have me start working for the summer and explained my role in the DIVE project.
I also had the opportunity to see how motion capture works as Dr. Joerg walked me through the hardware set up and also the general software procedure. Coming in with no experience and only a general knowledge of how motion capture worked, I was quite eager to learn about the system. Dr. Joerg is very knowledgable about the process and I slowly began to realize how MoCap would fit into the DIVE project.
Later on in the Week, I continued to learn about MoCap, DIVE, and a second project on Perception of Virtual Humanlike Characters in Video Games. Dr. Joerg gave me a brief lesson on the Uncanny Valley which is a theory that was once applied to things like robots about how people view humanlike objects. With this short lesson and with completion of the CITI programs ethics online courses, I learned that I will be working with participants to study the their perceptions of video game characters.
In addition, I had my first weekly meeting with Dr. Daily and her graduate and doctorate students. The meeting was quite intimidating at first because of the amount of projects and deadlines that had to be met. However, it was interesting to hear about the individual and team projects each of the 5 students would be working on. Even though I am a DREU student without as much experience as a PhD or Masters student, I plan on putting as much work as they do into my own parts of the DIVE project. Dr. Daily and her students expect the same out of me.
Around midweek things really picked up as I had my first test capture session. Dr. Joerg and I calibrated the MoCap system and then I was fit into a MoCap suit. After fitting, we began capturing or "filming" me completing Range of Motion activities. My main job for the rest of the week was to post-process the data, which was a tedious task but fulfilling nonetheless. After a day of frustration I finished the post-processing of my first capture session on Friday morning.
The afternoon of Friday was very exciting as I had my first capture session with people other than Dr. Joerg and myself. We were capturing some motions for another professor who needed to analyze data and compare it with accelerometer readings. This was very exciting for me because I was able to apply some of the things I learned throughout the weeks in front of people who did not know very much about MoCap. I tried my best to answer any of their questions and it helped my own understanding of how the process works.
Overall, Week 1 began and ended well. I see that I'll be working very hard this summer getting used to the MoCap system, applying it to the DIVE project, and also working on the Perception project. The coming weeks will most likely be als fulfilling as the first.
This week has been an extremely busy one. To begin with, I was tasked with posting research participant recruitment flyers to for another project of Dr. Joerg's. This gave me the opportunity to explore some of the many academic and student buildings on Clemson's campus. Also, Dr. Joerg had me working in the Vicon Blade software to post-process data. The software itself was not the easiest thing to learn so it learning the ropes was quite frustrating at first when compared with the older software I was learning to use in the previous week. However, after a few days of working in Blade, with much trial and error, I was able to complete the motion post-processing procedure.
By learning the software for the MoCap system, I am hoping to become proficient enough in it so that I can run the system smoothly for the capture sessions we will have scheduled later in the summer for the DIVE project. Also, in the process of learning the software through trial and error, I am able to have a better understanding of what works vs what does not. This way, I can update the DPA wiki MoCap Pipeline that acts as sort of a step-by-step tutorial for the system we have in the lab.
While working on updating the Wiki with my recent findings on how the software works, I thought it best to put as much information and detail as possible about my procedures. The other updates to the wiki page seemed to be sort of scarce and lacking detail. I felt that I could save someone in the future lots of frustration and time if I added some of the minute details that were quite hard to get used to in Blade during my experiences. I also added edited screenshots as how-tos for my procedures to act as visual aid for my updates to the wiki. I will continue to add anything to the MoCap wiki page as I further my work with the system.
On Thursday of Week 2, Dr. Joerg and I held an extensive capture session for the same small group of researchers from Week 1. They had to run quite a few more participants. It was interesting to see how Motion Capture can be applied in different areas of research because of the useful data such as X, Y, Z coordinates of any specific marker attached to the body. Often times, if people know anything about MoCap they may think it is mostly only useful for animation and video games but Dr. Joerg showed me that it can be useful in other areas as well. The capture sessions went a bit long into the evening hours but I did not complain and stuck through it because that is what is expected of me not only by faculty mentors but of course by the DREU as well.
On Friday, I attended three meetings of different variety. In the morning hours, I attended a meeting to meet many of the other DREU students, REU students, PhD and Masters students and a few faculty members as well just to discuss what is expected of us this summer with regards to dress, hours, and behavior. The meeting was not very strict and lecture based but more of a nice conversation environment. In the afternoon, Dr. Joerg, Dr. Daily, Kara (a graduate student) and I met to discuss our goals for the summer as a group in the DIVE project and I learned that we have a lot to do in a short amount of time. With Dr. Daily and Dr. Joerg both traveling, a lot of the work is on Kara and I to plan a lot of the interfaces and specifics for the DIVE project which is both exciting and quite scary. The work Kara and I do can really help the project in the future after I am done with my summer DREU at Clemson. The last meeting I had was with Dr. Joerg and another one of her students to discuss the research she is doing outside of DIVE with perception of virtual characters. While she is traveling, we will be running the experiment and recording the data for her to analyze. This is pretty exciting as I have not had much experience with research experiments and the like. I'm looking forward to helping out in this research project as well as DIVE.
Week 3 has been the week to set the work flow for the rest of the summer. While in the first two weeks I was tasked with learning the MoCap system, Week 3 I began learning Unity with the goal of being able to create an interface that can be used in the future and throughout the summer. The Unity work I am doing is very practical and a great experience because it allows me to brush up on my scripting skills. Also, in week 3 the meetings I have had with Kara (my DIVE project partner) and Dr. Daily's team are great for planning for the rest of the summer. Both of these meetings occur every week and really help to shape the work I'm doing on a week by week basis in order to meet the summer goal of having programmable dances in Unity.
Learning Unity so far has been both fulfilling and a bit frustrating. There is tons of support online through the Unity documentation as well as the forums. After viewing a few tutorials, I began to refer to some of my own UI mockups to think about what I can start creating to have to show at weekly meetings. Then, after researching the forums and documentation, I began to trial and error my way through some of the Unity GUI controls to create a very basic prototype UI to work on for the summer. After I got GUI that was roughly similar to my mockup established in Unity I began to think about some of the controls a user would need for the on-screen dance character. In a morning, I create a few simple buttons and key commands that allow the user to change the scale and rotation of a 3D object. So far, this seems to be enough to show at a meeting and so I will go back and view some more tutorials on animations.
On another note, I have also been trying to find participants for the Perception of Video Games study. Even with all of my postings on campus, I have not received many emails, but I have been asking people around the office if they would like to participate. It is pretty interesting to see some of the peoples reactions to the study once they are done participating.
Week 4 of my DREU experience has been as fulfilling as the rest. At the beginning of Week 4, I the study from Dr. Joerg on two participants. Kara and I also met with Dr. Alison Leonard, a professor working with the Research on the DIVE project with research interests in education through the arts. Dr. Leonard told us what dance movements would be best to capture from the Motion Capture end of the project, and then helped us define the goal of what dance movements we should aim to be able to program by the end of the summer. This helped define a more concrete goal for the demo. Then toward the middle of the week, I had a group meeting via Skype with Dr. Daily and the graduate students her lab. Each of us gave updates and used screenshare to give demos of our progress. Even though I had a bit of functionality to show for my Unity progress, I still did not have a clear direction with my progress in the DIVE project. However, later in the week I met with Dr. Babu, a professor with a lot of research in virtual environments. His lab works heavily with creating 3D immersive environments using the Unity software. Dr. Babu helped me think of step by step timeline and framework that can guide me in Unity in order to have a working demo by the end of the summer.
The timeline that Dr. Babu helped me outline really shined light on my summer. With the steps that he gave me, I can have a demo and a poster by the end of the summer, that I can take to other conferences as well. This is a really exciting prospect because it will be a great accomplishment to have something to show for the summer. I can maybe even give a small talk or demo at my school about the amazing work that Dr. Daily is doing in her lab to further computing education. For the last five weeks of my summer DREU experience, I will creating a 3D dance environment, creating an animation databank for dance movements in Unity, and mapping those animations to both keystrokes and UI. Then, I will have to create a simple scripting interface that will allow a user to use simplistic coding language with little syntax restrictions to control the dances. This will be a huge challenge but I will learn a lot through it and that will make me a much better programmer.
I spent all of Week 5 working in Unity to manipulate animations. My primary goal for week 5 was to import a rig with Motion Capture data and an Iron Man mesh into Unity and have him dance on screen. The Iron Man mesh was created by Kara for a different project and we had decided to import it into our project just for development purposes and will get a different mesh later on for the demo. By mid week, I had accomplished that goal. The dances would loop on screen and I could mix some smaller animations into the clip. However, I still needed to break this process down a bit more. Therefore, close to the end of the week 5 I had been looking into the older animation process of Unity, Legacy. This method was completely based on playing animation using code. I wanted to have Iron Man settle into an idle state when he was not dancing and then go into various animations from there. My thought process behind this was that if there was still some animation to the avatar while he was still, he would seem more natural.
The biggest problem I encountered with this was that Iron Man did not have an idle state animation clip created and packaged in his fbx file. The fbx file's animations were done using muscle definition from the Maya program. The animations he had were done using motion capture. However, the idle animation I created for him was done using the classic Keyframe animation technique. When I tried to blend these together, the result was not desired. IronMan would go into idle from wherever he stopped the previous animation. His body parts would remain where he ended the previous animation yet his hands and head would still sway as dictated by the idle animation keyframes. I decided to scrap the keyframe idle position in hopes that we will record them in the future using MoCap, then we can animate from that position on our future models.
Week 6 has been quite interesting thus far. This week I have realized that I am nearing the end of the summer and only have roughly 3 weeks left here at Clemson. It has gotten me to not only think about my project here but also coming back here to work as a Ph.D student. I really like the work that Dr. Daily is doing in Human-Centered Computing and I feel as if I would like to do something similar. I did not think about graduate school at all before I came to Clemson to participate in the DREU. However, by meeting the other graduate students in HCC, I see how beneficial the program is not only to the grads but also to the people the research can reach, including dementia patients, stroke victims, and children. Dr. Daily is going to help prepare me to apply to the program which I am really grateful for.
As for the DIVE project, I have been working with programming the dance animations using keyboard input. I have taken each clip and mapped it to a key. Also, I have taken multiple clips and attached them to one key that plays the dance. I have had great success with this, but my next step is to create a simple GUI that will allow me to control the clips on screen and create a dance out of them. Hypothetically, the user should be able to program a dance without having to code anything themselves. Next week I hope to have something to show for this step.
During week 7, my primary focus has been to build a simple GUI for the DIVE interface that demonstrates the ability to be able to control dance motions via onscreen controls. This functionality was particularly challenging because I felt the need to build the system with the thought that it could be expanded on again later. Therefore, it took me a few days to figure out exactly how I wanted to do this. In the end, I accomplished my goal and also learned a great deal about unity and animation queues.
In my code, I can add queued animations to an array and then return the values of the array to make the animations play. This is particularly helpful because I can programmatically keep track of what animations are in the queue. So far, the most problematic aspect of this functionality is that even though the animations are in the queue array, they also become part of a system queue in Unity memory. While I can easily clear my array, I have yet to find a way to clear the system memory queue. I hope to figure this out in Week 8.
Week 8 started slowly but became both exciting and hectic toward the end of the week. In the beginning, I struggled with trying to clean up the GUI for the DIVE interface. Dr. Daily and the team are hoping that I can get a clean polished design for the project and so I spent a day working through some mockups and searching for tools to help me do so. I finally came up with something by Wednesday, but I had not had much time to work on it as I was also helping Dr. Joerg run the Digital Games and Perception study. Then, on Thursday, Dr. Joerg, Kara, and I got the MoCap system fully functional in a different lab. This was exciting because later that day we had the big DIVE capture session planned.
During the capture session, we focused on capturing the Cha Cha Slide dance, Idle positions, and Ballet positions 1 - 5. The capture data was pretty great because we had all of the cameras set up in the new location versus 8 in the previous lab. Our dancer and model, Brooke, had a lot of fun dancing for the capture session and seemed very comfortable and natural with her performance. Then, we had some time left at the end of the session so we had her do some other motions so that we could have lots of data for future use. While the goal for this summer is to have the Cha Cha Slide programmable in DIVE, having more miscellaneous capture data is beneficial to the project just in case we want to add more motions to DIVE and play around with them to test limitations. Week 9 is going to be extremely busy between post-processing the data and loading it up to a new rig and throwing it all in Unity. Also I need to continue to polish the UI and add more functionality to make programming the dances more realistic.
Week 9 was spent mostly conducting the study for Dr. Joerg in the attempt to meet a deadline. However, we could not get enough participants for the study in time. After that, I began to put more energy into the final steps of the DIVE project. At this point, I needed to plan out how I wanted to wrap up the project based on the goals set up at the beginning of the summer. I still had to post-process the MoCap data and get it to Kara so she could rig it up and apply it to a character mesh. However, both Dr. Joerg and Kara were leaving to attend SIGGRAPH in California for about a week toward the end of week 9 and into week 10. Therefore, Kara and I could not communicate very well about the MoCap data. It took me a little over 6 hours to finally get clean data from the MoCap capture data and send it to Kara. Then, on her side she began to run into problems with applying it to a character.
Week 10 has been quite hectic. As my last week, I still had to finish my project, prepare a presentation, and finish up any other DREU requirements. By the beginning of the week, Kara still could not figure out what had been going wrong with the files I sent her, so I just continued to complete as much functionality for DIVE as possible. This functionality included character rotation, limb selection for dance clips, cleaning up the queue to make it more readable, and including a faux-scripting interface that outputs strings that represents a loop based on the number of iterations of a clip. There are few bugs that I can't seem to solve in a timely matter, but they are minor and the overall functionality works in DIVE. Toward the middle of the week I had planned to have the characters from Kara, however she could not get it to work and possibly did not have all the much time either while being at such a large conference such as SIGGRAPH. I decided it would be best to comment my code as best as possible so that future developers, including Kara, could find their way around. I plan on interfacing with Kara even after I leave Clemson to help her load the characters up into DIVE. I may even bring the files to my personal machine and do it myself.
As I round out the end of the week, I have time to reflect on my summer as a DREU student. I am extremely glad to have applied to and be accept as a DREU student. It has opened so many doors for me and led to me wanting to attend Graduate school here at Clemson University. I really enjoy the HCC department here, including both the faculty and students. Nearly everyone I have met has been genuinely nice and helped me in many ways. I have also learned that no matter your background, you should strive to make yourself valuable when working in a team. Even though I may not have the Computer Science background as many other students in the program, I used what I do know to program the DIVE functionality just by having confidence in myself. While I may have had a lot of resources to help me with the coding issues, I got through most of it on my own as a self-challenge to make myself better. This has been both a great learning experience and a incredibly enjoyable summer here at Clemson University as a DREU student.