Charlotte Lambert
Vassar College
DREU Experience 2019
DREU Experience 2019
ABOUT
My name is Charlotte Lambert and I'm a rising senior at Vassar College in Poughkeepsie, NY. I am pursuing a BA in Computer Science with minors in Italian and Mathematics. My expected graduation date is May 2020. I can be contacted at clambert@vassar.edu.
This website documents my DREU experience during the summer of 2019. Throughout the summer, I worked in the Computer Science department with mentor Julia Hockenmaier at her university, the University of Illinois Urbana-Champaign. Her main areas of interest are computational linguistics or natural language processing.
More information about Professor Hockenmaier can be found here.
PROJECT DESCRIPTION
Virtual World Context Encoding for Grounded Dialogue in Minecraft
Our project focused on developing a convolutional neural network (CNN) that can recognize 3D shapes to assist in creating two fully interactive agents to play a game of Minecraft. The CNN had three main applications within this project. First, it would help improve an Architect agent generate meaningful utterances to instruct a Builder agent to build a target structure in Minecraft. Second, the CNN will also be used to give the Builder agent information about what shapes they have built at a given point and thus generate reasonable utterances throughout the game. Finally, the CNN will be used in another model that predicts the next logical action to be taken by the Builder agent to complete the instructions generated by the Architect.
WEEKLY JOURNAL
Project updates
A journal of my project work with weekly entries in which I describe: my results, my findings, mu algorithms, my frustrations, etc.
|
WEEK 1
|
May 29 – May 31, 2019
|
For the first week, I focused on familiarizing myself with some important deep learning concepts involving neural networks. I also worked through tutorials and documentation to gain an understanding of important tools being utilized in the project, such as PyTorch. Then, I began to develop code to generate representations of three-dimensional shapes to be used in a convolutional neural network (CNN) intended to recognize shapes in a Minecraft environment.
|
WEEK 2
|
June 3 – June 7, 2019
|
During my second week, I continued to develop my code for generating three-dimensional shapes. I extended the code to generate several more shape configurations (including composite shapes) and wrote code that made it easy to generate a random sample of shapes. Developing these shapes was certainly challenging and called upon my ability to determine and visualize all the possible orientations of each shape in three dimensions.
Additionally, I spent this week collaborating to create the best, most reproducible code for running our CNN. I started to become more comfortable with the concepts I learned about in the first week by working with a concrete example initially written by Ariel Cordes, another DREU intern working with me on this project. By the end of week 2 we had developed a preliminary CNN to recognize the six shapes I developed.
Additionally, I spent this week collaborating to create the best, most reproducible code for running our CNN. I started to become more comfortable with the concepts I learned about in the first week by working with a concrete example initially written by Ariel Cordes, another DREU intern working with me on this project. By the end of week 2 we had developed a preliminary CNN to recognize the six shapes I developed.
|
|
|
|
WEEK 3
|
June 10 – June 14, 2019
|
My third week working on the project was focused on shifting our neural network from a multi-class classification to a multi-label problem. Initially we intended for our CNN to look at a shape and decide which single shape it was. This meant that we could not handle classifying composite shapes nor could we consider sub-shapes.
Once we had decided multi-label was a more logical method to accomplish our overall goal and had finished writing code to generate 10 distinct shapes, we had to define the different sub-shapes of each one and see how the network would perform. Unfortunately, this conceptual shift meant we had a lot of work left to do on the network to get it anywhere close to the accuracy level we reached when using fewer shapes in a multi-class classification problem. At the end of week 3, our network was performing with very low accuracy due to the increased complexity of our labeling system.
Once we had decided multi-label was a more logical method to accomplish our overall goal and had finished writing code to generate 10 distinct shapes, we had to define the different sub-shapes of each one and see how the network would perform. Unfortunately, this conceptual shift meant we had a lot of work left to do on the network to get it anywhere close to the accuracy level we reached when using fewer shapes in a multi-class classification problem. At the end of week 3, our network was performing with very low accuracy due to the increased complexity of our labeling system.
|
WEEK 4
|
June 17 – June 21, 2019
|
At the start of week 4, we were unfortunately still stuck on getting our network to correctly recognize simple shapes, but by the end we had achieved 94% accuracy on our test set.
Now that we had a strong foundation with the simple shapes, we moved on to adapting our model to recognize composite shapes as well. This task proved to be much more difficult, so we experimented with different network architectures and parameters. Still, by the end of week 4, we hadn't successfully been able to recognize composite shapes.
Now that we had a strong foundation with the simple shapes, we moved on to adapting our model to recognize composite shapes as well. This task proved to be much more difficult, so we experimented with different network architectures and parameters. Still, by the end of week 4, we hadn't successfully been able to recognize composite shapes.
|
WEEK 5
|
June 24 – June 28, 2019
|
There were two main focuses for my fifth week: beginning to understand how to model the Builder agent and achieving good accuracy on the task of recognizing composite shapes. This first focus posed the challenge of getting acclimated with unfamiliar code. After some time, I was able to generate the necessary data to study the utterances made by the human Builders that participated in the data collection. Additionally, since the Builder model had to be able to determine when and where to place blocks, I adapted the code to also record actions made by the the Builder players in the data collection. These two new sets of data served as the base for developing a builder model.
As for the second focus, we slowly worked up to being able to recognize two shapes within one world, however adding more shapes to the 3D space significantly reduced the complexity. This was beginning to reflect the same struggles we had trying to write a network to recognize just one shape.
As for the second focus, we slowly worked up to being able to recognize two shapes within one world, however adding more shapes to the 3D space significantly reduced the complexity. This was beginning to reflect the same struggles we had trying to write a network to recognize just one shape.
|
WEEK 6
|
July 1 – July 5, 2019
|
During my sixth week, we started to wind down on our shape recognition task. With additional training, we were able to recognize at most 4 composite shapes in one world with good accuracy, so our focus shifted to incorporating this model into existing code. Applying our model to the Architect model would hopefully improve the Architect's ability to recognize and describe shapes. While my coworker focused on that, I continued to work on the Builder model.
After I had compiled all the relevant data for the Builder, I started to train a sequence to sequence model to generate builder utterances. Once the generation had completed, I was posed with a new challenge: interpreting the results to determine what to change to improve the results.
After I had compiled all the relevant data for the Builder, I started to train a sequence to sequence model to generate builder utterances. Once the generation had completed, I was posed with a new challenge: interpreting the results to determine what to change to improve the results.
|
WEEK 7
|
July 8 – July 12, 2019
|
Now that shape recognition was mostly completed, the seventh week focused on finding the best parameters for our Builder utterance prediction model. I started by running a grid search to try out dozens of different combinations of parameters for the seq-to-seq model. Once the models had been trained, we decided to try a different type of model that would be able to use the information about current build structures along with the dialogue history to determine what utterance should be generated. This was similar to the model for the Architect that my co-intern was working on, so we ran grid searches for both the Builder and the Architect models with this new input data.
While we were working on the builder and architect utterance generation tasks, we started to think about what needed to be done to start working on the Builder action prediction task. This proved to be much more challenging to think about and would clearly require a more complicated model to accomplish.
While we were working on the builder and architect utterance generation tasks, we started to think about what needed to be done to start working on the Builder action prediction task. This proved to be much more challenging to think about and would clearly require a more complicated model to accomplish.
|
WEEK 8
|
July 15 – July 19, 2019
|
Throughout week eight, I continued to develop different types of models to accomplish the Architect utterance prediction task. We incorporated more data into the Architect model and experimented with different architectures for the neural network. I was able to learn more about the elements that make up a neural network through this experimentation.
While working on this, I also collaborated with my co-intern to modify our shape recognition neural network to be able to handle world states containing shapes of different colors. This was unfortunately more complicated than we had anticipated and taught me an important lesson in writing adaptable code.
While working on this, I also collaborated with my co-intern to modify our shape recognition neural network to be able to handle world states containing shapes of different colors. This was unfortunately more complicated than we had anticipated and taught me an important lesson in writing adaptable code.
|
WEEK 9
|
July 22 – July 26, 2019
|
In my second to last week, I worked a lot with the data. First, I collected statistics about different sequences of actions in the human-human data, such as how much time passed between an Architect utterance and a Builder block placement/removal. Then, I worked with my co-interns on the tedious but useful task of manually categorizing Architect and Builder utterances in the human data. Each agent had some set of categories into which each utterance would fall, so we read over each dialogue and added dialogue acts to each utterance. With this additional information, we might be able to more effectively evaluate the accuracy of an utterance generated by our models. To use this data usefully, we started developing a multi class classifier to read in a Builder utterance and output the dialogue act.
We also started to wrap up our work by thoroughly documenting our code and preparing a poster for our upcoming poster session.
We also started to wrap up our work by thoroughly documenting our code and preparing a poster for our upcoming poster session.
|
WEEK 10
|
July 29 – August 2, 2019
|
During the last week, we finished up our poster and presented it at a poster session with other DREU interns and student researchers from the university. The session went very well, we ended up winning the Best Research Project Award. After our poster session, I turned my focus back to working on the final report, which is the first technical paper I've ever written. This was certainly challenging for me but I actually enjoyed writing about the work we did which is certainly a turning point for my relationship to writing.
In between working on our poster and my final report, I continued to work on the classifier for Builder dialogue acts. This felt like the first neural network I had ownership over, so I was very proud when we were able to get 80% accuracy on our first test run.
Overall I feel very grateful for this opportunity, so thank you to the DREU program, the NSF, and especially my mentor Julia Hockenmaier.
In between working on our poster and my final report, I continued to work on the classifier for Builder dialogue acts. This felt like the first neural network I had ownership over, so I was very proud when we were able to get 80% accuracy on our first test run.
Overall I feel very grateful for this opportunity, so thank you to the DREU program, the NSF, and especially my mentor Julia Hockenmaier.