Research Journal
This site was originally a Wordpress blog, but since it will be hosted by the DREU program in the future, I figured it had better be scaled down some. These entries used to be in the "Weekly Updates" and "Research Notes" categories and so are all about my project this summer - and composed of weekly status updates, accounts of small triumphs, and lots of griping about bugs that just wouldn't fix themselves. But I didn't spend all my time in the lab, for stories about weekend and after-work activities, head over to the Outside the Lab page.
Week 1: Update
My first week at Washington University in Saint Louis has flown by, I can't believe it is already Friday! Due to a combination of study abroad and jury duty, I had to arrive here two weeks into the start of the REU program. I was very anxious about having missed all the orientation activities, and about trying to get a handle on a large-scale project when everyone else was already deeply immersed in it. I knew I would be contributing to one of two projects Caitlin Kelleher's research group is studying. One is a programming environment called Looking Glass that helps children (around the age of middle school) learn computer programming on their own by letting them create animated (and possibly interactive) movies. The other project involves creating a tool that enables therapists to create individualized games for stroke patients to aid in their recovery.
I did miss the orientation activities - and with it some of the better opportunities to meet REU participants in other labs and other fields of study. But luckily I arrived just after our group had finished up smaller, introductory tasks, and just before it was time to propose and begin the big summer-long research projects. My first two days in the lab were taken up almost entirely by brainstorming meetings, which turned out to be an excellent introduction to the grand vision of the project. I learned I would be working on Looking Glass, and after Monday I felt like I had a good idea of what the grand vision for Looking Glass is. I also learned that the focus this summer would be on creating an online community to support Looking Glass users. Mark and Terian are working on the website, while Michelle and I will be working on integrating resources created by the community into the IDE. I am still learning what already exists - but after a week of asking questions and creating my own world I think I have a pretty good idea. This seems like a great project and group of people, I am looking forward to the rest of the summer!
Week 1: More about the first week
Before I jump into my second week, here are some notes on the progress we made last week.
Warning: this is a long post with a lot of meeting minutes.
Monday Meetings
Monday was one huge brainstorming meeting, with a big break for lunch and some time at the edges that I used to start experimenting inside Looking Glass. All of the REU students had made a world* as a warm-up exercise to become familiar with the program, their examples, and mine, are included as examples. In the meeting our discussion was broken down roughly into three big topics:
- Enticing web visitors to try Looking Glass There are some big dreams in all three topics, one of the especially ambitious ideas in this one was the idea of being able to fiddle around with Looking Glass from within the browser. If users could make simple modifications to worlds, or even make a simple world that might be enticement to actually download the program and make an account. There was also discussion of many simpler measures we could take to suck people in. We thought it would be a good idea to make our sponsors (including the National Science Foundation) and our affiliation with Wash U much more prominent as a way to establish our credibility and allay any fears potential download-ers might have. There was also discussion on the extent to which we should provide support for schools. While our primary target market is kids who do not have access to formal instruction, it seems clear the program would work well in classroom environments as well. We weren't sure if we should go so far as a creating a separate teacher community and locked groups, but we thought we could explore those options. We also thought it might be worth bringing an educator into the lab and having them develop a sample curriculum - maybe even for English and language arts classes since Looking Glass is a storytelling tool.
- Keeping Looking Glass users involved in the community and programming Once they've signed up, we have to keep them around. There were all sorts of ideas in this category, drawn from our collective experiences in a variety of addictive games. Farmville came up quite often. There was much discussion of what some kind of points/badge/achievement system of rewards might look like, and we put that on the list of things that needed much exploration in future meetings. Along with status tracking, we also brainstormed ways to prevent boredom by keeping the experience always fresh and different: round robins, collaboration, and live programming (Terian is now working on this). Finally, concern came up regarding how we could keep the community positive (now part of what Mark is working on).
- Providing help and inspiration based on the community within the IDE Michelle and I are working on realizing some of the ideas we came up with regarding this final topic. We want the IDE and the website to be one Looking Glass experience and so we're going to be pulling things like how to information and snippets of inspiration from the community into the IDE. If you keep reading, you'll be hearing a lot about how we end up doing this.
Tuesday Meetings
The meetings were not over! The whole point system blew up into a huge discussion that is still ongoing. We had three types of behavior to reward: community participation, personal accomplishment in programming/storytelling, and creating and sharing a great world. There are all sorts of things we felt needed to be rewarded with some kind of point system: sharing worlds, asking for help, helping others, reusing code, commenting, and so on. Sarah is working on a way to evaluate the level of a user by looking at the complexity and sophistication of the programs they write. She is looking to be able to say "this user has learned how to do X," we thought these accomplishments should be rewarded separately with something like badges. We played around with the idea of having some characters cost more than others – again Farmville and Neopets came up. Soon we felt that that the complex economy with points, awards, badges, levels, and categories was waaay too complicated. I came up with the idea of using the characters themselves as the only sort of "currency" in the game. When a user is at a stage to be rewarded, the become able to pick a character out of a subset of some number of characters. Rather than having to deal with pricing the characters beforehand, we can let the community do the work and base the probability that a character will appear on how scarce it is within the community. This also deals with the problem of there being 1,000s of characters the user would otherwise have to browse through. People seemed to like this system, so we're sticking with it for now.
Wednesday Whack-a-mole and the Rest of the Week
I really got to work on Wednesday. I finally finished my first world, a sort of fake whack-a-mole. My grand dreams of a maze game turned out to be far too complicated, so I settled for learning how to manipulate the camera (sort of) and using some action listeners. LG right now is very limited in its game-making capabilities. It can only do very specific things with randomization, picking a random mole from an array of moles is not one of them. I could not find a way to make my custom "pop" method belong to all moles. Adding action listeners to each individual mole was pretty darn obnoxious. The rest of Wednesday was spent giving our proposals for our various summer projects and for the rest of the week we all began making paper, Paint, and Inkscape prototypes.
 |
 |
Inspiration drawer: drawer-based second draft of the community panel with the inspiration drawer open. |
Inspiration tab: original concept for the community panel with tinspiration tabe opened. This was the version used in my project proposal. |
* "world" is the word for what you create inside Looking Glass. Some further explanation of Looking Glass is probably needed here. Looking Glass can be used to create a movie, game, or some synthesis of the two. The user first creates a scene 3D space by dragging, dropping, and rearranging characters, props, and scenery. Eventually the existing models will be replaced by Sims donated by EA. EDIT: You can check out the about page for more information.
Week 2: Update
Until yesterday afternoon, Michelle and I (and Terian and Mark) had been making prototype after prototype. Despite the growing stacks of paper and directories of Inkscape files, it is hard to believe how much time we have put into our designs.
Our original tabs idea was early on replaced with drawers to keep the navigation around the IDE more uniform, but almost as soon as we had both developed our drawer-based designs to the point of user testing on Terian and Mark, we thought that perhaps pop-up menus launched from the menu bar would actually be more appropriate.
 |
 |
Here is a pop up window idea with drawer navigation. The idea of a scrapbook as something users could use to store things that struck their fancy started coming into its own. There didn't seem to be enough room in the drawers to display everything, and we questioned why some things needed to be squished in a narrow side panel at all - did you really need to be able to view both the contents of a scrapbook and your code/scene at the same time? |
The drawers seemed unnecessary in a pop up and took too much space. This was an experimentation with icons for navigation. |
We wrote up a Google Doc of use cases to decided when it would be useful to have content displaying beside your code or scene. We also had a long, agonized debate about when to make the user log in: should it happen in one place, or on an as-needed basis? We charts of what did and didn't require log in and that helped us to arrive at a centralized log-in. I'm finding this project is encouraging me to draw on all the tools I know to organize and clarify ideas. At one point, the four of us gathered in front of a white board and made flow charts to nail down (and try to simplify) the interaction between IDE and website. I had fun transferring the white board scribbles into pretty PowerPoint diagrams.
 |
 |
 |
Quite often, the four of us get together to discuss online interaction, communities, groups, contests, message boards, friending, contests, character selection, and on and on. The discussions are long, and sometimes surprisingly heated as there are several competing viewpoints among just the four of us regarding the extent to which the Looking Glass website is trying to be/should be trying to be a social networking site, how restrictive versus open things like challenges and messaging should be, and what the rewards system should be.
After hours of prototyping, it was a treat when Paul had Michelle and I start working with the code yesterday. It's only my second week, after all, and since we went straight to work with the prototyping, I still hadn't actually programmed anything yet. I was given the small task of making a button launch a popup window with some options for applying search filters. I knew that in a huge project with a large code base like Looking Glass, it was likely to be far trickier than it sounded ... but still, button and popup window.... I got quite a bit of experience building UIs in Java last summer, and have worked with Swing in various computer science courses at Pomona, so I didn't expect to be spending much time figuring out how to add text labels or formatting. But that is exactly what I've been doing for an entire day now. Looking Glass has something called Cheshire Cat. Cheshire Cat makes my task much more complicated. It exists for the purpose of creating tutorials. Instead of constructing a panel, adding a button and its action listener and calling it a day, the first step is to build an operation. The operation does what you want the button to do, the operation also creates the button. Then the button is added in a semi-normal way. Also, as a result of Looking Glass, I am never working directly with Java's Swing library. I am always working with classes within a "Croquet" package inside Looking Glass that extend the usual Swing elements. They appear to have their quirks. In order to create a simple check box, I had to deal with BooleanStates, SingletonHolders, and a whole host of classes buried in Looking Glass that I had never heard of or worked with before.
When I started trying to have my check boxes actually affect stored booleans today (instead of just looking pretty, which is what they did this morning) it got ugly. At one point the program wouldn't even launch! It turned out, after an hour, that there wasn't anything wrong with my code, some files had gotten corrupted and all I had to do was recompile everything before it worked. I'm hoping that there is a very steep learning curve with Cheshire Cat, but that once I get this popup working it will be smooth(er) sailing.
Week 2: Making Prototypes
After much reshuffling and digging, I had a popup window with check boxes operational by last Friday. It turned out some of my confusion regarding the time and place for tailor-made observers, states, operations was due in part to some flip-flopping over the long course of Looking Glass's development. While opening a pop up window has always been an operation, clicking a check box has not. I spent some time making custom operations for each individual check box, some more time removing them, and even more time adding them back in before talks with Kyle and Paul settled it all.
With my mini-project completed (check out the screenshot below), the making of prototypes continues. Michelle and I have more or less made the decision to go with the side-panel with drawer navigation. The scrapbook is what has evolved out of the "inspiration tab." It allows you to save characters, challenges, worlds (for remixing), and text to one centralized location. It travels with you between the IDE (a.k.a. the application/program downloaded to and living on your computer) and website. The idea is that as you browse characters and other users' creations you can save inspiring tidbits to your scrapbook and look at them at your leisure whenever you want.
The pictures below are of the latest prototype. The log in debate is back with a vengeance. Michelle and I both got feedback from our users that something is still very off. We might be too pushy with trying to get the user to log in. We're now thinking we should have gone with the traditional "Log in" link at the top right of the screen. Way back when, we had decided against that after comments that it was too hidden away in that place ... but as it's used by practically every website out there, we think it's standard enough to work.
There has always been a big question regarding the pop up character gallery. In the prototype below, the plan was to have it display only the characters the user does not already own. But testing confirmed the obnoxiousness of having characters split up into two different interfaces. Should the pop up display all characters? How can it best distinguish between them? How do we indicate to the user that they can't drag characters from the pop up into the scene editor? I need to put some serious thought into the terminology of this pop up. Is it a gallery, library, storefront ... something else? And into how these characters should be displayed. One suggestion was to put them in picture frames or somehow play on the looking glass theme. Not sure how I do that without it becoming bulky and busy, but that is what I'm experimenting with now.
 |
 |
 |
 |
 |
 |
 |
P.S. Don't forget there are explanations about what Looking Glass is and what these various panels, editors, and modes are on the about page.
Week 2: Change in the Scene Editor?
 |
 |
 |
As expected, interesting frames, while interesting, take up way too much space. Also, they don't match anything else about the UI. |
Displaying both owned and not-owned characters in the gallery. Distinguishing between the two with a button and gray background. |
A drastic idea that (probably) won't be realized, re-organizing the scene editor and gallery browser entirely. |
I really want to be done with making prototypes before the end of this week. Today was all about wrestling with the question of whether the characters you do and don't have should be displayed together. Turns out the character gallery is not as simple as we had assumed.
Display Together? |
|
| Yes | No |
|---|---|
|
|
Week 3: Update
 |
A last-minute design that everyone seems to like. |
After our Friday meeting, a lot of ideas about the character gallery finally came together. Right before leaving I whipped up this design, which seemed to solve most (?) of the problems the other designs had suffered from. It combines those previous ones, thoughts about mode switching, and consideration of what tasks the user should be focusing on while browsing characters, with inspiration from Gmail, Hulu, and Amazon. After a long, frustrating week of prototyping I was relieved to come up with something that seemed Right. I'm very much ready to start coding again and looking forward to getting started next week; I never thought I would spend this long designing!
Week 4: Update
 |
No idea why this was happening, or what it was that ultimately stopped it. |
This week went by shockingly fast. On Tuesday I finalized the design from last Friday and thought about how the character credits, Sims creator, notifications, and 3D Text and Billboard features would all fit into it. Then I set about making a button in the character gallery open a panel in the scene editor. Courtesy of Croquet and Chesire Cat, this was a mess. I'm not sure what was causing the weird background overlay in the above picture, but it seems to have disappeared. At one point I lost at least an hour because I was accidentally launching the "Faux Scene Editor" instead of the "Move and Turn Scene Editor." Fun fact: there are 2 FauxSceneEditors, one that extends AbstractSceneEditor and one that extends AbstractInstantiatingSceneEditor (which itself extends AbstractSceneEditor).
Thursday was consumed almost entirely by a meeting of the whole lab to discuss the database. That is Mark and Terian's project.
Some outstanding issues:
The whole system of gaining new characters is still vague and undefined. There are some big problems - ease of gaining character credits via remixing versus difficulty of winning a challenge, should you be able to create as many humans as you like for free?, should we differentiate between a dragon and a potted plant when it comes time to spend a credit (at some point, don't you need potted plants? should they come free? isn't 99% of the Sims composed of rugs, paintings, chairs, and potted plants?), Every time it comes up, the resulting discussions suck up hours and never seem to reach conclusions. As a result, we keep tabling the issue. Eventually, we'll have to confront it.
I think we also at some point need to familiarize ourselves more with the Sims assets. Last week I boldly took it upon myself to play the Sims, something I hadn't done since elementary school! I took it home to play with the pets expansion pack, but I had issues installing that so instead I spent a whole afternoon constructing a mansion. The items are so different from the existing Looking Glass and Alice ones, it will completely change the tone of the stories users create. It also makes it easy to spend even more time in the scene editor, adjusting the position of lamps and tables instead of programming. This is already something of a problem, but it might get much worse.
Week 4: Weird bugs
 |
 |
I'm not sure what caused Wendolina the Clown to disassemble ... hopefully it wasn't something I did? |
Something happened to some of the drawer titles. I was looking at how the drawers worked ... maybe I inadvertently made changes? |
I hope these aren't too serious.
Week 5: Update
 |
I spent all last week working on realizing the "transparent gallery" design. Transparency turned out to be more complicated than it really should have been - the weird repainting of the entire scene editor, the loss of transparency on mouse over ... so I'm going to deal with that later. I successfully rearranged all the various split panes composing the IDE. It is sort of horrifying how many there are - the change between Action and Scene editors is largely a rearranging of split panes. I also got the transition between "add to cast" and "use credit" modes working and had some minor successes with formatting.
Mid-week I needed to start working with character-like things to start on the interaction between the "transparent gallery panel" and the "cast panel." I was reluctant to mess around with the characters in the existing character gallery ... they are accessed through a tree and it looked like it would take several days to figure out how to use them without the current set up. The tree structure will presumably disappear when the creation of the database (and later, the introduction of the Sims) makes it obsolete. I considered at one point just passing around text to avoid wasting the time of dealing with it all, but in the interest of having a still-functional program, I decided to do some really ugly things with the tree to make the characters appear at all. I'm starting off week 6 working on the horizontal scrolling featured in the design. It, like everything about croquet, is turning out to be harder than it seems like it should be.
Week 5: Incremental Progress
 |
|
This picture demonstrates two problems that have been cropping up. The first is the sad disintegration of the popcorn cart, which is not a new bug. The second is the immortality of one of the buttons from the transparent gallery. If a button is dragged, sometimes, but not all the time, an image of it will be frozen in a certain place on the screen. That image persists long after you have stopped dragging and even closed the gallery. Clearly some sort of repainting issue. I've turned off dragging for now since there's nowhere to drag to anyway. |
Showing my successful moving of some radio buttons. Edit: discussed in more detail in the week 6 update. |
Some screenshots of how my changes in the scene editor are coming along.
Week 6: Update
 |
The early difficulties of getting the "transparent gallery" to appear at all were replaced with relatively clear sailing once I had carved out an area to work with and could fuss with the look-and-feel and start on the navigation inside those panels. By the end of the week I had the "get more information for a character you don't own" configuration up and running. As part of the process, I created an "arrow scroll panel" class to handle the horizontal scrolling of multiple rows of components envisioned in the design. I'm pretty proud with how flexible it's been, but I can't figure out what is wrong with how it is working the cast panel at the bottom.*
In the code that resulted in the screenshot in this post. A "cast selectable panel" was added twice to the "cast panel." This selectable panel used to exist in the form of radio buttons (top left-hand corner in this screenshot), and my amateur attempts to convert them into proper buttons has been ... well, that will come along eventually. You can see them appearing once as an irregularly spaced row of grey rectangles (the yellow one is selected). Why are these rectangles appearing only once despite being added twice? Err ... I added it twice to show that the selectable panel was working, just not appearing in the arrow scroll panel. Then I slapped the green "Add to Cast" button in the arrow scroll panel to show that the latter exists and can have things added to it. Since I had no problems getting the arrow scroll panel to show the 1,000+ Looking Glass characters, or those 3 dummy challenges, my current theory is that there are magical listeners/observers out there attached to the selectable panel and that the arrow scroll panel is not listening/observing correctly. In addition to being skeptical of any theory involving magic, I'm sure why one extra layer of panels would block the listening/observing abilities.
I made precious little progress on the cast selection problem over the course of rather many hours, so I've shelved it temporarily while I get searching functional again. I like that this project has all sorts of smaller sub-tasks to it so that I can jump around pretty easily when I get stuck and need to let theories simmer for awhile.
On Friday I was getting very anxious about the non-existence of the all-important database. I now have FakeCharacter, FakeChallenge, and FakeTag classes ... it's getting a little crazy. Friday's meeting revealed that I'm not the only one worried about getting the database running before the end of the summer, so on Tuesday we're supposed to have a meeting about that.
* For reference ... This is the screen layout I'm referring to:
Transparent Gallery |
Side Panel |
Cast Panel |
And this is the structure of an "arrow scroll panel" where < and > are buttons that, when clicked, scroll left and right, respectively:
Top area with (optional) title and room for buttons |
||
< |
Area where list of components |
> |
is displayed in multiple rows |
||
Week 6: Tagged tags
 |
 |
Current progress on browsing by tags. This is a character info view of the transparent gallery. |
The kitchen tag getting confused. This is a tag info view of the transparent gallery. |
Before I get search working, I need to get tags working. This is turning out to be a nightmare. Of course, the current characters aren't tagged, but they are to some extent organized within directories. The tree structure that I complained about earlier this summer was based on this organization. I figured I would tag the characters with the names of the folders they're in as well as their own name. This seemed simple, especially when I got fake tags appearing (example, back here the ChesireCat is tagged "textbookdemos" and "newstufftemp") but somehow I managed to run into problems.
Most dire now is that somehow its getting confused between what is a tag and what is the name of a character. When it generates a list of related tags, it includes characters in this list (as in the "search term: kitchen" screenshot above). This might seem reasonable since I did grab names and turn them into tags. But it really should not be possible. A tag is an instance of a FakeTag while the name of a character is a string that can only be accessed through a FakeCharacter. I know that is is grabbing actual character names because of the capitalization. I'm now forcing tags to be lowercase in about three different places simultaneously, so there's no excuse for the "Table table Toaster toaster" business going on - Java should be able to tell the difference between FakeTags and strings. The solution to this problem probably lies in the fact that both FakeTags and FakeCharacters extend the abstract TransparentGalleryViewable class I used to make browser-style navigation and display less of a headache).
I'm really not sure why there don't seem to be any problems with two characters named the same thing. In the screenshot attached to the post is a particular case I'm aware of and have been keeping my eye on. There are two characters named "Horse." One is stored in Environments > Medieval and the other in Objects > Animals. This creates the need for 3 things - 2 characters named Horse , and 1 "horse" tag. It seems to be successfully doing this... but the whole concept of "related tags" (should medieval and animal really be related because they share horse? I think there are just not enough tags for this to make sense) has proven to be a bit of a mess. I thought at first that I could do some hocus pocus and give tags parents and children ... but then I discovered the case of duplicate directory names. For example there are at least two "animals" directories, one under objects > animals and one under environments > Hawaii > animals.
Today has been a little frustrating, especially since I know most of this will go away once there is proper tagging. But that might be a long way out and I need to get this thing useable by the end of the summer!
Week 7: Update
 |
 |
 |
Working "character info" view of one of the horse characters (the medieval one) |
Working "search terms info" view of a search of "horse princess" |
Working "tag info" view for the "horse" tag |
Tags work! I was right yesterday when I said it should not be possible for tags and names to be both getting grabbed. I was also right about forcing tags to be lowercase in about three different places. ... But I was wrong in thinking that the TransparentGalleryViewable class has anything to do with my problems ... and wrong in thinking there were two characters named "Horse."
I found my bug when I noticed the big "DO NOT USE TO TAG AN ITEM!!" note that I had left for myself in the documentation for the private getInstance(String s) method of FakeTag. Right below it was the registerItem(String s, TransparentGalleryViewable i) that is supposed to be used for this purpose. Well, was I misusing my own method despite the large warning in all caps? ... Yes. Yes I was. During the mess of tagging tags based on the directory structure I used the getInstance method to tag a tag with its parent(s) and children. Sigh. So silly. Once I discovered my mistake edits to two lines of code fixed everything.
Search turned out to be shockingly fast to implement. It only took about 15 minutes or so thanks to all the abstract classes I had set up yesterday. My first search for "horse" revealed the existence of a third horse I had never known about. Success!* Search and tag browsing are both working! I'm glad to end this week on a good note.
* Although on the first time it didn't find the fourth horse, "horse2" ... but that is almost fixed now.
Week 8: Update
This week was spent exploring the extent to which my excitement about tags working last Friday was totally premature, trying to fix tags, getting frustrated with tags, working on the paper I have to write for the DREU program, and helping Mark and Michelle prepare for their presentations. The two of them were nominated to give talks this Friday and all the Washington University RE program REU students (as opposed to DREU program REU students like Michelle and I) were required to make posters for a poster conference in the afternoon. We helped them do three rehearsals before the fact and they both gave great talks and did stellar jobs answering questions! They've (probably) written more about it on their blogs: Mark's blog, Michelle's blog.
Week 9 and 10: Updates
 |
 |
 |
 |
 |
The lab wasn't quite the same without Mark and Michelle. With Jordana's guidance I conducted formative user testing. I asked 5 users to create a scene using 5 items. They were given a theme - old west, cabin in the woods, or classroom. I estimated it would take 10-15 minutes, it actually took anywhere from 2-40. I felt awful for the 40 minute case as that was partly because Looking Glass crashed halfway through. I think that was a problem with the items themselves (perhaps related to the familiar disintegration issue?). It wasn't impressed with all the positioning and movement. I wish I had more time to keep working on this project, specifically, I wish I was going to be around long enough to see it hooked into the database/website/Sims!
One of the biggest takeaway messages from the testing was about environments. I know my design needs to make it clearer, but I'm worried this is really a larger issue with how scene editing works. Users looked for wall and fence tools. They were confused about how to create interior scenes. I remember that time I spent playing the Sims, i.e. the time I spent building my dream mansion. It's a real problem steering people away from building. Kyle and I talked about having pre-made settings - locations maybe even pre-populated with furniture and such. I think that is a good idea worth looking into. Hopefully the online community will encourage users to share pre-built settings, but I almost think it might have to be a different mode or something. Maybe when you start a new project, that's when you pick. Oh well.
I spent the last week picking at longstanding bugs, working on my paper, and working on a poster (the final results of which can be found at the bottom of this page). It's been a really great summer experience! It's going to be sad and strange leaving behind Looking Glass and the lab. I sure will miss everyone - I hope we can stay in touch. I'll be looking forward to when Looking Glass goes live online. Thanks for reading!