This week was spent finishing up various parts of my research experience.  On Monday, Kyle and I went through all of our user study data and entered it into excel.  Removing the mercy condition, the results are as we hypothesized.  However, this leaves us with only 12 users, so Kyle will have to run some more user studies in the future.  I, however, will not be here for these. :(

The rest of the week was spent getting my aspect focusing working (which it now does...yay!!!!!) and making my poster for the symposium.  Here is my poster, with the demonstration of what the aspect focusing looks like as well.

Mini picture of my poster:

If you would like to see a larger version:

Right click (or mac equivalent) on the picture and select 'view image'. From that page you should be able to enlarge the image.

P.S. Thanks for a great summer.  I'm going to miss working with all of you guys!

Last Friday began with another user study.  Caitlin, Kyle and I then met to discuss where to go from there, and the changes we would need to make before the next test on Monday.  Afterwards Kyle and I decided to come in on Sunday to work on these instead of staying really late on Friday.  So on Sunday I fixed up some of the existing user study worlds, and remade some others.  I added easier to grab remixes, so the kids wouldn’t have to use zoom out in order to get the right action (they never use it).  I also created a really simple starter world and remix for it, that we will walk the kids through at the beginning of the study, so they don’t get completely stuck on remix.  That’s the majority of what I did on Sunday.

On the following Monday we had another user study, and things went pretty well.  We came back with a much smaller list of things to change that we usually have.  Afterwards Kyle and I had a meeting with Dennis about different things that needed to be fixed, and how to go about doing some of that.  Without any immediate changes to the user studies needed on my part, I was free to move onto my personalization project, which hasn’t seen much action as of yet. 

The first step was to create the user model, so Monday evening I made a simple class that was the starter for the user model.  On Tuesday I world a lot more with the user model and added a few more actions to track.  Basically what the user model is so far is a Boolean for every action a user can do (that we can track in the transaction history)  and based on what that Boolean is set, it will print out a different message of whether they have seen it before or not. Doesn’t do much o0f anything exciting, but Kyle figured that this was a good first step in constructing it.  It has been working so far, which is pretty neat.  I do like when things actually work. ;)

Wednesday was the same mostly.  Worked more on the user model, trying to add more cases.  Unfortunately I had to ask Kyle to work out a bunch of eclipse issues (which I may or may not have caused), and that took a bit of time.  I started listing out individual cases for expression statements (dropping in things like move, turn, say, etc.), as I had them all lumped together previously.  I thought that after you did a few of these, regardless of whether it’s a say, setOpacity, or a turn, a user will kind fo get the hang of it, and not need the tutorial to tell them exactly how to click and drag a statement.  But, I figured we’d still want to track each individual thing in the user model, regardless of how it will affect the tutorials, so I am adding in each case of Expression Statement.  I got all the procedures for characters, but still need to add the ones for world and camera.

And today, Thursday, I (surprisingly) did more of the same.  I realized today just how vast the amount of trackable data in Looking Glass is, as my list of components to track keeps getting larger the more I play around with Looking Glass to find everything users can do.  I also worked today on making the user model neater, and making it flow better.  I also worked on an output method, so instead of having to write out each individual statement, output will just spit it out for me, based on cases.  So no more ridiculous amount of print statements.  So far, I have gotten the user model to track all character procedures, action ordering boxes, the play functions, declaring methods, and changing between states. I have also added more Boolean trackers for other actions, but haven’t actually written in the code to track them yet.

This process is painstakingly slow sometimes, and hard to follow sometimes.  The transaction history window isn’t the always the easiest thing to read, but at least I am making progress.  I now feel Michael’s pain, when he had to look through all of the ast nodes his first two weeks. Haha.  Oh, and Kyle and I had a meeting at the end of the day to address the user study stuff for next week.

Working on the user study stuff took up pretty much all of this week.  I started off fixing an issue we were having in the user studies with procedures.  The participants were being confused by the adding procedure step, so we resolved to just have Looking Glass do that part for you.  This is what I spent Monday doing.  I set up the framework for the action, and had it print out when a step in the transaction history would be skipped. With Dennis’ help, we got it to actually skip more or less correctly.  It wasn’t a hundred percent, but it was skipping some things.  However, we ended up not using this for the user studies and opted to instead just have users deposit the code directly into the main method, and not deal with procedures at all.  Good news is I will have to do a similar task for my own project on personalization.  Finding the correct transactions and checking if they have been activated or not is how I am going to monitor the basic actions in the user model.  For instance, checking if someone has ever done something before, like used a do together, or even played their world.  I can scan the transaction histories on the remix tutorials, and if I see a do together instance, I can mark off that they have now seen a do together.  This is the first level of personalization, introducing concepts.  And say that they haven’t seen one before, but they are going to in this tutorial, I can change the notes generated so they introduce a do together.

Tuesday was another user study.  I spent the morning before we left fixing up some things we needed to test for that day, and I spent the afternoon beginning to create the nine worlds we would need for the actual tests, and having a meeting with Dennis and Kyle about the various issues we needed to fix before the next user study on Friday.  On Thursday I finished these worlds and began trying to test them, by making common mistakes users do, and making sure the tutorials didn’t blow up…..unfortunately they did…a lot.  So, I ended up working with Dennis, Paul, and near the end of the night Caitlin, to try to get the new worlds ready for a user test we had the next morning.   In the end, the do together worlds, and the count worlds were proclaimed ready! Hurray!  We didn’t manage to get the for each up and running, but hopefully we will soon!

This past week I did a lot of swapping back and forth between user study stuff and working on my project, like the last week.  However, as we are progressing, I am finding out that the two are relatively connected.  So, working on one is helping the work on the other. 

I spent some time looking more into ways to make tutorials more interactive, and discovered a few papers about Tangible Interfaces for Collaborative Learning Environments.  The researchers have developed a prototype system that "watches” as students play with Tangram pieces on a physical tabletop, and acts as a "guide on the side” by offering help at appropriate times.  This specific case does not so much apply to Looking Glass, but I thought the idea behind it was fairly interesting.  So for us, the user model can keep track of what a kid interacts with (apparently it’s also in Transaction History) during the puzzle-tutorials, and help nudge them in the right direction.  It knows what they are doing, and it knows what they should be doing, so it can help direct them to their goal.  I definitely like the idea of users having something to solve, instead of just mindlessly clicking buttons.

I went to a few user studies this week, one on Monday and another on Friday.  We have seen similar results for most of the participants we get.  Caitlin, Kyle and I had a meeting about the user studies, and discussed what was going on, and where to go from there, which seemed very beneficial to me.  I always like to get status updates.

We also had an epic meeting about badges, that consisted of creating post-it diagrams/creations and sticking them all over the walls of the conference room.  I missed part two of the badges meeting, so I’m not sure of what was decided, but I feel the work I did on Tuesday for my part was helpful to my user model.  A lot of the aspects Michael, Patrick and I were working with could be things to monitor and track in the user model.

I’ve also started working on the first concrete thing to add to my project of personalizing tutorials, and it also fits nicely along with the user study.  The idea is minimalizing the amount of information we throw at users each time they have a tutorial.  The way we have it now, we are presenting too much information, and they just tune out and click through, while not really absorbing any of the information.  So, that when they get to the transfer task we have created, they can’t do it, since they don’t remember any of the steps they did previously in the tutorial.  To address this issue, we thought that perhaps offering information in segments was a better approach.  The tutorials would start out telling the users how to play the world, what a do in order is, and maybe what a do together is, and steps containing any other information would be done automatically.  I feel that this would create more direction in their learning, as well as make it easier to learn concepts in general.  You don’t throw a whole textbook of information at someone, you give them chapters at a time.  So subsequent tutorials would introduce a few other new concepts, and so on and so forth, with increasing difficulty, as the user model decides they are ready. 

With all this in mind, I started working on sketching this out Tuesday afternoon, and a bit before the user study we had on Wednesday.  I have since then been working on actually implementing this in the code, so that it can be tested in our user study on the following Tuesday.  This has mostly meant digging around in the code and trying to figure out what everything does.  I feel like I’ve made some headway here, but I haven’t quite located what I need to, to make these changes to the tutorial process. I will continue working on this tomorrow, and into Monday if I haven’t completed it by then.

This week I started off with editing a few aspects of my proposal, and making some worlds for the User Study that I was helping conduct on Thursday.  I had various restrictions to abide too, so it took some thinking to start creating compelling examples.  I couldn’t use any getPart functions,  add details or custom numbers, which limited me to only moving full characters around.  I did manage to come up with some examples, and Kyle and I narrowed down the specific ones we liked best.  I then made a few minor changes to them to make them more user-study-friendly.  We also had an extensive meeting about everyone’s project proposals, which lasted most of Tuesday, but I did get to start thinking about the changes/additions proposed to me in the meeting.  

Wednesday I thought more about my project; took notes, made sketches, made some lists, etc.  Specifically, I started making a list of aspects, actions, trends we should track in the "user model” that Kyle suggested.  I spent some time working with the Looking Glass software, trying to find every part of it that a user could interact with, the normal things like do togethers and move actions, but also things like object or camera markers, listeners, arrays, or variables.  And I liked the idea of the user model tracking other factors, like how often they’ve used something, or when it was last used.  So if a user has seen how to use a count loop before, but the user model tracks that they haven’t used it in a while, and are instead repeating blocks of code, the tutorials could reintroduce the count loop.

I also thought a bit about how to make the tutorials more interesting.  I researched this, and looked into various ways of interactive teaching.  I thought of instead of focusing so much on text, maybe using more images in the process, or giving more examples of what code could look like.  Personally, I learn best through examples, so this idea seems intuitive to me, but I’m not sure about how middle-schoolers might view it.  Other ideas for tutorials included making it into a game somehow, or some sort of puzzle to solve.  I think people might pay more attention that way.  So instead of just having the blocks of code to drag in in order, perhaps the order would not be specified, and users would have to arrange the puzzle to get the correct order (like Caitlin suggested).  Or I thought that there could be several code blocks at each step, and only one would be right, and users would have to find the correct one to do the described action.  This would have to be at higher levels though, once users understand the basic concepts, and can ascertain what code does before they play it. Perhaps the "test” structure would help kids learn more.  Instead of just mindlessly following steps, they’d be required to think about which action to choose and why, thereby learning something…hopefully.  It works in school, right?  Just the content of some of my random ideas/sketches.  I also modified the User Test worlds again on Wednesday, to get rid of some complications they could cause in the study.

Thursday I spent getting ready for, and actually doing the User Study.  I think that this experience was extremely helpful.  I really got a feel for what concepts kids know, can figure out, and have trouble with.  I realized I was still not exactly thinking form the user perspective in my designing, and I feel that this experience was a real eye opener.   Kyle and I had a meeting afterward about changes we need to make to the Remix/Tutorial process before the next User Study on Monday, and I started working on some of these issues, so that the next trial will be able to progress smoother.  I am looking forward to the User Testing on Monday.  I find that I rather liked the process, and think that they will be continually beneficial in my efforts to make Looking Glass easier for kids to understand through the personalized tutorials.

High-level problem:

For my project, I want to do two things.  The first is to finish revamping the remix process and second, to look into the idea of personalized tutorials.  Therefore, my project proposal deals with two high-level problems.  First, what if a user cannot find the code that she/he is looking for, even after going through the tutorial for locating it?

 When remixing, you are using another user’s code, and sometimes people may place code in spots that are not intuitive to you.  By using tools already available in Looking Glass, users can try to narrow down where that code may be.  My project would also deal with how to make the remix process in general, easier to use and understand.  The second high-level problem has to do with the second aspect of my starter project, which regards personalized tutorials.  Often in a program, tutorials are one-size-fits-all, which can lead to boring over-explanations to more experienced users, and under-explanations for beginning users.  Learning should not be one-size-fits-all, and the personalized tutorials would see to that being corrected.

Why it’s important:

Remixing is a very big part of the Looking Glass software; a process that sets Looking Glass apart from other programs designed to teach kids to program.  The aspect of reusing code from other users is something unique to Looking Glass.  If users are not able to find the correct line of lines of code for the actions they wish to capture, they lose a significant aspect of Looking Glass, that is why users being able to correctly and efficiently use this process is very important.

When going through the proposed starter projects, I was very interested in the idea of personalized tutorials, but did not have a chance to approach it in my first two weeks.  I would like to approach this subject in my subsequent weeks working with Looking Glass this summer.  As I stated above, tutorials are usually created to be applied to any and all users, however that might not be the best way we can do this.  The idea of more personalized tutorials would allow the user to still learn new things, as all good tutorials should do, but also learn them at their appropriate skill level.  If a user already knows how to create a new method/procedure, there is no reason to tell them all the steps how to do this in a tutorial regarding using multiple procedures or something of the like.

What I’m doing:

Remix Stuff:

I went through two different possibilities of how to implement the help process.  One of them, like Kyle asked, was a wizard-type action, where it would guide users through the process, checking each step of the way if the code was what they wanted.  The second tries to get the users to interact more with the features of Looking Glass, instead of just hitting next a bunch of times, and waiting for the program to fix the problem for you.  It is more useful for learning.  The second version is like a tutorial itself.  It gives the user a list of possible scenarios in which their code might be, and instructs the user how to proceed.  I think after the user testing, I will have a better idea of what kinds of code placement scenarios kids have the most trouble with, and make sure the help program covers them.  I’m still in the process of figuring out exactly how this process is going to work, and whether a wizard-type application might be better or worse for users to use, and how to get users to use this process in general (possibly changing the name, and the location of this process).  Perhaps even a combination of the two?  Something that requires user interactivity so they won’t get bored, but also makes them use the feature of Looking Glass and doesn’t do everything for them.  Caitlin and I only decided on this general design on Friday, so I haven’t had much time to devote to picking this process apart, or possibly coming up with a better way to do it.  I’m not entirely happy with either solution so perhaps I can come up with something better in the future.   

The next change I want to make is to the general Looking Glass interface, specifically changing the step forward/backward action.  Aright now it is set to step back and forth by miniscule time units, so you might end up having to press the button several times to move back even a second.  And even if we made the increment larger, a user might skip over an action they were trying to find.  I think that the best solution is to make the step back/forward button be based on changes made in the Right Now panel, so based on actions instead of time.  Users are looking for a time to find the action associated with it.  By making the step function action-based users can just skip a step!

Another usability change I would like to make is to add an "exit” button to the progress tracker in the Remix process.  While users can press the stop playing button that carries over from the "play and explore” interface, since the interfaces are the same, it would be easier if there was a clear "exit Remix” button on the progress tracker.  And perhaps we could consider not having these two things have the same interface?  Like removing the remix progress tracker when a user isn’t actually remixing, or removing the stop playing button when a user is actually remixing….Just a thought that occurred.

Tutorial Stuff:

The tutorials will be based off of actions that a user remixes.  So, for every action a user remixes, the user can go through a tutorial that will show them how to recreate that action.  For any aspect of Looking Glass (Action ordering boxes, functions, procedures, adding details, creating methods, etc.) there are varying skill levels, and the material in the tutorials will reflect these different levels.  If a user has never created a method, if they need to create one in a tutorial, it will walk them through every step.  And likewise, if they are familiar with it, the tutorial will just tell them to create one and then move onto the next step in the tutorial process.

I went through a couple different places to put the tutorials, but ended up on the decision to put a button at the top of the Methods Panel, by the Challenge, Share and Remix buttons.  I thought that having it up there with the other "Actions you can do in Looking Glass” buttons seemed like a logical place.  Another location could be in the Open World pane, and add a tab for tutorials.

It would have a spot to choose a world, and a sort option. Then I also thought it would be useful to add a Preview spot, and a spot where the full description is given.

Another idea regarding this process was which actions to make tutorials out of.  At first I thought to just make any action a user remixes available as a tutorial, but maybe they’d end up with an annoying amount of useless ones that would just clog up the list of tutorials and make it so they couldn’t find the tutorial they were actually looking for.  So the solution I came up with was giving users the option to make an action into a tutorial when they remix it. So, the best place I thought to put this option was in the final dialogue box to the remix process, where users name their action and cast characters.

There are five different levels of "difficulty”, that is, five different skill levels for the tutorials.  The level will be based on the badges you earn, but I also wanted to make it a little independent as well by having the users need to use an action several times per level to reach the next one. (and not just level up because they got the badge for it, but have both the badge and multiple usages as requirements).

The levels are: beginner, intermediate, advanced, expert, and pro.  Descriptions of each level are as follows:

Beginner:  The user has never seen and never used that aspect of Looking Glass.  There has been no use of the actual code, in Remix or on their own.  So if someone was a Beginner in Do togethers, the tutorial would tell the user where it’s located, how to use it, and afterwards explain what a do together is.

Intermediate:  The user has seen said actions, and maybe grabbed it in a remix, but hasn’t actually manipulated the code themselves.  At this level, the tutorial will still tell the user where to find do togethers, but it won’t give them a definition, as they probably already know it by this level.

Advanced:  The user is familiar with the aspect, and knows where to find it.  They are more familiar with adding details, and changing inputs on actions like say, turn, etc.  They have remixed it, and maybe modified the action before, but haven’t created one themselves.

Expert:  They had no help from tutorials or Remix, and created basic versions of this aspect by themselves.  This is the first level that users are given pre-created lines of code to drag into tutorials, since doing these would be tedious as they are already quite familiar with these concepts. (For example, a user could be given a line of code that already has a function assigned, a duration, etc.)  I also thought of getting rid of the stencil-cutout version of the tutorial at this level and up, to give users more freedom to choose what actions they want to do.  They are given the correct tools and code blocks to recreate the action, but maybe they want to change it a bit, or add something else, so they are given a layout where they can still edit anything they want.

Pro:  At this level, users are very familiar with the aspect.  They’ve created it themselves several times, and with varying levels of difficulty. (Like Reily’s badges, ex: nested loops, multiple details, custom variables, etc.)  This level also uses the "stencil-free” version of the tutorials, and users are given larger code blocks, instead of just lines of code.  It would break the methods up into manageable blocks that users could put in place.

The way I’ve presented this makes it seem like there is only one broad skill level a user can be on, but I think it would be better to have each aspect have its own skill level.  So a user can be an Expert in the count action, a Pro in do togethers, and a Beginner in variables all at the same time.  I made an example of this, where a user is familiar with do togethers and do in orders, but isn’t very practiced with adding details.

Another thing I thought of adding, but wasn’t sure where to put, was some sort of reward for adding comments to your code.  When going through the Remix process, one of the problems faced by kids unable to find the correct code was poorly named methods.  By giving the users some sort of prize, or experience points, or something of the like, when they add comments to their code, explaining what certain points do, perhaps the code can become easier for other users to understand.  Could be a possible solution to the poorly names methods problem.

I spent most of my time this week brainstorming, sketching,  and brainstorming some more about specific aspects of the remix process. Things to add, things to remove, things to change, etc.  I made a few grammatical changes to the existing text to try and make it clearer, after doing some research on what makes a good tutorial.  The first aspect I focused on was the progress tracker bar at the top of the screen.  The main addition to that was adding another panel that would allow the user to easily exit the remix process, since there was no obvious way to do this, other than hitting the "Stop Playing" button in the world viewer.  Caitlin and I figured that a clear exit was a good thing to add.
The next small change I thought of was combining the Cast Character dialogue box into the Description one, so you end up with only one dialogue box at the end of the process.  We would also need to remove the  character roles panel, since the Cast Characters needs to fit there.  However, with the preview captured action video in the same panel, you wouldn't necessarily need the character roles panel to remember which character does each action.
I also read a few papers by Caitlin and Paul relating to the remix process, and how users go about finding the code for a certain action, which lead into the next chunk of brainstorming.  I did a little bit of thinking about the tutorial process, and then focused on remaking the "Help" function that you can access from the menu of any line of code.  I think this could be a great tool to help users find the code they want.  I thought of many ways to do this, but none of them seemed to get at the core issue.  How can users most easily locate the specific code they are looking for?  What can they do if the action the tutorial process has led them to isn't the one they wanted?  This process is ending up something like posing questions to the user about where the code they want might be, and showing them how to progress from there.  This has mainly led to me making a list of all the possible situations I can think of that a user could end up in, and how this process could help the user find their code.  I also spent a bit of time looking through the submissions to the looking glass site, to get examples of various code structures one could expect.

My two week starter project is to: first, rethinking the remix process, making it more streamlined, etc.; and second, once the badges and rules are introduced, to be able to generate custom tutorials based on what badges a user has earned—in other words, the creation of personalized tutorials instead of just the basic tutorials for everyone. 

I spent Wednesday afternoon familiarizing myself with the remix process, and brainstorming about ways to make it more efficient, as well as noting a few possible issues.  Thursday morning, I was introduced to the actual code and the updated version of the remix process, which already included many of my "efficiency propositions”; things like making the tutorial part optional, and making it take less steps, creating more ‘back’ buttons, etc.  For most of the morning, I looked over the code, to get a feel for what each part did and how to go about changing things.  Caitlin and I had decided to remove the Preview section on the progress bar, and integrate that into the description pane that follows, as well as reorganizing the pane and removing the start and end screenshot panel.  I got everything but the Preview Captured Action panel completed.  I will continue to work on that tomorrow, and hopefully be able to fully integrate the Preview captured action panel.