Final Report


Generally when someone creates a spreadsheet they have an idea how it should work that goes beyond entering formulas.  They know what ranges a cell's value should stay within or they may have an idea what kind of relationships exist between cells, anyway, this is all extra information that a user currently does not have any way of telling to the system.  Assertions are our way of letting a user communicate more constraints or information to the system.  Assertions are a type of guard that a user can enter on a cell, say on a spreadsheet.  As of the current implementation, a user can enter a specific range, say 0 to 10, on a cell that conveys to the system that values in this cell cannot lie outside that range.  Dr. Burnett  and her group of students are working on implementing other forms of assertions, so, for example, a user can convey a relationship constraint between cells.

Here is a black and white picture of the interface of Forms/3 (the boxes on top of the cells are Help Me Test assertions- you can also have user and system assertions):

Before I came here for my research experience, Dr. Burnett and Dr. Cook together with their students, conducted a study that was interested in finding out if assertions would prove to be helpful in debugging spreadsheets.  They were not interested in finding out whether users would actually use assertions in testing their spreadsheets and if their own assertions would be helpful in debugging.  This was an interesting question that we thought should be investigated.  So, when I arrived, it was my job to be the the primary designer and administrator of our new experiment into whether end-users would use assertions or guards in a spreadsheet and if so whether they were useful in debugging.

I'm currently writing up a tech report which details the experiment design, participants, tasks and materials, and the tutorial.  We are going to combine our report of our experiment with the previous study of assertions because they are closely related into one large tech report.

I don't expect the final copy of the tech report to be finished for awhile, but do I want to share with you  some of the major sections of the report that I've written.

Experiment 2
Experiment 1 established that assertions are useful in debugging.  In order to build upon these findings, we wanted to examine whether end-users would enter assertions and if their own assertions would be helpful in finding and correcting these errors.

Design and Procedures
Since Experiment 1 established that assertions are helpful for participants in finding errors in spreadsheets, we wanted to learn if participants would use assertions without any previous instruction on what they were or how to enter or edit them.  As in Experiment 1, we conducted a controlled study in six smaller sessions using a Windows computer lab.  Subjects from three sessions worked in the WYSIWYT plus assertions environment previously described in section (**find out which section Experiment 1's environment is described) and the subjects from the other three sessions used the same environment but with the addition of Help Me Test (HMT) assertions.
For each group, the instructor led a 25-minute tutorial on the features of Forms/3, introducing only basic language and GUI features, while each subject actively participated by following along and working with several example spreadsheets. No training on testing theory or test adequacy criteria was given to either group. To control for amount of exposure to the environment, both groups of subjects were given identical amounts of training time.
We set up this experiment in a similar manner to the Experiment 1 because we hoped to be able to compare some of our findings.  The design of our experiment was essentially the same.  We asked our subjects to test two spreadsheets to make sure they worked according to their description.  If participants found any errors they were told to fix them.  We used the same spreadsheets as Experiment 1.  To account for user exploration and learning we increased the allotted time for the experimental tasks.
We collected data during the experiment from post-problem questionnaires and from electronic transcripts that captured all of the participants' actions and activities within the Forms/3 environment. At the end of the session both groups answered questions about their understanding and use of assertions.  The HMT assertion subjects completed a few additional questions about their understanding and use of HMT assertions.
Before  the experiment, we conducted a cognitive walkthrough of the tutorial and the tasks we wanted our subjects to perform.  The goal of the cognitive walkthrough was to make sure the tutorial gave our subjects the proper skills in order to complete the experimental tasks.  We stepped through a task to try to figure out what a participant's goal might be, and then determined what actions a participant will doing at each step to  reach his or her goal.  We ran into a few problems with our tool tips for the features of Forms/3, in most cases they weren't descriptive enough and didn't tell the participant what they could do with it.  We decided our tool tips needed to be good enough to teach someone what the features are and how to use them.  We hoped that by changing the tool tips it would be easier for participants to learn though exploration.

In order to iron out the creases in our experiment and find an appropriate cut off time for the experimental tasks we ran a total of four pilot subjects.  One of our pilot subjects was a think-aloud subject.  During a think-aloud study subjects are asked to talk or think out-loud while they do each task.  Usually think-aloud studies are conducted to understand reasons for behaviors.  We gave our pilot subject unlimited time to complete her tasks.  We wanted a think aloud pilot subject because we wanted to know what a typical user would be thinking and interacting with our system.
The other three pilot subjects were not think aloud.  They were given as much time as they needed to finish a task.  We wanted to get timing data from our pilots to determine what a suitable cut off would be for the experimental tasks.  We believed that learning about assertions would take time and we wanted to figure out just how much more time it would take.
Our pilot subjects also helped give us additional insight into our experiment.  As an experienced users of our system, it can be difficult to remember how it was first learning about it.  For example, one of our pilots ran across the assertion number line and asked "this does what for me?" at which point we realized our explanations for this tool was not good enough for people to understand what it was.  We had not expected or really thought about someone opening the number line instead of writing an assertion in the text box.  Our pilot subjects made us reexamine our tool tips to see if they were giving enough information for the user to learn what assertions are and how to use them.

Recall that in designing our experiment we  wanted to keep everything as close as possible to Experiment 1 so we could compare our results.  At one point, we thought we might even be able to use the tutorial from Experiment 1.  But in using their tutorial we would have run into some problems.  First, experiment 1 did not use the Help Me Test feature at all, so we would have had to teach that in the tutorial.  But 35 minutes was already too long for the tutorial,  adding anything else would have just made it longer and we did not want that.  The longer a tutorial gets the lower a participant's attention span is going to be.  The lower the attention span, the lower the retention rate and the less a participant takes away from the tutorial.  This can confound the experiment because someone who does not know what he is doing is not going to be as effective of a measure as someone who took away everything from the tutorial.  Second, Experiment 1's tutorial specifically taught assertions.  The tutorial covered what purpose assertions served, how to edit them, what assertion conflicts and value violations were and what steps a participant could take to get rid of them which could lead to finding an error in the spreadsheet.  But teaching assertions in the tutorial seemed to go against what information we were trying to collect.  We were worried that if we had taught participants what assertions were and how to use them, then  it would be clear to participants that we wanted them to use assertions.
To solve the above problems, we made two  important changes to our tutorial.
First we left out any information or discussion about assertions in our tutorial so that we could be confident that we were not giving our participants too much information or leading them in any way.  Our tutorial covered all the same topics as the tutorial for Experiment 1 with the addition of HMT but minus assertions.
Our second change was to alter our tutorial so its strategy focused more on teaching through exploration.  The strategy for Experiment 1 was to present detailed information to participants about Forms/3; our goal was to guide our participants on their journey though our system by encouraging exploration and discovery of explanation. We hoped that if we encouraged exploration of our system, our participants might stumble into assertions and learn how to use them on their own (in fact, many of the participants did indeed use exploration to discover assertions and learn how to use them).  We encouraged exploration based learning and behavior by teaching subjects to hold their mouse over features in the tutorial and emphasizing that if they needed help and could not remember what something was all they had to do to remind themselves was to hold their mouse over the thing and wait for a helpful pop up message to appear.  Participants were also given three exploration periods during the tutorial in which they were encouraged to learn more.
 For the tutorial material that was explicitly presented by the instructor we used the following techniques.  We followed a demand-driven strategy in providing information; we gave our participants information on a feature only when we thought they would be curious about it and not before.  There were no separate  sections for the discussion of each feature in the tutorial.  We integrated the teaching of each feature into a whole.  If participants were curious about something in our system they could freely act upon that during one of our exploration time periods.
Following from our new tutorial strategy, our tutorial shortened to 25 minutes.   Even more important, with respect to assertions, our new strategy eliminated some of the differences between ours and a real environment by removing the element of the teacher.  The participants only way to learn about assertions was then to dig it out on their own - just as they would have to do in a real environment.
We also realized that our new strategy put extra pressure on our tool tips to perform, since they were the only way for participants to learn what the features of Forms/3 could do for them.  We did not give our participants a reference sheet like in Experiment 1, instead we taught our subjects to use the tool tips in learning about features and if they could not remember what something was they could easily go back to the tool tip for a reminder.  In fact, the tutorial strategy played a critical role in sharpening our vision about the importance of tool tips not only to the tutorial but to the overall approach to assertions.  Our vision shifted from tool tips as ad hoc helpers to tool tips as assessors to an organized integrated explanation system.
about me mentor research journal final report summer fun my roommate and I