My Weekly Journal
What exactly have I been up to?
Before the Official Start Date:
School let out in the beginning of May, I
wasn't supposed to start my mentorship until midway through June, so with
all the time on my hands and really no clue about what I was going to research,
I was panicked. I thought, I have no idea what I'm getting myself
into, I really don't know what I'm doing, why and I doing this? I
have to walk into a project that has been in the works for years and somehow
understand it all. Nuh uh... I said. I'm going to go and meet
with my mentor before I start so that she can give me some papers
to read. This way I can come in with some background about the research
and not feel so left behind and out of the loop. So, I met with Peggy
and she showed me around the building that I would be spending most of
my time in and the computer that I could soon call my "own." I got
my wish. I was given plenty of papers to read so that I could familiarize
myself with the research on Forms/3
that has been done already and Peggy talked to me about it all (but
I was kind of in a "new place" haze and didn't really understand what she
was trying to explain to me until later). I left Corvallis and went
home to Portland a little better off than I was the week earlier.
So, I read the papers on my own time and thought about what questions to
ask the week I started up. Which brings me to week 1. :)
Week 1
I moved in on Sunday and met my roommate Laura.
She's computer science graduate student at Oregon State University and
she's also in the Forms/3
group. She has been very helpful in getting me familiarized with
the project. Thanks Laura! It's great getting to room
with someone who I can talk to about my questions and also get advice about
graduate school. It's like having two mentors for the price of one!
:)
On Monday I was kind of thrown into the research
process. I started Pair Programming with Aaron Wilson (he's a cs
undergraduate student here) working to fix some bugs in the code.
I didn't feel like very much help because there were so many different
Java files created for the research project and I hadn't a clue what they
did). Then we had a Tech meeting to go over the problems with the
code on the Java side (It was getting too messy) and after all of that
Laura helped me get my computer all set up so that I could run Forms/3
on my computer and get acquainted with it. Ok, so then in the mix
with all of that I got registered for an independent study credit,
my student ID, access codes to various places, and my computer account
all set up. So, phew, by the end of the day my brain was dead and
all I wanted to do was go to sleep. Sigh... But Laura took me to
the Library and I got a good book to curl up with before I go to bed.
So, Tuesday was a little better, I'm starting to
get a better feel for things. Laura, sadly, left for Alaska (fun
for her, kinda sucks for me) and I found my way to work ok from our apartment.
Phew! hehe... Today we had a couple of more meetings to discuss
various features of a Help Me Test Button and what exactly it is we want
it to do. Then after work was all over, I went swimming at the pool
to blow off some frustrations and doubts about what I got myself into.
Swimming was a BIG help, and I am able to find my way home again.
Wednesday, I found out that I'm going to be in charge
of designing an Empirical study with subjects in which we will be testing
to see whether users will enter assertions into cells on spreadsheets.
And after work I got lost trying to find my way to Fred Meyers and had
to ask a woman on the street where it was . That's when I found out
I had turned completely the wrong way and had go all the way back and walk
another 10 blocks or so. That's what I remember most about Wednesday.
Thursday, I lead a group meeting about our research
questions and whether or not we could compare the results we hope to get
with a previous study. I was concerned that if we altered the tutorial
too much than we could be inadvertently biasing our subjects into using
or not using the new feature we're testing. I was also concerned
that if we changed the tutorial or the tasks we want our subjects to do
then we might not be able to compare the two studies. In our meeting
everyone discussed these issues and now I have to develop a tutorial and
tasks for the users to complete.
Friday! yeah! Peggy introduced me to
Donut Hour which usually happens during the school year every Friday at
3pm. Only we had home made strawberry shortcake instead! mmm...
much better than a donut. I worked on this web page for awhile and
then Prashant (a graduate student here) walked me through his tutorial
that he gave when he conducted his study. I found that to be of help
since I have to design one of my own. Boy, I really feel like I've
got more than I can chew. I hope that I can do it all.
Week 2
I've been working all week trying to get organized
and fit all of the pieces together so that I could start creating a tutorial
for the Empirical Study. I've been meeting almost everyday with Dr.
Burnett to talk about what belongs in the tutorial and how to organize
it all. My goal this week is to get the tutorial written. We're
testing two groups. One that has the "Help Me Test" feature's assertions
and then one that does not. I've been working on creating the tutorial
for both groups and trying to keep them as similar as possible.
Week 3
What a hectic week. On Monday I had to present
my tutorial to the rest of the group for criticism. I was so nervous
about it. I didn't want all of my hard work over the past two weeks
to get torn apart, but then I did also want the rest of the groups opinion.
I had spent the weekend in Portland with my family and didn't get to finish
the tutorial like I had planned. Instead I came in really early on
Monday to finish it before our 3pm deadline. I was hoping to get
some screen shots of the program but it wasn't working like it was supposed
to. So, Laura spent a good part of the morning trying to help me
figure out what was wrong with the program, but we inadvertently messed
up some of the files and had to get them restored from the backups performed
the night before. The great thing about all of this was that I got
my tutorial to a finishing point and then presented it to the group.
I was expecting to get lots of criticism, but instead I got lots of compliments
about how smooth it was. Yea!!
Wednesday we ran a cognitive walkthrough of the
tutorial and the tasks we wish our subjects to perform. The goal
of the cognitive walkthrough is to make sure the tutorial gives our subjects
the proper skills in order to complete the tasks we give them. We step
through a task to try to figure out what a user's goals could be, and to
try to figure out what user's will be doing at each step. We ran
into a few problems with our tool tips for the features of Forms/3, in
most cases they weren't descriptive enough and didn't tell the user what
they could do with it. So, we hoped that by changing the tool tips
it would be easier for users to learn though exploration. We designed
the tutorial to encourage exploration. Since we want to find out
if user will enter assertions on their own, we didn't want to teach them
how to use them. We wanted to simulate (in a sense) a real life environment
where someone is given a product and asked to use it. Therefore,
our tool tips needed to be good enough to teach someone what the features
were and how to use them.
Week 4
Orion fixed the tool tips from last week and we
ran though three pilot subjects to debug our experiment to make sure we
have everything running properly before next week. We found out from
our pilot subjects that our tool tips still weren't good enough.
They seemed to show interest in assertions but they didn't know what the
syntax was or what good assertions were to them. So we made a tool
tip fix to hopefully fix that problem. We also seemed to have a problem
with the the cursor disappearing when users tried to edit a formula.
This frustrated our pilots to no end. At this point, we're not sure
why this is happening except that maybe it has to do with the version of
jdk we're running on the machines.
Week 5
Experiment Week:
Since we made some changes to the interface we wanted
to have one more pilot subject test it out before the experiment.
Our pilot seemed to understand the purpose of assertions and how to enter
them so the tool tip appears to have worked. We also noticed that
that the Help Panel was not showing up on our spreadsheet GUI during our
pilot session today. We talked about it and came up the the idea
that maybe it was the version of jdk that we were using since we
were compiling in 1.2.1, but we were running it on newer version.
So for the experiment we decided to try running it using the 1.2.1 version.
This seemed to solve the problem we were having with the cursor disappearing,
but it didn't solve the problem with the Help panel. So we decided
that we could just say something about the bug in our tutorial so that
users wouldn't worry and they would know that the Help was still working
the way it should. So, we should be about ready for the experiment.
On Wednesday, I was rushing around trying to get
everything ready for the experiment. I made hundreds of copies and
stapled lots of packets together with the help of Orion.
We also checked out the projector to make sure we could set it up and make
sure it was running properly. I didn't want any surprises the morning
of the experiment.
Thursday was the longest feeling day I've had in
a long time! We scheduled three experiment sessions for Thursday
10am, 1pm, and 4pm and one on Friday at 11am. Since I had gotten
everything organized the day before I wasn't too stressed about getting
to work really early to set up. I made it to work by 9am, grabbed
everything that I needed off my desk and left to go set up the computers
in the experiment room. Laura and Orion helped me launch the
proper policy file on all of the computers (so that users could only use
the features we want to have available to them) and load all of the
spreadsheets for the tutorial and the experimental tasks. In the
first group only 4 people showed up! 50% less people than we expected,
but in the end the numbers all seemed to work out. I was a little
bit disappointed that more people didn't show up since I had worked hard
at communicating with our subjects about times and places. I guess
it's hard for some people to get out of bed that early in the morning.
<shrug> Aside from that, everything ran smoothly for the rest of the
sessions.
After the session on Friday, we had a meeting with
Dr. Burnett to discuss how the experiment went and the paper. We're
trying to write an experiment paper that we hope will be accepted into
the International Conference
on Software Engineering (ICSE).
Week 6
This week I've been working on organizing the data
into spreadsheets so that we can then transfer the data we have into a
statistical analysis program. From there we can analyze the data
and hope we get the results we want.
Dr. Cook helped me analyze the first round of results.
We had hoped that we would get statistical significance between our two
different groups regarding the correctness of their spreadsheets.
We did not, however, achieve the results we were looking for. Our
next step is to look at a different cut off point. We would like
to compare our study with one previously conducted, but their data used
a different time limit, so we're going to cut off our data at their time
limit to see if we get any different results.
Week 7
I started writing out the experiment section for the ICSE paper and
I've been looking for articles for the related work section of the CHI
paper. We were looking for articles that had to do with shocking
or surprising users, or work in visual programming that related to assertions.
I've also been working with Orion analyzing data in order to find something
of significance. Dr. Burnett and I have also had a good talk about
graduate schools and I've spent some time looking for information on the
web about graduate programs in computer science.
Week 8
This has been a wild week full of ups and downs.
Orion and I have been frantically trying to find something, anything in
our data that is a significant finding. So far we've found that 98-99
percent of the time if users choose to enter assertions on their cells
then they are correct. This is a good result because in the previous
study users did not have a choice to enter assertions, they were already
put on there (for simplicity sake) by computer scientists. So, our
result shows that user's are entering good assertions. We also
found that in comparing the correctness of the two experimental spreadsheets
with our subjects who used assertions verses the previous experiment's
no assertions group, our subject's correctness was significantly better
than the previous study's subjects on the Grades spreadsheet. In
one sense, this is a very good result, but on the other hand we were unable
to get significance on the other spreadsheet problem, Weekly_Pay, so it
is not as strong as a result. One of the reasons we think we might
not have gotten this result is that the Weekly_Pay problem was not as difficult
as the Grades problem and thus the no assertions folks were able to find
most of the bugs without the help of assertions.
We also found out that subjects are still using
their first task to explore the different features of Forms/3 and are not
scoring as high on their first task than the second task in correcting
the spreadsheet errors. Since there were different number of
bugs in each of the spreadsheets we first had to normalize the correctness
scores by taking the number found over the number bugs possible and finding
the percentage. We could then legitimately compare the correctness
scores of Task 1 and Task 2.
We also found that those people who use assertions on Task 2 perform
significantly better than those subjects from the previous experiment who
did not have assertions.
This is a good result because it shows that people who use assertions,
use them correctly and are able to find more bugs than people who don't
use them, however, we only had 10 people doing the Weekly_Pay spreadsheet
first whereas we had 20 people who did the Grades problem first so we run
into some validity problems in comparing Task 1 against Task 2 .
Our numbers were a little unbalanced and so we started to seriously think
about conducting another study to get more subjects in order to strengthen
our claims.
It's crazy, I guess I'm learning first hand that
sometimes experiments don't always work the way you had intended and that
you may not always get the results you're expecting. I'm learning
that when you design an experiment you should try to create it in a way
so that you'll always get something useful out of it. No matter what
kind of results you get. Dr. Cook, Dr. Burnett, Orion and I all talked
about the results we have and debated whether running another study to
get some more subjects would be worth it. In the end, we decided
that in order to build credibility to some of our findings we really do
need to run a few more subjects. So, even though this is the 8th
week of our study, and we really should be winding down and trying to write
up our results and findings, instead we're going to run two more study
sessions.
Week 9
We decided that Tuesday would be a good day to run another couple of
sessions for the experiment. We were hoping for at least 10
subjects to show up, but given that it was really last minute and summer
classes were letting out, getting 5 wasn't horrible. We had hoped
that with the data from these new subjects we would be able to support
and validate our previous results. Support is not what we got; instead,
we lost all of our important significant findings so our experiment now
is a little worthless. We did learn what worked and what didn't so
that if they want to run the experiment again they can make improvements
and hopefully those improvements will allow for significance.
After we looked at the new data, I started working heavily on the experiment
write up. Dr. Burnett decided, as a result of our lack of findings,
to turn our experiment write up into a tech report. I suggested that
since we based our experiment on a previous one, that we also include the
write up for that experiment in the tech report. So, now I've got
more to organize.
Week 10
I've been trying to finish my write up for the tech report so that
I can leave without having anything hang over my head. I will likely
make minor changes once I'm at school again and the tech report is more
complete. I've also been busy this week trying to finish up my web
page. That's when I realized I didn't have many pictures from the
summer so Laura and I drove around the city and took some pictures.
I've got some of them up my web page under the sections, summer fun and
My roommate and I. Check them out; the links are on the bottom of
the page. :)
Laura and I also had a party for everyone in our office over at our
house so that we could have one final get together before I leave.
Friday I went to Marc's masters thesis defense. Marc is also
part of the Forms/3 group. I thought it was wonderful to be able
to go to his thesis defense because it gave me and idea of what I could
be going through in a couple of years.