My first day here began with a project-overview presentation by the graduate student mentors (Sreedevi Sampath, Sara Sprenkle, and Emily Gibson) who are heading up the project that Holly and I will be working on over the summer. We were given a glimpse of the project as a whole as well as an idea of what our part will be in it. Holly and I will primarily be responsible for developing an open-source tool that will fit into the Web Application Testing framework that Sreedevi, Sara, and Emily have been working to develop. This tool will be a visual interface that will allow a user to navigate through the expected and actual HTML results returned from the execution of logged user sessions on a web application.

Our tasks for the week were as follows:

1.) Create a web-based mock-up of the tool to simulate general navigation without functionality.
2.) Develop back-end Java classes to handle file retrieval and differencing.

Challenges:

1.) I have had to get myself up to speed on HTML, JSPs, and Javascript. Fortunately, HTML is fairly simple and my knowledge of Java helped me sort out JSPs and Javascript without much trouble. I have now developed a pretty little mock-up.
2.) We are having difficulty with our file-handling. We need to be able to display files off the local hard-drive within frames in the HTML. However, all the convenient tools which JSPs offer to include outside pages seem to only want to look for pages on the local web server or on some foreign web server (with a complete url).

We have resolved our issues with file-handling. We can now access files from the local hard drive by utilizing Java to read lines from a File object and print them out within an HTML page for viewing within a frame. With this challenge out of the way we were able to improve our mock-up by doing real-time analysis and display of two local HTML files when the tool is initialized.

Other Challenges:

1.) Ensuring platform independence. Right now we are depending on the use of the UNIX command "diff". Holly has been looking into the possibility of creating our own differencing class to ensure platform independence. It turns out that the problem is quite a tricky one. That issue may be put on the back-burner for a while.
2.) Integrating a directory structure. The mock-up was not designed to allow for navigation through a directory structure of test cases and requests. Sara Sprenkle provided Holly and I with an actual directory that had been generated based off of the logged requests to an actual web application. I am in the process of creating a way to iterate through individual requests, and test cases (groups of requests).
3.) Holly has been working on a more flexible form of navigation - something akin to a browse function which will allow the user to visually navigate through the directory structure. She is running in to difficulties because she needs to use a Java applet but Mozilla Firefox (in which we are operating our web-based tool) is not recognizing the installed applet plugin.

This week did not see a lot of progress, due to intervening events, but the progress we did make was good. Both Holly and I got around a few of our problems by coming to a better understanding of how servlets can be used in the background of a web application to do the "heavy lifting." I have now successfully integrated the directory structure of test cases into our application, by communicating with a helper Java class through a Java servlet. Holly is also utilizing a servlet to organize the dynamic construction of a browser window which displays the levels of the directory and allows the user to navigate to any test case in the directory.

Other Challenges:

1.) Creating reports that summarize the results of the test suite. Decisions that will have to be made include: Should we generate reports test case by test case or test suite by test suite (a major difference in time to generate)? How should we organize the data in the reports? What resources should we assume are available to help gather the data for the report?

This week also saw a lot of time away from our computers. Emily Gibson is preparing for a talk she is given at a conference. Holly and I sat in on two practice runs to provide input on possible points of confusion and slide problems. We also had a group meeting with our three grad mentors and Lori about the progress we were making on our project. On Tuesday almost everyone from our lab sat in on Sreedevi Sampath's dissertation defense. I found that to be a very profitable experience. It was nice to see what that actually might look like. Sreedevi passed of course, then we went out to celebrate with her at a local restaurant.

The biggest reason why this week did not see more progress is that on Thursday the power went down. It was down all across campus for about half an hour. But even when it came up everywhere else, half of our lab was still dead. Holly and I got an impromptu day off because we had nowhere to work from.

The visualization tool as of Friday was in pretty good shape. I integrated the new directory browser into the main page by setting up a browsing option. If the user wishes to have the directory browser open, the other contents of the screen will shift to the right and the browse window will be displayed. However, if the user wants to have a larger view of the HTML pages being compared and does not wish to browse, they can also close the browse window and the contents of the page will shift back in to place. When first entering the main screen of the tool we chose to not have the browser window open because the contents of that window do not always generate quickly enough. By keeping the window hidden at first we ensure that when the user opens up the browser window they will actually have something to see.

Another new feature was a pane at the bottom of the screen which lists the actual URL requests that resulted in the HTML pages being displayed. So the user can not only see the comparison between pages, but can also keep track of the calls that returned those pages.

A significant amount of my time in the fourth week was spent reorganizing our code. I also need to spend a good bit of time doing Java documentation on the code that we have now. I have been very remiss in doing that. It will be a rather time consuming task.

Other Challenges:

1.) Creating reports that summarize the results of the test suite. Holly is currently working on this problem.
2.) Making the tool more portable. Right now our biggest problem is that we are running a few Perl scripts from within a Java class, but we're really using the Java to execute the Perl scripts on the command line. Because the scripts are running on the command line we have had to hard code the absolute path, so the tool is only working on my machine at the moment. I have been looking into this issue. There are ways of calling Perl scripts directly from Java (without utilizing the command line), but they seem to be a bit complex and my ignorance of Perl makes it difficult to unravel the examples that I have been looking at.
3.) Cleaning up. There are lots of little loose ends left to be dealt with. Holly and I are hoping to deliver a very usable product by next week at the latest so that we can move on to the next phase of our project: writing a tool paper and doing some independent research into better oracles for error-detection when comparing expected and actual HTML results.

This past week was very short, due to the 4th of July holiday. Holly came in on both the 3rd and 4th, but I was out of town visiting my brother, so I will now have to catch up some time.

I spent Wednesday, Thursday, and Friday doing a lot of tidying up. There were quite a few cosmetic, structural, and logical holes left to be sealed. I kept running through the different functionalities of the tool, finding discrepancies and errors as I combined different actions in different ways. It's not a glamorous job, but it has to be done. I have taken on most of these little tasks because Holly is working on our last big problem: getting the error report option in working order.

I say that the error report is our last big problem, but there are also two other significant issues to be mentioned:

1.) The tool is still not as portable as we would like. We are using a context-variable in the web.xml document to stipulate where the Perl scripts can be found because we have not yet discovered a good way of making Java and Perl get along without utilizing the UNIX command line. Also, we have been using the UNIX diff command which is, obviously, platform dependent.
2.) Our use of the Eclipse development environment has caused some problems with our dynamically generated jsps. Some of the actions taken within the tool cause new versions of a jsp to be generated, but we often find that old version of the jsp continue to be loaded within the web application. One of our grad mentors, Sara, just recently realized that this is because Eclipse is not always copying the right files over to the web server at the right time. We are hopeful that this problem will be an easy, though vital, fix.

We have come to a satisfactory stopping point in our development of the tool. The needed functionality (along with some "nice-to-have" functionality is in place) and the bugs seem to have been worked out. There are surely more bugs to be found, but Holly and I expect to uncover those problems while actually using the tool for its intended purpose: viewing/analyzing the expected and actual results from the replay of user sessions on a test application (DSpace - electronic publications web app). By doing so we hope to not only strengthen our tool by uncovering quirks but also to gain new insight into more sophisticated oracles (algorithms for comparing the expected and actual results and locating potential faults).

While Holly and I hope to contribute to the development of a better oracle before our summer tenure is over, we must also devote time to writing a tool paper which describes and evaluates our visualization tool. We have hopes of submitting this paper to a conference with a fall deadline. We also hope that any work done on a new oracle could be the foundation of another paper, but we'll see how that goes.

The technical difficulties of this past week dealt mostly with transitioning our project from running within the Eclipse development environment, to running independently on Resin. We also had to deal with a performance issue when dynamically generating the browse window for navigation within the test suite. Holly re-organized the algorithms responsible for this maneuver and things are running much more smoothly now. In short, it works!

This week was spent analyzing data from replayed user sessions and brain-storming ideas for better oracles. So far most of our ideas are along the lines of decreasing the number of false-positives: detected "possible faults" that are in fact not faults. Holly and I have each developed some ideas and have begun to implement them. Hopefully, we will be able to run some sort of case study to compare the performance of our refined oracles with the original oracles in terms of false positives and false negatives.

While Holly was working on her improved oracle she discovered a problem with one of the original oracles: struct++. This algorithm utilizes a Java archive that was acquired from a third-party, but it turns out that the archive itself (which has classes that are designed to parse HTML documents) has an error. Holly is looking in to converting the algorithm from Java to Perl.

I too am dabbling with Perl in order to implement my new oracle. I first tried my ideas out in Java, but the running time was unacceptably slow. Having never written in Perl, it took a little while to figure things out, but I think I'm on the right track now.

Holly and I both wrapped up our new oracles this week. The heavy-lifting of each was done with Perl, but Holly's required a Java front-end to do a little pre-processing before the parse. These two oracles have been integrated into our tool. The integration process wasn't too hairy, but it does bring up the question of extensibility. It's very possible to add and take away oracles, but it can't be done simply at the drop of the hat. Adding a new oracle to the main functionality of the tool (visualization, navigation) only takes about ten minutes. However, there is the dynamically generated fault report to be considered. There are also two supporting documents for the tool which, presumably, should be generated before the tool is ever used for a particular test suite. These two documents fuel the fault report and allow requests in the navigation window to be highlighted if a potential fault has been detected in them. Every time a new oracle is added, these two files would have to be re-generated by running the test suite again.

Aside from producing a tool paper, Holly and I will also be presenting a poster on our tool at the Summer Undergraduate Research Symposium here at the University of Delaware. Work has begun on that, though I don't think it will take a terribly long time.

There is also a possibility of a second paper on our new oracles. In order to do this we will need to run several test suites through the various oracles and compare their results. We already have seen an improvement in the number of false positives avoided, but we don't know how well our new oracles will perform in terms of actual fault detection. Will they miss things that the other oracles picked up?

Our tool has grown up to this point in such an organic way, that it is very easy to look back and think, "Now why didn't I design it that way from the beginning?" One of my biggest jobs on this project has been putting all the pieces together. I designed the front-end and the interface between the front-end and the back-end. I also helped put some of the back-end together, though I think Holly really deserves more of the credit in that area. So, when it came time to start adding the oracles way back near the beginning of this project, I did the work of automating the oracle execution in tandem with the initialization of the tool and setup of the ability to navigate from oracle to oracle. What I didn't think of at the time was how easy it would be to add new oracles to my system. What I realized later was that, although it didn't take too terribly long to add a new oracle, you needed to change a lot of little things in a lot of different places to get it right. This wasn't difficult for me, since I designed the system and knew where everything was. But it wasn't going to be very fun for anyone else.

So, this brings me to my accomplishment of the week: greater ease of extensibility. The addition of a new oracle now only requires that a single line be added to a configuration file. This line includes information about the name of the oracle, the parameters it takes, how its button should be labeled, etc. Once this information has been added to the config file, the tool does the rest. The Rendered and Raw versions of the HTML documents will always be shown, but you can now add up to five other oracles to the tool. The only stipulation is that they must be written either in Java or Perl.

The rest of my time this week was spent on other minor improvements and creating our poster for the symposium. Holly and I have split up different sections of the poster to work on. What it essentially breaks down to is Holly is in charge of tool features and I am in charge of everything else (intro, background, evaluation). This division will probably roll over into the work on our paper as well.

On Monday and Tuesday Holly and I finished up the poster. We each had a copy printed for ourselves. It's big, it's blue, it's beautiful. Wednesday was the symposium. It went very well. I will always be grateful to my four years of competitive speech & debate in high school - it makes thinking on your feet a lot easier. Not that either Holly or I got grilled by anybody, but we did have a few people stop by with good questions.

The rest of the week, which only lasted through Thursday, was spent writing the paper. We were both hoping to completely finish the first draft by the time we left lab that afternoon, but after staying a little later than usual it still wasn't quite ready. This was partly due to my having to fight with Latex, which I have never used before. Oh well, it will get done.

It seems odd that the summer is over. It has all gone by very quickly. I feel very fortunate to have had this experience - not only because of the work, but because of the people. I think I'm going to miss Delaware. = )