My final week was entirely devoted to organizing all my data from the rest of the summer. 
     First, I worked on the spreadsheet of the randomly sampled changes and their analyses. It looks great and very organized! Professor Lu spent some time reviewing it and sent me a few clarification questions so I worked on finalizing that in my last couple days in Chicago!
     I conducted a final collection of # condition variable routines in head revision vs. # changes per condition variable in development history. Professor Lu and I discussed that despite there being no global condition variables in the project, it will still be helpful to get an idea of how often the more commonly used and less commonly used condition variables are changed in development history. This study ended up yielding pretty predictable results, but it's still good to have! 
     We also came up with the idea to check how many new condition variable lines are added when large chunks of new functionality are added in development, so I wrote a script to collect this information. It turns out, not a ton of new functionality involving condition variable routines was added, but of the few that I found they yielded pretty consistent data suggesting that well-written new functionality code should not require more than 2 or 3 condition variable routines. However, this might be less applicable to projects with less abstraction and self-defined data structures. Now that I'm wrapping up my work on Apache I'm beginning to wish I had had time to do a similar analysis of a second project to see how many of my findings are unique to Apache (which I imagine is a very uniquely written project) and how many are pretty universal. I will definitely be interested to see what Professor Lu's next paper says in terms of this.
     After finalizing and organizing all of those results, I began writing my final paper in the last two days at UChicago. Unfortunately, I will have to complete this at home but hopefully it'll only take another couple days!

     Week nine has been the heaviest week for manual analysis. Although certain types of changes, such as stylistic refactoring, are easy to pinpoint and classify, there are certain categories of change that are quite hard to detect and require a lot of rooting through different versions of code. For example, the changes due to race conditions or other under-synchronization are difficult to classify because they often require me to sort through the entire rest of the file outside of the diff content and determine where the blocking or signaling thread that's causing the race condition is located. Since we want this analysis of the 60 randomly sampled changes to be very comprehensive, I am spending a lot of time on writing detailed explanations so that Professor Lu can refer to them as her lab moves into writing condition variable synchronization algorithms in the future. 
     Something I've found interesting in doing my analysis has been the relatively high number of changes related to adding abstraction and increasing code reusability. It appears that more changes are related to following good coding practices than correcting actual correctness problems. I believe this constant effort to adhere to good development practice might account for the relatively low number of correctness issues encountered (at least in regards to condition variable synchronization). I am going to write about this a little bit in my final paper, because I think it'll be interesting to compare Apache to another project which maybe does not rely so heavily on abstraction and special reusable structures. I would be curious to see if another project maybe requires more correctness fixes if they do less refactoring for performance and abstraction and code reusability. 
    At the end of the week, I began writing up the spreadsheet of the findings of my manual study. It's hard to organize this into something readable but containing enough information about each individual change. Since Professor Lu will be using my results in her future papers to justify the decisions her lab will make when creating certain multithreaded development tools, I would like to make sure my findings are as clearly explained and organized as possible, but this is proving challenging. She has met with me once this week to review what I've come up with so far and she has made some helpful suggestions so in the next week I hope to come up with final results that follow all her specifications and will be very helpful in the future. 

    Throughout my progress in manually classifying the changes sampled by my script, I've met a few setbacks. Having the opportunity to take in-depth looks at the new and old version code as well as the corresponding DIFF content for many revisions allowed me to evaluate the way my original analysis script was classifying certain things. For example, it refrained from determining a certain line to be a MOVE if that line was added and subtracted in multiple areas throughout the DIFF content (for one revision). Although in general this was a pretty good idea since certain lines were added and subtracted too frequently to be sure of where the actual intent was to move them, I realized that the DIFF content is conveniently divided into sections of code change (individual change blocks), which should be a more accurate limiting area for an upper limit on adds/subtracts within it. So, instead of considering something an ADD+REMOVE instead of a MOVE if it appears added and removed multiple times within the entire DIFF content, I do this now only if it is added and removed multiple times within one change block. So far this has helped tighten up the classifications a bit, but figuring out a better system for this was a bit of a setback since I didn't want my final graphical data to have a falsely large number of adds and removes and hardly any moves (which was previously the case and HAD been some cause for concern). 
    After figuring out a better system for that, I continued to perform manual analysis on the different changes my script had found. I found unfortunately that a majority of the changes the script has found are refactoring changes. Around revision 91,000 it appears that a new API is introduced to replace the POSIX pthread_cond library. I already knew about the existence of this API (it's the APR_cond API I mentioned in the beginning weeks) however, this change from one condition variable API to another created a sudden influx of removes and adds that really hardly count as adds or removes but more as replaces. However, since the APIs are slightly different and take different parameters, considering them REPLACES is kind of iffy. So, I have a meeting scheduled at the beginning of next week with Professor Lu where we will speak about this trouble.
    Besides this problem, the other type of change I've found very commonly is small moves to avoid race conditions. For example, often a signal will be moved from after a mutex unlock to before the mutex unlock, which makes sense since it's generally good practice to hold a lock when signalling on a condition. So any of the changed I've found that actually seem to be debugging changes are making sense to me-- there just unfortunately don't seem to be as many as we had hoped in comparison to refactoring changes!

            This week my primary focus has been accumulating some numerical data using the ouput of the main analysis script. Of course, the data I have gathered this week is pretty preliminary because I haven't performed extensive manual analysis yet to break the overall categories of change down into subcategories determined by the (perceived) motivation for the changes. For example, when evaluating the amount of general additions and subtractions of synchronization routines over the course of development, we may not find this information quite as useful as how many of these additions/subtractions were intended to solve race conditions vs. how many were simply refactoring changes. I began this manual evaluation stage this week as well but will continue that further into the following week. The graphs below are the results of my script's analysis:

     As it is clear to see, throughout the development process, the number of removals went down and the number of synchronization additions went up substantially. I imagine that as the number of servers increases over time, the need for synchronization would increase. I am going to spend the rest of my time here primarily performing manual analysis to determine what the increase in synchronization correlates to as well as what the many possible motivations for such changes would be. So far, I have created some scripts to randomly sample the changes my original script has found so that I don't necessarily have to manually analyze hundreds of changes (we don't have that kind of time left). I've begun reading the LOG files and determining subcategories of change motivation for the randomly sampled changes. I plan to continue this throughout next week and continue meeting with Professor Lu to discuss what I'm finding.  

   This week, Professor Lu and I discussed why the script for tracking changes for specific condition variables didn't work out. I realized that Apache utilizes their own structures that contain condition variables as attributes and therefore what seemed to be the same condition variable (queue->not_full for example) is really a different variable depending on which queue is being referenced (queue is a parameter passed into a function and it is of a certain structure with the condition variable not_full as an attribute. Since almost all condition variables used in Apache are attributes of some structure, the script I ran which counted the amount of changes made to each condition variable yielded essentially useless results. It took a while of analysis to determine why no global condition variables were being initialized anywhere throughout the Apache trunk. 
    Now that we've determined this, I am planning to search for places where these structures are initialized and see if maybe any of these are global, which would at least give us a good ballpark estimate of how many different condition variables are used and how many are being affected by certain synchronization-related changes. 
    Despite this setback, I've been doing the following to try to get some data:
    - tracking the different structures whose condition variable attributes are being referred to. This won't give us the data we had initially hoped for about the number of changes made to a condition variable's synchronization routines correlating to the number of overall synchronization routines involving that variable but it could be interesting and we'd hope for pretty similar results despite the variables not being global.
     - writing a script that tracks initialization of the different structures containing condition variables as attributes. As I stated above, it'll give us some idea as to how many different variables are being used at any given time.

     This week I've been working primarily on speeding up the analysis script. Since the initial version of the code was used to examine critical section change, it checks out the old version of the code as well as the diff content and runs extensive analysis on both of these. Professor Lu and I determined that such an extensive code reconstruction wouldn't be necessary but removing this forced me to make a lot of changes to the structure of the rest of the script.
     I came up with the idea to write a separate script that determines which revision numbers contain synchronization-related content and then use this output as input for the analysis script. This way, the script doesn't waste time reconstructing any irrelevant code. This script took a couple days to write because I wanted to be extremely thorough in checking for all the synchronization-related commands (since the script takes a long time to run on even the faster server, I don't want to have to run it more times than necessary). 
    Besides major refactoring to improve performance, I also reworked the script a bit after analyzing the output it was having. It counted many lines as MOVES because certain lines were removed and then added back in the same spot just in different context (i.e. the code surrounding certain lines was changed and the developer just happened to choose to retype that line instead of working around it). While technically the line was moved, I decided this should be counted as a separate change category. However, it was surprisingly difficult to pinpoint this type of change via the script so it took a while to get this worked out. 

        TASKS I'VE COMPLETED THIS WEEK:
                - developed algorithm in script to find entire blocks of added/removed content containing condition variable routines, then compare them to each other to distinguish MOVEs from ADD/REMOVE pairs. This is useful because in certain revisions, there are lines which are added and removed in multiple locations so if we have entire blocks of code (more than a single consecutive line) added in one place and removed in another, we know this most likely can be classified as a move. I also wrote the part of the script that classifies single lines which are added and removed as MOVEs only if the single line is just added once and just removed once. Otherwise, it is too ambiguous to be classified as a MOVE and these cases will just require manual evaluation.
                - wrote a separate script that tracks the number of references made to every single condition variable in the HEAD revision of the Apache server project, i.e. if the line apr_cond_signal(this_cond_var, this_mutex) is encountered, this would count as a reference to this_cond_var. The purpose of this script's findings is to help us later check whether the more commonly used condition variables are changed more in development. We (obviously) predict that this should be the case.
                - the findings of the above script were a bit surprising because I found that there are only 19 condition variables, with 5 of them actually being pointers (so really we only have 14 actual global condition variables) with a total of 44 uses throughout the project. I believe this seems a bit light so I've looked back over the script, which seems to be correct, and considered whether I had missed any condition variable routine APIs in my initial surveying of the Apache Portability Runtime library but I don't see anything. Although these results are surprising, I have noticed in the manual analysis of the code repository I've done so far that Apache's source code is organized such that all handling of the request and response queues are conducted in only three different directories and not spread over many files. Since this is the part of the server that primarily uses the signal/wait routines, I can see how it's possible that not many condition variables would be initialized. The next step Professor Lu and I have planned is for me to write a script that searches explicitly for the initialization of global condition variables just to check the findings of this other script.
                - Unfortunately, my main script tracking ADDs/REMOVEs/MOVEs/MODIFs has still not managed to run the entire way through the ~2 million revisions without encountering errors, so I've been continuing to address these as they come up. Since it often gets through ~50,000 revisions before encountering an error, addressing them all is a slow process but today I put a try/except block around the call to my script so that instead of stopping its progress it will move on and just print the error for me to address later. This way I can go address all the remaining errors at once and have some results to analyze in the meantime (not sure why I didn't think of this fix earlier, but now it'll be just a couple days until I have all the results and errors and can really get close to finishing this initial version of the script)!

     Upon our arrival at UChicago, Michelle (fellow DREU intern with Professor Lu) and I met with Professor Lu and her PhD students for lunch, then met with the tech staff and administrative staff in the CS department to get our CS accounts and UChicago IDs set up. Our first meeting in which we talked about our project was Wednesday, June 17th. 

     In the meeting, Professor Lu spoke to us about the overall goals of the project (which we were already pretty familiar with due to having spoken to her before arriving), and we got into some detail about multi-threaded software. More about the project goals below! As neither Michelle nor I have ever written concurrent software before, Professor Lu explained the basics to us and assigned us a series of readings in an online textbook (along with practice code to implement) so that we could further familiarize ourselves with writing concurrent programs. We spent the rest of the week (through Sunday) working on the textbook examples and exercises so that when parsing and analyzing the source code revisions for our subject applications, we will be able to determine the effects of different critical section changes. It was helpful that Professor Lu assigned us text with lots of code we could implement because writing my own code and moving around the lock/unlock/signal/wait commands to see the effects of different practices allowed me to get a really good understanding of the types of changes we'll be seeing in our case study on Apache and Mozilla (as described below).

     Project Goals: Michelle and I will each take a subject multi-threaded software project (Michelle will take Mozilla and I will take the Apache HTTP Server Project) and create scripts as well as perform manual analysis to determine what types of changes pertaining particularly to signaling/waiting on condition variables are most prevalent and cause developers the most trouble in creating multi-threaded software. Obviously both these software projects are highly multi-threaded so they lend themselves well to such a study. They are also open-source so we can access all changes in the projects' version control repositories (SVN in my case). Ultimately we hope to gain some insight about what challenges developers face when writing multi-threaded software so that future researchers might be able to develop automatic algorithms and tools to assist with adding synchronization to software. 

     I completed my background research on concurrent programming and concurrency bugs on Sunday, June 21st and that night I began reviewing the APIs of some of Apache's concurrency-related modules. My goal here was to familiarize myself with the wrapper functions Apache developers would use to implement locking, signaling, and waiting procedures whenever they don't directly use the POSIX library for this. This proved to be a good idea, because in our next meeting with Professor Lu (the following day), when I told her about my findings, she confirmed that another undergraduate student had written a similar script to analyze MySQL but had not considered the possibility of wrapper functions for synchronization procedures, which set that student back a bit in her progress.