To be clear, Tuesdays and Thursdays are the two days where project leads are expected to work with the whole team for the day. Likewise, Monday, Wednesday, Fridays are marked as individual work times, but my group usually works together quite a bit to make sure we're all on the same page with how to move forward, and historically we've had a lot of tutorials and presentations on these days. I'm not sure if I've ever mentioned that before, but maybe if you've been keeping up with this blog you've already noticed the pattern. Anyway, since today was a Thursday Anat was with us for most of the day and Valentina was there too, along with all the regulars. 

     After lots and lots of intense discussions this week, today was more of a work day where we went about on different tasks trying to implement some of the code and strategies we had talked about. Of course, as soon as you fingers hit the keyboard you realize other major flaws in the plan that we had not discussed yet, and then we would have to talk about how we were going to handle it, but for the most part we did actually get a good chance to make some coding and programmed progress. First off, we made some real headway with the routing formula that Frank, Kivan, and Rohan were checking out in different facets. Frank had some great ideas about creating a matrix with all the routing distances we would need to use so that we would only need to put in one request to OSM, which we hypothesize is going to be the step that takes the most time. Time is critical in our program because we want it to run in real time for the dispatchers to be able to reroute requests. This is why we've been working as hard as we can at each step to steer clear of computationally expensive solutions or algorithms that would take too long to output an answer. Right off the bat what comes to mind is how after talking with Anat, Kristen presented a great idea about switching the order of the feasibility steps so that the capacity check would come before the rerouting checks, since we believe the routing steps will be more computationally expensive and take longer to process. Kristen and I discussed it for quite some time talking about how it would need to be structured to work with the rest, and how we would need to change the other two routing checks (pickup and drop off insertions) to comply with this new code.

     Our morning started off today with a presentation by Tracy Hilliard, who is the Director of Data Integrity for the City of Seattle's Human Services department. She talked to us about public data and what the city of Seattle is doing to provide more public data and publish it on the internet. It was interesting to hear her talk, but I felt like I didn't learn anything novel from her presentation, and while the content seemed like it would be greatly connected to what we were doing (using data about public transit versus talking about public data from the city) there wasn't much I gleaned from it that I saw as directly relevant to my work. After her presentation we took a group photo. See if you can find me! (Hint: look to the right of the red shirt.)

    Before our next session, my team regrouped and talked about what we had come up with as an outline of pseudo code yesterday, and we kept going trying to fill out more of the outline, while at each step addressing any concerns that arose. The really difficult part of outlining what our algorithm is going to look like is that the pieces are all very inter-related and to work well together we all have to have a really good understanding of what each part does and how it fits into the whole. 
    Our team session was interrupted shortly though by the next presentation that was being put on by Google Cloud Services. This presentation also didn't seem exactly relevant to what we were doing, and I'm not sure that we'll use their services at all in what we're doing, but I liked their presentation a lot more because it was interesting for me to hear about all the services that they have to offer. I believe it is worthwhile to have these presentations just because they make you more cognizant of all the things out there that you can use and what different things you can do with them. However, they are always more memorable when you can relate them to a specific project you're already working on at the time. 
    Anyhow, we spent the rest of the afternoon working as a group to work out all the kinks in our plan. Written down, this makes it sound like we didn't really do that much because really how much can you possibly need to discuss a simple outline? Let me assure you though, that this is no easy feat and is critically important in moving forward with the project. As I mentioned before, the different pieces of the overall algorithm have to work seamlessly together, and therefore they must all be very well defined. Also, another big issue that we keep running across is the trade-off between greatest optimization to the system and creating something that is very computationally expensive. We have to keep in mind that this program is going to be expected to run rather quickly so that the changes it suggests to the schedule can be changed and implemented right away. In addition, we kept running across dilemmas about what the end product would really need to look like for the dispatcher, and how they would be giving us information, and how it would be reflected in the data we would be given to work with. Due to this, we kept a list of questions for Matthew going as we discussed everything, and we are hoping to have another chat with him tomorrow so he can clear up some of these details that we're still unsure about.
    I think the best way to characterize today, this week, and just our project in general is to look at that moment today when Anissa, the eScience ethnographer, after having been watching us for at least an hour discuss different aspects of the outline commented, "Wow, your project really is very complicated," and when I agreed with her she went on to say that it was much more complex than it first seems when you abstract the problem and just say that we're trying to optimize routing. It's true; whenever I talk to others about our problem I feel like it sounds very simple in concept, but what they don't realize is just how limiting and complicated all the constraints we have for this particular system can be. It's as if every time you look at it and go, "OK! I understand, and I know what we have to do!" someone else interjects and says, "Right, but have you considered how x will affect y?" and you realize you forgot that small detail of x, which only comes up in the situation with y, and you're back to the drawing board again. 

     Today was a bit....trying. We spent pretty much the entire day as a group discussing and trying to figure out how to move forward with the optimization problem, which in the end we decided to change into a greedy search problem instead. After spending a large amount of time trying to get everyone on the same page with the dynamic programming concept for optimization and defining state variables in that way, we eventually just decided to nix the whole dynamic programming concept and go for an all in one checking step that would greedily insert unscheduled requests into the schedule for the day, and do it in an order that was based on the lowest cost for insertion at each step. There are three main constraints that will determine if inserting an unscheduled request is feasible: checking capacity, checking for inserting the pick up, and checking for inserting the drop off. Each of these entails quite a bit so at the end of the day we started trying to create some pseudo code for what each will need to do and what functions might need to be created for each. For example, checking capacity does not only mean that the bus can afford to put the unscheduled ride's client and their party on the bus at pick up, but the bus should also never go over capacity late, which may be the case when they pick up more people before dropping off the unscheduled request's party. Checking feasibility for inserting a pick up is also pretty involved, if not more so because you have to check which runs it can be inserting into, then check if it's feasible for any of those, then insert it and pass on what the schedule would look like after the insertion. Inserting a drop off is similarly involved. After an insertion method is considered feasible, the cost of inserting it will also need to be calculated so it can be compared to other feasible insertions. Beyond all this, we also realized that we needed to clarify some of the variables in the 18 month data, and see if we could get them to be added into the 15 minute data that we're continuously receiving. All in all, it was a lot and the day was long, but I think we're more on the same page now.

     The morning started off with our 9 AM meeting with Anat to talk again about how we are framing the optimization problem. Talking to Anat this morning was helpful to me because I think that when I was framing the problem in my mind I was making it much more complex than she says we should allow it to be. Basically, we're not going to be allowed to move around any already scheduled requests to optimize further, but we're only altering requests that are modified due to adding in the unscheduled requests (due to things such as bus breakdowns). It seems like our group is a bit more on the same page now, but I'm still looking forward to talking about it tomorrow with the whole team.
    From there we went to a tutorial about working with data frames, arrays, matrices and the like in python and using pandas. It was very informative and I really liked how Bernease presented the material, but it went by really fast and I'm worried that I didn't really retain much of it because it just all flew by so quickly. At least it was all in an ipython notebook so it will be easy to reference and look back at later when I'm actually doing work in python. 
    Afterwards we talked a bit more as a group (sans Anat) about what the optimization problem would look like. We tried to work through an example to show us what implementing an algorithm would really look like. Frank was trying to work through it in python while Kivan and I attacked it using R. I think our main problem is figuring out right now which runs will be able to fit in the newly unscheduled routes and then afterwards fitting a cost with them. 
     In the afternoon we had reading discussion group, but I only got to stay for the first activity where we talked about what kind of visualizations might be helpful in representing our data. I had to leave early because I was meeting with our TCAT group in the CSE lab. There I presented the literature I had come up with on battery interrupters (pictures included!) and Zhitao and Annelise also shared what they had created for the project. We also talked about how we were going to try to help someone create a 3D model of their hand, and in preparation we were trying to create 3D models of our hands. However, we ran into some difficulties with uploading the pictures and syncing with the correct accounts, so this ended up taking a while and by the time I had to leave we still hadn't come up with the model of my hand. :(