July 16 - July 20

I tried to debug the future weights predictor, but nothing seemed to be wrong. It was solving for the transformation using all the right matrices. The even stranger thing was that when I used it on the game logs from the qualifying round, it was doing really poorly on some games and not so bad on other games. I don't really know what makes some of those games different. Aysun started applying the same predictor for future computer prices, and it wasn't working for her either.

After that, I started implementing a component prices predictor that estimates the supplier's available capacity for any given due date, which the component prices depend upon. The people who had worked on Botticelli before me had already come up with the pseudocode for this idea, so it didn't take too long. However, it performed rather poorly. Joe (who was working with the supplier model in Botticelli itself) and I brainstormed some ideas for improving it, but none of them did. One of the ideas was to look at the prediction errors every day and adjust the estimated supplier actual capacity accordingly. This occurred to us because when we looked at the errors each day, they seemed to consistently stay on one side or the other for a number of days, suggesting that our predictions were off due to the fact that the supplier's actual capacity is determined at the start of each day by a random walk. After I implemented the idea, the predictions became worse, which perplexed me. But it could be because I didn't choose the ideal parameters (i.e. how many days do the errors have to stay on one side before the actual capacity is adjusted, how much to adjust the capacity by, etc.).

Anyway, since this predictor's performance was much worse than that of the other ones I had implemented, I turned my attention to improving the quadratic programming approach. I added more features, such as the past customer demand, the supplier available capacity estimated using the same method as the one I implemented earlier, and past component prices from supplier offers. I thought the past customer demand might affect component prices, because agents might order components based on customer demand in the past. I did not think the future customer demand would affect prices very much. With so many features and different combinations of them to test, I was no longer satisfied with just commenting out different parts of the code. Instead, I added the features to use as a numerical argument to the predictor's constructor. I assigned each bit in the number to a different feature. Then I wrote a script to test various combinations of the features on the qualifying round game logs overnight. When I came back, I found that the best combination of features was the mean supplier actual capacity from the market report and the past component prices.

Still, the quadratic programming approach does not perform as well as the first, simplest method I implemented that came from Botticelli. I tried more modifications, such as taking a weighted average of the price prediction now and the price prediction using the average weights from past games. I also tried to vary the lookbehind (the number of days into the past from which to use data in the quadratic programs) depending on the prediction errors. Neither idea worked very well. The first made predictions worse, and the second didn't change them very much.

The Prediction Challenge final round is next Tuesday, and I'll probably use Botticelli's supplier model for current component price predictions and the quadratic programming approach for future component prices (since it seemed to be better than the simple method, at least on the qualifying games). Aysun said that she'll use Botticelli's method for both current and future component price predictions, since it is the best for both. This is rather depressing... we've done so much work, but haven't really improved on the original simple methods. But I guess research scientists have to face this kind of disappointment all the time.

The SCM and CAT (market design) final rounds also start next Monday, and everyone is really anxious and doing a lot of last-minute coding/testing/debugging. We've all been staying late and even working on the weekend. On Sunday, I stayed until around 2am and Joe and Mauricio apparently stayed the entire night (Amy brought us ice cream!), getting very little sleep. I hope they'll be okay.