After the long weekend, Thomas and I each had a couple ideas for confidence scores.
Lenore suggested that we take the average of the top half of DSD scores only, to build in a certain amount of pessimism when we create false shortcuts in the graph by adding false positive edges into the graph.
Thomas thought we could start by taking the average of either the top or bottom half, depending on which is farther away from the original DSD score. He also thought we could count the number of scores that are greater and lower than the original, and average only that side.
My idea is to separate the DSD values into bins to find peaks in the distribution. If adding/removing a specific edge creates a big enough change in the network, there will be two peaks - lower peak if a shortcut is created, and higher if one is removed. We can then decide if we'd rather be optimistic or pessimistic, perhaps based on the size of the peak.
We spent most of the day implementing our ideas (and deciding how to implement them).
Ended up needing to optimize the matrix averaging code further so Also made it more modular so that we can drop in these new methods without having to tear apart the code each time. The former improved the performance by about 30%, and the latter didn't take very much to implement since the code was organized well enough in the first place. One of the many times I've been thankful for my program design class!
As per a suggestion at the undergrad meeting, we decided to invert the confidence scores to evaluate our current methods. If we have a good method, taking (1-confidence scores) should completely break, producing very very low prediction accuracy rates. We'll rerun DSD overnight to verify this.
Wow, a huge surprise! DSD did NOT tank when inverting confidence scores. Yes, the accuracy went down, but not NEARLY as much as we would have expected it to. After a great deal of discussion, we decided that this is indicative of a less than optimal measure of confidence, noise in the method, or a bad method in general. Lenore believes this is mostly due to noise in the method -- with only 100 networks, we can't possibly be able to evaluate the changes adding and removing edges makes on a large enough scale. Our confidence methods may be a bit rough as well, but once we find a new way to add confidence, we can reevalute this bit of it. We might have a new idea, internal to DSD, but I'll share it when we develop the idea more.
My aunt Natalie, mom, and Maya came out to visit this weekend! We went out to dinner last night, and spent a lot of today shopping. In the morning we stopped at a cute cafe where we discovered cayenne ginger ale -- it was surprisingly great and so refreshing! I'll have to look up the recipe later.
Spent the morning with Natalie, mom, and Maya before they flew home. It was a fun weekend with them!