Well, this is it, the last week of work. I don't have time to start anything new, but I do have to time to wrap up all
that I wanted to test. This includes comparing various values of temperature, with the same goal reward and value for lambda, as
well as various values for lambda with the same temperature and goal reward, and finally seeing whether a higher reward really
does tends to stabilize the total reward the agent manages to collect.
First of all, different temperatures: I compared T=10, T=15 and T=20. In all other respects the agents were the same,
with lambda=0.5, fast disaster learning, and the goal reward was +2500. The trend is that smaller temperatures are better. They
learn faster in the beginning and have much higher values in the 1st episodes (see 1st graph below), they have a higher maximum
total reward obtained (see 2nd graph below), and less variance (see 3rd graph below).
This is pretty much the results that were expected. In the 24x15 world, the same happened, but the higher temperature
managed to find a hidden goal that the lower ones couldn't find, and eventually the reward was greater (see
). The bigger world has no hidden goal, so there's no advantage in the higher temperature.
Secondly, different values of lambda. I've compared L=0.1, L=0.3 and L=0.5. Once again, the agents are the same in all
other respects; in this case, the temperature was 15, and the goal reward was +3500. The results are got weren't very inspiring.
There was hardly any difference at all between the three values. The only difference I can see is that higher
lambda (0.6, for instance) don't work at all. This is strange behaviour considering how little difference there seems to be.
The results, though not spectacular, were also expected. In the smaller world, eligibility traces made no difference at
all in stabilizing the final reward, or increasing it. They helped find the second goal faster, or just helped find it when the
agent did not explore enough to find it on its own (see week 5
Lastly, I have another comparison between different values for the goal reward. As I noticed last week, the higher the
reward, the more stable the final reward collected by the agent. It doesn't really increase
the total reward, but the extra
stability is nice, giving a higher average. The reason for this is most likely that having the higher reward increases the
probability of taking the path to the goal; this is obviously an advantage in the 30x30 world
it reduces exploration, which, as we saw in the 24x15 world
, is critical for finding hidden goals.
Well, that is the essence of my findings over the whole summer of work. I learned a lot about reinforcement learning,
and it was all very interesting. I used some neat techniques that sometimes really helped the agent out. All in all, I think the
results were interesting, but that something isn't quite right about it all. The fact is, it really depends a lot
world the agent is working in. In the larger world, lower temperatures were much
better, no question, but it was the opposite
in the smaller world, on which there was an optimal temperature, neither too high nor too low. The use of eligibility traces didn't
have this kind of problem. It was useless but not problematic in the larger world. However, the use eligibility trace slows down
considerably the computing time, which will be a problem as the environment gets larger and larger. My conclusion is that, while
this is all very useful in small, slightly controlled cases, in a large, completely unknown environment, there is just too much
fine-tuning to do on all the parameters. Most likely real progress will be done in another direction; probably clustering states
together will be something to look at next.