I finished the first version of the updateQ (state-action value update) and coded any missing helper function
or accessor function, as well as a main function for testing purposes. I got disgusted with Windows when trying to
compile so I switched to Linux.
Several hours of debugging have finally produced something useful. The agent selects its moves according to an
E-greedy policy and updates its state-action values according to Sarsa learning. The code still needs to be cleaned up
to make access to functions clearer and just more user-friendly in general.
After cleaning up the code (and adding a Pos class to make dealing with the agent's position niceer), and debuging to
clean up the 'clean' code, I got it working again. The agent is learning to avoid the cliff, but unlike the results predicted
in the book, the agent skims very close to the edge rather than taking a slightly longer, safer route.
I spent a few more hours tweaking everything to be more readable, and to have be possible to stick as closely as possible
to the algorithm
in the book (this meant improving
my updateQ function). The agent still likes to go close to the edge, but more a good part of the time it now selects the
safer path after 100 episodes.
I made a few big changes to the structure of the code. Rather than just SarsaAgent I have an EGreedyAgent as a base class,
which SarsaAgent extends (so only updateQ and constructors need to be included in it). I then coded a Q-learning agent
QAgent, that also extends EGreedyAgent. It works perfectly, always choosing the optimal path (except when exploration makes
it do sub-optimal moves). I also obsessively commented the code the way I like it. This part is done for now, and we can run
some tests using both agents
All in all I had a lot of fun coding these two agents, and seeing them actually learn to avoid the cliff in different
ways really made it worth the whole effort. It went pretty well, with lots of debugging, but nothing horrible and drastic
(like searching for the -same- bug for hours. I was fortunate enoguh to have a seemingly endless series of small, easy to spot
bugs in the code). It was a good experience in coding in Java from scratch, and really reinforced (so to speak) the ideas
behind Sarsa and Q-learning, as well as what I read in the earlier chapters of the book. I am planning on posting the
results of the experiments probably next week.
As my final work for the week, I built the first version of this website. Sections included are personal information, some
information about my mentor, Professor Doina Precup
, information about the project and this journal.
It is for the moment a very minimal site, but will hopefully improve as I get some inspiration for the layout.