This journal will describe my research experience and maybe sometimes some social events.

This was the first conference I attended. It was an interesting experience, I had the opportunity to see what a conference looks like. The topics discussed were much above my level so I could follow little from what they were saying at the 25 min presentations of their papers. The general talks which happened on the 3 consecutive mornings were more understandable. I enjoyed the Tuesday talk - Small Brains, Smart Minds: Vision, Navigation and 'Cognition' in Honeybees, given by Mandyam V. Srinvasan and the Wednesday Talk - Ultraconserved elements, living fossil transposons, and rapid bursts of change: reconstructing the uneven evolutionary history of the human genome by David Haussler.

From the presentations of papers the most interesting one was predicting moves in the game of GO (Bayesian Pattern Ranking for Move Prediction in the game of Go). The second most interesting was Topic Modeling: Beyond Bag-of-Words, by Hannah Wallach.

Friday was my first day at the lab in Stephenson Research Center. I was introduced to the people who work in the lab and the other REU students. They showed me the projects they working on - genetic programming, using reinforcement learning to develop an algorithm that can beat human players in the game of GO, severe weather prediction, robotics - finger recognition and glove that detects the human motion and displays different parameters among others. And of course I saw the bions hanging from the ceiling that interact with each other and with people. When they are turned on and there is no one around they all make a buzzing sound and emit a blue light. When a person approaches they "get excited" for a few seconds, i.e. play a different, happier and more pleasing to hear sound sequence. Then they "get scared", sensing the change in their environment and go quiet for a few second until they "habituate" to the new environment. Then they continue to play the excited sound sequence until the person goes away, when they return to the normal buzzing sound.

1. I attended the first REU meeting in person (because before I was listening from the phone and following online from Miami). There were presentations about how Pruning works and Relational Data Sets, Movie Database to predict if a movie will get an award.
2. Implemented the gridworld reinforcement learning program, using the Q-learning algorithm from the Sutton and Barto book.
3. Worked on understanding the C code with which the bions work. Now I know which function controls the initial sound and blinking LEDs. I changed the code a little bit to make one bion play a simple version of a song. But the song sounded strange, because there were no pauses.
4. On Friday we had another REU meeting where all the students had to make a presentation about their summer plans.

The presentation on this week's REU meeting was how to write a good scientific paper. We had an interesting activity where we divided in pairs to write a hypothetical Main Ideas, Hypothesis, Approach, Results and Conclusion for a new vacuum cleaner that works as a robot and cleans the floor by itself. We had to improve on the existing Roomba.

I changed the code of the generate_music function so that now it really can play a simple version of Jingle Bells :) Unfortunately the capabilities of the bion are limited, so a more complicated song such as Imagine won't be able to be coded and given as parameter in that function. I read little bits of the book the C Programming Language by Kerninghan and Ritchie to help me with the programming. I really like the way this book is written.

I have also been working on this website and this was the most wonderful experience! I loved making a website and learning some HTML. It's fun :).

On Saturday morning from 8:30 am to 11 am, we played disk golf with some of the other REU students and one of the professors. It was the social activity of the week and was a lot of fun. It was in a very nice park and we enjoyed the nice weather (sunny and dry, ~80 F) while throwing disks(look like frisbees) and watching some beautiful and long disk/frisbee flights. Then I went to the mall with my roommate, just a regular American Mall, but it was still nice.

On Monday, there was the usual REU meeting. The first presentation was by Dr.Terran Lane on Expectation-Maximization. He was connected through the Access Grid together with his students from the University of New Mexico.

Here are some of the key points:
• algorithm that takes data and returns a label(cluster ID) for each data point
• Natural Language Processing, The Gramatical Tree Structure, reference to Hidden Markov Model Algorithms
• Classic Latent Var Model
• SLAM - Simultaneous Localization at Mapping
• Joint Probability Distribution
• Marginal Distribution
• Chinese Restaurant Process - best answer to date

On Tuesday and Wednesday all the people in the lab were mainly occupied with the bions. There was a new program which would allow 100 bions to be programmed at once,as opposed to before - one at a time. We had to install that program on all 1000 bions. I worked in a team with another person, I was the programmer, he was the support person. So basically I would execute about 5 simple actions and he would have to plug in the bion into the AVR programmer and power. It was very repetitive but it was worth it... since after that 100 bions could be programmed at once. On Wednesday we had to put the chip back into the plastic box and glue it together. It was fun because we got to talk to each other in the process. There were interesting conversatios about Grad School, the correctness of the IQ test and others.

On Thursday and Friday I worked on coding the Reinforcement Learning algorithm. So far I have done a simulation for the state space and the action space. The state space is the last 3 notes played by a bion and its neighbor. The action space is an octave, 8 notes and silence. I am coding a simulator for the bion/person and bion/bion interaction, so that I could test that my RL code is working correctly, before I program it into the bions. It is easier to test on a simulator, because there won't be the problem with sensors not responding.

Social events: pictionary and dinner at Dr. Hougen's house this Sunday. Other than that Norman is a town where the buses run only once an hour and things are too far to be fun to walk. I don't have a car, so I don't see many things in town. But my roommate Laura told me that there isn't much to do in Norman, in Oklahoma City maybe, but I haven't been there yet.

This week's REU meeting was by Dr. Terran Lane, about Graduate Schools. It was interesting and informative and so was the paper which I read "Applying to Ph.D. Programs in Computer Science" by Mor Harchol-Balter from Carnegie Mellon University. Most of the rest of the time in the week, I spent writing code on and debugging the Reinforcement Learning bion simulator. Originally I had done the person getting in proximity with the bions and leaving to be randomly chosen every time step. That did not work on the learning because the bions were getting rewards randomly, no matter what note is played and that was not the idea. So I changed the scenario(under Dr. McGovern's guidance) to the following: a sequence of notes is played, which is the reinforcement learning goal. This is what this example person wishes to hear and the bions should learn to play that sequence. On Thursday we thought we got the learning working, since it was learning to play a sequence of one note in 4 or 5 time steps. The other 95-96 times, the note played was 3. We plotted a reward learning curve on MatLab. It had lots of oscillations, but the general idea was evident, it did jump up from negative to positive reward. The learning is not done yet on 2 or more notes and we are still working on it. The decision so far is that linear approximation for the state value will not work and we will have to think of another function and two of the other students working in the lab suggested using Neural Networks. They explained to me what Neural Networks are in general and suggested to read the Artificial Neural Networks chapter from the Machine Learning book by Tom Mitchell.
On Tuesday we had a student meeting organized by one of the graduate students, where he explained to us what a Hidden Markov Model is.

Neural Networks will not fit in the memory of the bions (1k RAM and 8k programming memory) so we decided that we will use a Genetic Algorithm instead. To begin and test if the algorithm works we made the population to be 10 bions, each having a song of 3 notes, and a fitness level. The fitness function calculates the fitness based on the notes played compared with the desired sequence of notes. The goal is to get all 10 bions to play the same sequence. The bion songs were randomly initialized in the beginning. Then we start a loop of creating new generations, each new one should be more fit. If there is one note in the correct order that matches the desired sequence, fitness level is 0.33. If there are 2 matching notes fitness level is 0.667 and if all three match fitness is 1.0 . If the diference in fitness between two bions was small we did a crossover of the 2 songs, if it was greater the song of the better fit bion was copied into the not so fit bion. The pairs of bions were formed by each bion sending a song to a randomly chosen other bion from the population. The loop ran 10000 times and at different times. I tested it 6 times. The first time at the end of the loop , there were four bions "playing" the desired song. The second time five bions , the third time - four bions and the other 3 times , no bions played the desired song in the last iteration. So I think that this was a simplified version of the Genetic Algorithm outlined in the genetic algorithms chapter of the Machine Learning book by Tom Mitchell. I started making changes to the code, so that it will be exactly like the algorithm in the book, i.e. I would have to probabilistically select members of the old population to be added to the new generation and then probabilistically select pairs for crossover. Then I will have to mutate some of the new members. So that's what I will continue implementing in C next week.

This week's REU social event was visiting Bricktown, which is the entertainment place in Oklahoma City, it has nice restaurants, so we had dinner there and walked around. We also saw the memorial built in memory of all the people who died in the bombing of a building by Timothy McVeigh in 1995.

I finished the simulation and it ran well. I modified and generalized the fitness function, so that it can work for any sequence length. The fitness gets higher as more notes are played in order from the beginning. So if the first note is played fitness is 1/length, if first two are played, fitness is 2/length and so on. The slight disadvantage of this is that fitness is evaluated on fewer note combinations, which makes the program go through more iterations to reach the final goal. With the previous fitness function, which was only good for song length 3, the average was about 400 iterations to have 10 bions play the same given 3 note sequence. With the generalized fitness function, the average was about 880. (I say about, because the average is based on 20 runs). The greater the population and the longer the note sequence, the more iterations are required. The goal for 5 notes is reached in 8963 iterations, but longer sequences don't give a result in a reasonble time (a few minutes). After 5-10 minutes I stopped the program. This may be a potential limitation for the song length that can be played on the actual bions.

The rest of the week I have been working on integrating the simulation code with the biontemplate.c and the bioncore.c. Communication between the bions is possible by sending one value at a time only (currently it is a nybble, I will have to change it to int). The solution Dr. McGovern and I decided on, was to encode the bion_id, note and note index (indicating which part of the sequence the bion is on) as certain bits from the integer and then on the receiving end to decode that information back into individual int or char values. I wrote the functions to encode this information as well as the fitness(to be sent seperately as another int) and to decode the received integer value, using bit shifting and masks. The change from the simulation is that simulation assumes all bions being in an array of structs, while on the bions, each bion should run the same program and the evolution on the group of bions as a whole happens through communication.

The social event this week was a visit to the Art Museum at the Oklahoma University Norman Campus. I enjoyed it.

This was my last week at the Stephenson Research and Technology Center in Norman and the end of my summer program. I continued working on integrating my simulation code with the bioncore.c . After some thinking we decided that the value to be communicated will have to be a long and we will need to use 15 bits from it to encode the information to be sent and 4 bits for start/stop communication and parity (as specified in the bioncore file). I had to change some of the code in the bioncore as well. With the assistance of Dr. Fagg, we tested and debugged the changes, using an oscillioscope. Dr. Fagg showed me how to read the bits on the oscillioscope and explained to me how the graph is generated. It was very interesting. Finally the code was error free and communication of long data types between the bions was possible. There is still some more work to be done, but the project was very close to its end. I wish I had a week more so I could see if the original goal of having the bion evolve a song, based on the person's wishes (the person stays if he/she likes the sound and leaves otherwise) was successful. It was 7 pm on Friday night and I was tired and couldn't see the error in the function I was working on. And I suspected that for the final product to be done, there would be needed few more days. I flew back to Miami on Saturday. This summer was a very interesting and valuable experience and I definitely reccomend it. :)