Weekly Log: Working at Northwestern University
|About My Mentor
First and Second Week | Third Week | Fourth Week | Fifth Week | Sixth Week | Seventh Week | Eighth Week | Ninth Week | Tenth Week
June 1-12, 2009
My first assignment as part of joining Dr. Joseph's group was to read up on current literature
about the topic at hand. This proved helpful because I gained knowledge into what has already been done in this field. Below is a list of current research
on the topic at hand:
- A systematic methodology to employ error-tolerance for yield improvement
- Application-Level Correctness and its Impact on Fault Tolerance
- Design Methodology to trade off Power, Output Quality and Error Resiliency:Application to Color Interpolation Filtering
- Multi-media applications and imprecise computation
- Reduction of Detected Acceptable Faults for Yield Improvement via Error-Tolerance
Now that I knew the basics of the OS and FFMPEG, it was time to start my part of the project, which was to profile FFMPEG and see which MMX procedures were most used when making conversions. This is helpful when trying to manipulate code because if a procedure is only used .01% of the time when encoding or decoding then it would be useless to modify it. At first Dr. Russ Joseph suggested Gprof which FFMPEG can be easily configured with it enabled. When I finally managed to do a test run, it ended up giving pretty good results. Whenever I needed help with configuring any part of these programs or I was stuck on anything, I would either simply Google it or I would go onto to IRC FFMPEG channel where fellow users were able to help me out. Users on this IRC channel as well as comments online suggested I starting using Oprofile to profile FFMPEG instead of gprof. Oprofile is more useful in profiling because it profiles the whole system and does not need programs to be recompiled like Gprof. The only modification needed in the running programs are to enable debug symbols. This is done easily with FFMPEG with just using the --disable-stripping flag when running ./configure in terminal. I then configured Oprofile to filter out all results except debug symbols coming from FFMPEG.
The day I got into Chicago, I met up with two new friends I just met a couple weeks before when going to Chicago to scout for apartments. I
arrived to Chicago on May 29th to set up my belongings in my apartment early. Luck had it that Northwestern University was actually holding an event that weekend called Dillo day. It is always
hosted on the last Saturday of May of every year. Dillo day was a huge event where a large field turned into a stage and various singers performed. It was a great experience and I had lots of fun.
Outside of Work
On Monday, I met Dr. Joseph for the second time and the students that he advised. They had their own lab room and everyone had their own computer. I also met a fellow student, Giang Hoang (G for short), who would be working with me and Dr. Joseph on this project. Both these guys were really nice and got me set up in my own desk. I will be working with them for the next 10 weeks and I'm glad they are really helpful.
On Friday, I was invited to a lecture by Philip Wadler, a professor of Theoretical physics. The title of his lecture was Well-typed programs can't be blamed. It was a fairly interesting lecture even though some concepts flew over my head at times. Throughout the rest of this week I finalized the approach of what output I need to produce from the profiler. G, one of the students also working under Dr. Joseph, has given me modified code this week so i can compare the profiles of both the unmodified code and the modified error correcting code. Oprofile is very powerful as I can modify the code, recompile it and then run it again and it can compare both profiles and documents its change in running time. I have now started on the bash script to automate these processes and give me the results that I need.
June 15-19, 2009
This week was concentrated on putting my website together and finishing up my bash script to automate the process of profiling FFMPEG. I put in checks to make
sure the user had root access when running the script and also set up a flag, -c, when you just wanted the script to clean up the results it had output. It automatically creates folders and starts
profiling when encoding from PPM files to JPEG files. This conversion is done 1000 times that is achieved through a loop in bash script. After this is done, the script shuts down the profiler and outputs the results
of the processes and procedures FFMPEG goes through. This process is done using both the unmodified and modified version of FFMPEG on four different .ppm images. Then, the results of the modified and unmodified versions of FFMPEG
are compared. Also, a different folder stores the assembly instructions that ran when encoding through FFMPEG. Once I fine tuned everything and everything ran smoothly, I showed and sent the results to G and to Dr. Joseph. As I have mentioned before, I also
spent time this week building this website. Now I am going to write bash scripts on decoding from .jpeg to .ppm files and also from .ppm files to different video formats. The video formats include: x264, theora, .3gp, and Xvid.
Throughout the remainder of the week, I found many test video sequences to use for converting into these tested formats. It was decided that we should put these video formats in the same container format to use throughout all the encoding processes. At first the video test files I found were frames of a video in .ppm files. These weren't ideal since they were in black and white, so I decided to look around for more. I found some good samples from an FTP site by the Video Quality Experts Group(VQEG) and have decided to use those. These are in .YUV formats though and so our methods of testing will have to change. Instead of encoding PPMs to the video formats, we will convert the .yuv files to the different video formats.
This week, I got to finally explore the city of Chicago. Since Northwestern University is in Evanston, a suburb of Chicago, making trips
by the train took about 45 minutes. This trip was best saved for the weekend. I went to go see Millenium Park, a huge park in the middle of Chicago which holds
an outside auditorium and many works of art. I finally got to see the bean and gaze at the beauty of it all. Being here in the middle of Chicago with new friends was an amazing experience. It
helps that Northwestern University is on the quarter system and my University is on the semester system. Because of this, I was on break while they were still in school. It made it easier to
find new friends that are still in school because no one is gone yet. As it turns out, this week is the beginning of summer vacation for NU students. Most of my friends are going to leave
, but, gladly, some live in Chicago. It was really nice meeting new people and one my friends actually graduated.
Outside of Work
June 22-26, 2009
The start to this week was rather different. Instead of coming in at the usual 10:30 AM, I have decided
to start my week off with coming in at 8:30 AM to do more in the daylight. Also, there have been problems in encoding the YUV files into other video formats. It seems that the YUV
files I have downloaded have no information that FFMPEG can pick up on to convert the files. This includes the framrate, resolution and the format the video was in. It seems these videos were
4:2:2 YUV files with different resolutions and framerate. So I had to test each individual file to make sure I have chosen the right formats in order to convert the video. This was a big hassle and took
a good amount of time but I have finished finalizing the parameters I would need when converting these videos into other formats. I have also picked up several other test video sequences from two other sites.
This is good because now I have a wide range of different video sequences with different resolutions and framerates. These resolutions range from 176 x 144 to 1920x1080.
I also dedicated this week to studying the different codecs we'd be using for sound and video encoding. I'm glad I took the time to do this because it further informed me of the codecs we'd be using. There are so many formats out there, it would be hard to choose only a limited set to work with if I did not study some of them. I mainly only focused on the codecs that were popular. FFMPEG is all open source so all the encoding and decoding for this program was made by scratch from developers. It takes time to design and code a codec so any recent codecs that just came out are not likely to be implemented yet. This throws the new blu-ray format (VC-1) out the window. What we have, though, are other many codecs that are wildly popular and others that have been used immensely in the past.
I have started asking around Northwestern if there was any way I could get a wildcard, a card students use that gain them access to the bus and gym
facilities. As it turns out, since I am in no way affiliated to the school, there is no way to get a wildcard. This would help greatly since I have to either take the train, they call it the el here,
or drive my car. It's a 30 minute walk to tech from my apartment but the weather has not been that great. It's beneficial to know what kinds of transportation are at your disposal because it can
really help out economically. Also, I have begun to learn to cook for myself because eating out all the time is a huge drain of money. My mother has guided me to begin cooking some homestyle Peruvian
cooked meals. It's great learning these meals especially since I'm so far away and can't really eat the meals she makes at home in Florida. Also, since I have a YMCA gym membership in Florida, I
could use the gym located in Evanston. Luckily, the gym is only a few blocks away from my apartment. It's always good to start something new when you're in a new city and by yourself.
Outside of Work
June 29-July 3, 2009
The main video codecs that were chosen are H.264, WMV2, Dirac, MPEG1, MPEG2, and MPEG4part2. MPEG1 is one of most popular codecs to date and also
a very old one. This codec was meant to compress VHS quality video and a hardware device that can't decode this standard is virtually unheard of. MPEG2 is the successor of MPEG1 is is used
everywhere. Its compression technology is used in DVDs and sending video over cable and satellite. WMV2 is an older version of the windows media video format that is popular on the web. WMV3 is
the newest format, but it is the same codec as VC-1 so an open source alternative codec has not been developed yet. MPEG4part2 is a newer codec coming from the Moving Picture Experts Group (MPEG).
Both DivX and Xvid are implementations of this format and is very popular when ripping video from dvds and other mediums. Ripping is the process of transcoding from a medium like a DVD or Blu-ray
disc and storing it onto your hard drive. It is a lossy codec but the file sizes are often much smaller. H.264 is the newest standard that has come from MPEG and includes a more advanced set of
video compression techniques. H.264 was made to be highly scalable, encoding videos for phones to encoding blu-ray quality video. Up until now, all these video codecs that were discussed use the
discrete cosine transform (DCT) as the main method for compressing video. The only video codec that is different is the Dirac video codec. Dirac actually uses a totally different method than the
DCT. Dirac actually uses the discrete wavelet transform(DWT). Dirac is a fairly new codec that just got introduced in 2008 and it would be neat to read on this codec because it is fundamentally different
than all the others. As you can see, this week was spent on reading up mostly on these codecs.
On Sunday, June 28th, there was a gay pride float parade in the district known as "Boystown." I have never been to a float parade so it
was interesting to see and it's always good to experience new things. I went with two friends and saw most of the float parade. It was exciting and often hillarious to see the many
different floats. it was a fairly interesting experience and I was glad to go. Also, starting this week, is Taste of Chicago. In this event, restaurants from all around Chicago set up
tents in a huge area around Millenium Park and, with tickets, you can have a small taste of what these restaurants have to offer. I went with friends at first and later went with Dr. Joseph
and his students. Also, several performers sang in the outside auditorium. Among these artists were most noticeably, Counting Crows and Ne-Yo. It was a great week to be in Chicago.
Outside of Work
Saturday, July 4th, also marked Independence day! Happy Independence Day! Instead of seeing the fireworks in downtown Chicago, a friend and I instead decided to stay in Evanston and look at the fireworks displayed here. Near my apartment, there was a small beach entrance. We went there to see the fireworks and it turned out to be a good time. The town was full of festivities this week.
July 6-10, 2009
This week, I am spending on making bash shell scripts for video encoding and profiling each instance. I wrote one up beforehand for MPEG1 but now
it was time to start work on the other codecs. I figured out that the Synaptic Package Manager on Ubuntu can not be used for installing the codecs or FFMPEG. which is the program I'll be profiling.
This is because the version of FFMPEG that comes from package manager has to be freely distributable so all the restricted use codecs were not installed. I instead had to download the source files
and compile it myself like before but now I have to add more flags in the configure file. Just in case you were curious, compilation from source are as follows:
- ./configure --enable-extra-warnings --enable-libmp3lame --enable-libvorbis --enable-libx264 --enable-libxvid --enable-libtheora --enable-gpl --enable-nonfree --enable-libfaac --enable-libfaad --enable-libamr_nb --enable-libamr_wb --enable-pthreads --enable-debug --disable-stripping --enable-libdirac --enable-libschroedinger --enable-postproc --enable-static --enable-nonfree --enable-ffplay
- sudo make
- sudo make install
Not much went on this week. I learned some more dishes and tried to make it a habit to go work out at least every day. Over the weekend, people
play soccer at the fields and I joined in with my co-worker, G. It was great to find other people to play soccer with. Also, the weather has been getting better to play soccer. In the first two
weeks in Chicago, the weather was very hot, which I hated because my apartment did not have air conditioning. Never underestimate the power of A/C. I was on the top floor too, so it felt like
a sauna without a fan. I expected it to continue being very hot throughout the whole 10 weeks but, to my surprise, it cooled down immensely. The weather would stay aroudn 60 and it would either
rain or be very cloudy. This was better than boiling in your own room. I also visited downtown Chicago to see the Sear's Tower. This tower is currently the tallest building in the United States.
It cost about 20 dollars to go all the way to the top, known as the sky deck, and it was well worth it. On the top floor, there were compartments that stuck outside the actually building that
was made of 3 inch glass. Underneath your feet, you can see the tiny cars and people. This also gave an amazing view of the skyline. On top of that, the sun was setting, which made for an especially
beautiful picture of the skyline.
Outside of Work
July 13-17, 2009
I have finally finished making the shell script for the video conversion and tested parts of it. I am having some problems with encoding
into some formats and am looking into answers. It seems that each video conversion needs different parameters when encoding and decoding from different codecs. It seems like this part
won't be just a simple copy and paste job for the different video codecs. Getting most of these down seemed easy except for x264 and Dirac. The encoding for x264 took a little bit of research
on my part to get the proper parameters right but, by the end, the video output finally gave a good quality, non distorted video. For the Dirac video encoding, it seemed that no matter what
parameters i put in with the conversion, I still got a greatly distorted video. After some odd hours of scouring through the internet, I finally found a set of decent parameters to give me a
decent video output. The only negative aspect that came from these parameters is that timestamps weren't added onto the output video, so you couldn't pause, rewind, or fast foward the video to
look for any artifacts.
On another note, Dr. Joseph asked me to look into the innards of how the video files are actually encoded. I read a few documents and some websites about DCT and DWT but that was just an overview. For each codec itself, these transforms were only a small fraction of what each codec does to encode video files. I looked for documents pertaining to getting an in depth look on these codecs and one of the developers for the x264 codec and ffmpeg program suggested I read a book called H.264 and MPEG-4 Video Compression by Iain E. G. Richardson. I began reading this book and it went fairly in depth on the subject on the H.264 codec without getting into too much technical details. It was fascinating how a encoder actually worked and the neat tricks the encoder implemented in order to reduce space while keeping image quality as well. As it turns out, lossless video compression just reduce redundancies within the data but the size of the file is only modestly reduced. Lossy video compression gains a high amount of compression within videos but some of the actual data within the video will get lost. This means that if you use a lossy compression algorithm, you cannnot decode the encoded video file and expect an exact replica of the video you converted from. This will help us though, because since Dr. Joseph and G are dealing with correcting data calculations with a huge data error to calculations with only a modest amount of error. Using a lossy video compression algorithm can help hide this error even more.
It sure is a small world isn't it? I recently met a graduate student from Northwestern University on the train. As it turns out, when I went up
to Cornell to visit, he was the one who actually housed me and another student as part of a program. This was amazing that 4 years later, we would see each other again. He also happened to live
below me at my apartment. We met and hung out for a few hours one weekend and I got to meet his roommates. I also got to see many new movies with friends this weekend. Some friends and I also
drove around Chicago this weeekend to visit communities. We had the chance to visit Humboldt Park, Little Village, and many others. We eventually parked and walked around these communities. Later
on, I had the chance to go with friends to see a impov comedy group called Comedy Sportz. In addtion, I got to visit the Art Istitute of Chicago which contained a huge amount of art and a lot of
walking. The galleries were huge and after 3 hours of walking around, we have barely seen half the exhibits. Moving closer to the next week, I started missing home a lot since next week will be
my birthday and usally I do a family dinner and celebration with them.
Outside of Work
July 20-24, 2009
This week, Dr. Joseph felt the students he advised didn't know each other's projects. He told us by the end of the week to have a short five minute
presentation ready to tell each other what we have worked on so far. This task wasn't so bad and I was happy to show what I have learned so far. Reading more on the inner workings of the H.264 codec
got me even more interested. As it turns out, for encoding a normal video, the video must first be encoded from the standard RGB color space to the YUV color space. The RGB color space splits the
values of the colors within a pixel for red, blue and green. The YUV color space splits the values of the brightness, red, and blue values of each pixel. This color space is more ideal for data
compression because humans can perceive brightness (black and white values) more clearly than color values. The pixel values for brightness is kept the same while the color values for each pixel
can be downsampled to 1 value per 4 or 2 sets of pixels. The codec then tries to remove as much temporal and spatial redundancy as it can. With temporal redundancy, videos often have the same
objects inside many sets of freams because they are shot 30 times a second in a video. This means a lot of the regions inside sequential frames of video have the same object and it would be
redundant to save the same objects for each frame. Instead, the encoder saves one frame(I-frame) as a whole in a set of frames and uses pieces of that frame to construct the other frames in that set from
it. In case an ojbect in the video moves, only a vector needs to be saved that shows where the object from the I-frame moves instead of saving the object itself. With spatial redundancy, often parts
of a frame are copies of other parts of that same frame. Imagine filming a brick wall. For the most part, each brick is the same, and only one brick could be saved in the image. When reconstructing
the image, that one brick only needs to be copied over to the rest of the frame. Deleting these redundancies saves a huge amount of space. Just looking for redundancies isn't perfect though, because
objects in a video are always moving or turning. The pieces of this frame that are already saved are deleted from the current frame and what you are left with is a residual. This residual then undergoes
the DWT or DCT process to further reduce the size of the video. This is just a broad overview of the main processes an encoder does to reduce the size of the file.
By the end of the week, I finally got the video encoding scripts to fully work. As it turns out, I needed to use GIT and SVN to get the latest codec files and ffmpeg files in the repositories. The repositories for these programs have the most updated versions of these files and contain a huge amount of bug fixes and changes. I installed the latest version of FFMPEG, x264, and Dirac and the problems with them were fixed. X264 added more parameters to help with the encoding process and Dirac can now work without a special workaround to encode a video with timestamps. This week was great. I left the script running over the weekend to profile instances of encoding 1000 times so I can make sure I got all the files needed.
This week marks my birthday week. I was born on July 23rd and there were a lot of festivities around. First, my friend surprised me at midnight
with a cake and lit candles in hand to wish me a happy birthday. I was working late that day and she surprised me at home. I am happy to have caring friends. Secondly, the Millenium Park outside auditorium
hosted a very special group that day. The group's name was NovaLima and they are an Afro-Peruvian band with electronic influences. They were amazing on stage and caused everyone to stand up from
their chars and dance in front of the stage. Everyone was happy and joyful and everyone of all ages danced to their music. I guess I lucked out that this coincided with my birthday. Also, later
on that night, NovaLima was going to be at an afterparty which me and some friends decided to go to. It was an amazing experience. It was a small bar and it played their music all night long.
In the end, I also got their new CD signed by the artists for asnwering a trivia question. It was a good night.
Outside of Work
July 27-31, 2009
This week I was told to start looking into making a list of all the procedures that were called when encoding videos that had any MMX or SSE
Intel instructions. These instructions were the main instructions we were looking at since they were SIMD instructions. Our hope was to fix problems that were caused in this section of the CPU and
hopefully use this to extend it to fixing hardware problems within graphics processing units (GPU). This was a step into the right direction because all the GPU deals with mostly is SIMD instructions. SIMD
instructions work well in multimedia because instead of using one instruction to change a value one at a time, one instruction can be used to change multiple values at a time. For example, if you want
to change the brightness of a screen, you could use one instruction to reduce the brightness of six pixels on the screen instead of just one. This greatly reduces the numbers of instructions and also does
these changes in parallel. In order to find these instructions, Dr. Joseph suggested I use a python script. I had little knowledge of python but it ultimately proved beneficial. The script takes in the oprofile
results file and matches it to a list of procedures that have any MMX or SSE instructions in them. This list of procedures was taken from another file that contains the debugging information from
FFMPEG, X264, or Dirac. Once all the procedures that contained these SIMD instructions were found, those procedures were spit out into another file.
Dr. Joseph also wanted me to look into the C files from FFMPEG and take a procedure that is full of SIMD instructions and describe what is actually happening. He wanted me to create an example inside a well known procedure that codecs use when encoding files. We chose the DCT, IDCT, and DWT transforms. I looked into the file that was run when executing the DCT of a file and it proved to be fairly confusing since most of these procedures are written in inline assembly inside the c file.