John Rivera

Home Journal Paper

Plan of Work

Work: Research and development to captioning software for the deaf and those with low vision.

Completed

Implemented the server and front-end for the captions, partial video streaming and accessibility features such as contrast and font sizes.

Proposed

A complete captioning and video streaming system that streams visual aids to the screen along with captions in a browser window.


Week 1 (June 2 - 6)

To be completed:

Work on captioned video -- improving compliance with standards and improving usability.

What was completed:

The pre-existing captioning app demonstrating modifying the speed of a captioned video, as well as modifying the display of captions, suffers from several issues, one being the non-standard method it uses to store and display the captions (using JSON, stored in a invisible iframe on the page, to display the captions). Another issue is inability to be viewed on mobile devices (iOS, Android, et. al.). The WebVTT and the <track> HTML5 tag was introduced as a standard for displaying captions on the web, and is supported by the major browsers, with the exception of Firefox, which needs to have it enabled via about:config (full support is projected to be coming in Firefox 31).

We aimed to address this issue, and at the same time implementing access to the video and captions independently, so we can accomplish asynchronized viewing, as well as custom views outside the normal captions view.

To this aim, we converted the JSON caption file to a WebVTT file and successfully parsed the WebVTT file and displayed it as a separate entity from the embedded video. This enabled us to control the timing of the captions, allowing for asychronized viewing of the captions and the video. Bugs relating to time formats (which affects the captions after the 1-minute mark, as well as affecting the calculation of the duration of each line) were solved.

Work on improving the aesthetics, usability and flow of the video & captions has begun and is on-going.


Week 2 (June 9 - 13)

To be completed:

Continue work on UI of the captioned video view, and working on establishing requirements for improving usability of the AVD for low-vision students.

What was completed:

Speed adjustments, including fast forward and skipping, were added to the video UI to reach further feature parity with the previous version.

Worked on adapting the WebVTT captioning UI from last week to YouTube videos. The goal was to achieve the same UI created for WebVTT videos that is applicable for any YouTube video.

Successfully obtained and parsed YouTube subtitle files via JavaScript and the link provided to us by Prof. Raja (http://video.google.com/timedtext?lang=en&v= -- which, coupled with the video ID of the desired YouTube video, allows one to download the subtitle XML file for that video). Successfully connected to and made use of YouTube API, which allows for the loading, manipulation and interaction with YouTube videos via JavaScript. The API allowed for the asynchronized viewing of the parsed subtitles and the video itself. Currently, the user is prompted for the video ID, when is used to obtain the subtitles and load the video. Added color inversion and font size changing to the UI, as well as playback rate adjustments, to reach feature parity with the WebVTT UI.

Accomplishments this week

  1) Added playback rate adjustments to WebVTT interface

  2) Obtained and parsed YouTube subtitle XML files via JavaScript

  3) Learned about, and used, the YouTube API with JavaScript

  4) Implemented asynchronized subtitles interface with YouTube videos

  5) Implemented playback rate adjustments for YouTube videos

Aims for next week

  1) Research Google Chrome extensions

  2) Learn how to create a Google Chrome extension

  3) Begin to implement YouTube asychronized interface as an extension


Week 3 (June 15 - 20)

To be completed:

Towards a goal of making the YouTube asynchronized subtitles UI worked on last week into an browser extension, the week will be spent researching on how to create a Google Chrome extension, and beginning to build the extension. The goal is that a user would click the button when on the page of a video, and the UI would display automatically without having to prompt the user for the video ID.

What was completed:

Researched creating a Chrome extension based on our previous work on asynchronized captioning. An extension was created which displays the video and displays the captions separate from the video, and little else. The captions does not pause at the point the video’s timestamp. From within the extension, the YouTube API was inaccessible beyond the initial display of the video, so the timestamp, as well as controls and other features, couldn’t be accessed. After further research, including digging around in another Chrome extension involving YouTube videos, we concluded that this is likely a bug. Even the other extension suffers from this (as evidenced by the ‘undefined is not a function’ errors in the console when running the extension, both ours and the ones we found).

We considered some alternatives, including downloading the videos directly and interacting with it as one would a normal HTML5 video. However, the YouTube HTML5 player generates a random URL for the video each time the page was visited, and the URL has expiration parameters -- in other words, the url will expire after a certain amount of time. This makes it very difficult to download the video locally, especially for display on one’s own page (or extension in this case) without using YouTube’s API. We investigated some JavaScript libraries that claim to accomplish what we are looking for, but upon testing, none of them work.

Finally, we decided on a hybrid approach. We created a Chrome extension that obtains the video ID of the video the user is viewing, and open a new tab pointing to a webpage on a server which displays the interface we designed for viewing the captions.

Having moved the interface to a remote server, the calls to the YouTube API were successful, and we were able to create the interface we set out to create.

The interface was designed so the user would naturally discover the features. The contents were centered, and the controls are naturally placed beneath the video. The user would then see the buttons for controlling the subtitles, and the subtitle area directly below.

Accomplishments this week

  1) Researched on how to create a Chrome extension

  2) Investigated alternatives for activating the interface

  3) Successfully implemented the interface via an Chrome extension

Aims for next week

  1) Complete the user experience report for accessibility for deaf and low vision persons


Week 4 (June 22-27)

To be completed:

Work on, and complete, the corroborative paper on the user experience of low vision users using what we have developed thus far.

What was completed:

Completed my portion of the corroborative user experience report detailing my experiences and evaluation of classroom and online lectures using current accommodations as well as the asynchronized software we have developed thus far.

Towards the end of the week, we decided to corroborate on a new accessibility software tailored for C-Print or CART users in the classroom, incorporating the asynchronized concept developed thus far as well as the work done on video analysis and notifications. This is dubbed Project NA (Project New Accessibility).

Accomplishments this week

  1) Completed my portion of the corroborative user experience report

  2) Started Project NA with a broad idea of our goals and aims

Aims for next week

  1) Establish structure and features of Project NA and begin work on the front end aspect

June 27: Deadline to submit to ASSETS 2014 Poster


Week 5 (June 29 - July 4)

To be completed: Build a plan, and begin concrete work, on the New Accessibility project.

What was completed: Settled on a structure of the project; created a diagram showing the flow of the project -- the server receives video from kinect and both analyzes it and sends a stream to the client; the server relays C-Print or CART to the client; the server is also responsible for notifications resulting from the video analysis, which is sent to the client via HTTP and the vibration device via Bluetooth. The client accesses the program via a web browser, and is able to customize the appearance (change font size, invert colors, adjust view styles, etc).

I then worked with Warrance Yu on the front end aspect of the project. I created a functional proof-of-concept of the three main views of the user-facing portion of the program, as well as animations between those three views.

Accomplishments this week

  1) Settled design of Project NA

  2) Worked on front-end, completed proof-of-concept of the three views

Aims for next week

  1) Refine the user-facing UI, and possibly integrate notifications if progress is made on that front.


Week 6 (July 7 - 11)

To be completed: Refine UI of Project NA and possibly integrate notifications, pending on progress on that front.

What was completed: The UI was vastly more polished -- the transitions now work regardless of order, and there are no overlapping of UI elements. Progress was made on the notifications server, and it was decided it would be based on Python and WebSockets. Currently, the server is single-threaded, and sends three notifications signals, then a message, which was received by the client. The notifications display, but the message is not (yet).

Accomplishments this week

  1) Polished and working client UI.

  2) Created a working server. Currently single-threaded, and sends three notifications in intervals then a message.

Aims for next week

  1) Refine server application.

  2) Enable the server to “stream” messages instead of requiring the user to hit Return.

  3) Work on multi-threading, so the server can send notifications and messages without blocking.

  4) Have the messages displayed on the client UI.


Week 7 (July 13 - 18)

To be completed: Refine server program, display messages in the UI, work on a more “streaming”-based solution for messaging and multi-threading.

What was completed: Student and captionist UI created, with messages passing between the two using WebSockets. The captionist’s messages are relayed live, while the student’s message was sent upon the student hitting the return key or clicking the “Send” button in the UI. Threads were created in Python to handle each message type and direction. The video stream was also implemented using WebSockets using its own thread, sending a JPEG frame formatted in a base64 string to the student client.

Accomplishments this week

  1) Implemented messaging between captionist and student and vice-versa.

  2) Implemented views of the messages in the UI

  3) Implemented video streaming using WebSockets..

Aims for next week

  1) Implement video pause and catch-up

  2) Test on an actual server


Week 8 (July 20 - 25)

To be completed: Implement pause-play with the live video stream, and test on an actual server.

What was completed: We switched from a threaded approach to the server to a events-oriented approach using greenlets in Python (using the gevent library), which enabled a more efficient use of system resources. The video stream server was split off into its own program to more effectively use the OS’s own multitasking capabilities and and to make the code more readable and scalable. The application was tested on a live server with success. One drawback is that the web camera must be attached to the server itself. I worked on separating the web camera from the server, having the computer to which the web camera is attached stream to the server, which then streams to the clients. This introduced excessive lag and slowdown, as excessive compression and communication was required before the frame finally reaches the client.

Accomplishments this week

  1) Pause and resume implemented with the live stream.

  2) Successfully tested on a live server.

  3) Implemented ‘relaying’ of video to the server. This introduced excessive lag, so the solution is scrapped.

Aims for next week

  1) Attempt to find a solution to the streaming problem.

  2) Complete the patent disclosure.

Week 9 (July 28 - Aug 1)

To be completed: Attempt to find a solution to streaming the video to the server and complete the patent disclosure.

What was completed:

Attempts were made to find alternative solutions to the video streaming problem. One of the possibilities is to have the server relay the video directly from the source to the clients without any manipulation, and only grab frames at intervals for analysis. Work on this possible solution is ongoing.

Accomplishments this week

  1) Researched alternative solutions to the video streaming problem.

  2) Outlined several possibilities including streaming h.264 encoded videos directly and having the server handle analysis and notifications only.

Aims for next week

  1) Continue work on streaming video.

  2) Work on, and prepare for, the presentation.


Week 10 (Aug 3 - 8)

To be completed:

Continue ongoing work on the streaming problem, and work on the powerpoint and prepare for the presentation.

What was completed:

Work was continued on the streaming video problem. It was decided that the scale of the project is too large for a week’s work, and that it will be continued in the fall semester. The powerpoint and presentation was completed and presented to the research advisor.

Accomplishments this week

  1) Outlined plan for continuing the work in the fall semester.

  2) Prepared for, and presented, the project thus far to the research advisor.