DREU 2015: WEARABLE AAC
PROJECT
ABOUT ME
Hello! My
name is Lauren Gaber, and I attend the University of
Michigan Dearborn. I am a junior majoring in computer science in the Computer
Information and Science Department, planning to graduate around April 2017. You
can reach me at lgaber@umich.edu.
ABOUT MY MENTOR
My mentor
for this project was Professor Shaun Kane, who currently teaches at the
University of Colorado Boulder in the Computer Science Department where I
performed the research internship. He studies accessible user interfaces and
mobile human-computer interaction. His website can be viewed here.
RESEARCH PROJECT GOAL
The research
project goal was to work on developing a wearable assistive augmentative
communication (AAC) device that could help a non-communicative person in a
wider variety of contexts, such as in mobile situations like horse riding
therapy.
DAILY JOURNAL
WEEK OF 5/18 - 5/22
THIS WEEK'S GOAL: Get set up, IDs, make
a research plan
5/18/15
I got a desk, ate
food with people, finished reading and taking notes on VizWiz paper, still need
to get bus, library, and key passes. Maybe one or two more
things? I forgot what they were.
Goals for 5/19:
- read more of the research papers listed on
"Research Plans and Ideas" doc with milder notes (the 10 page
VizWiz paper took 3 hours to read and take notes on, I don't know if that's
normal but it seems like a lot.
- focus on/ outline more specific research
project & side task ideas and run them past Shaun
- ((Feel free to add any goals))
5/19/15
Estimating the
viability of sequential image adaptation/analysis (and reading more papers)
seems to be the next step. Basically the idea is developing easy software tools
and steps for adapting sequential images (such as building instructions or
comic media) into scripts for the blind to read or follow. Some parts can be
automated, but others cannot. Also, making comics from video was talked about.
I really want to be a fairer judge of how useful it actually will be
(basically, the idea is in reducing the difficulty of or expediting the process
of translating a certain type of image or visual thing (in this case sequential
images) into an accessible format)... I'm waffling about it. Needs
more work and viability estimation. Might use OCR,
thought about font pairing with OCR. Something that
can identify and read/transcribe multiple fonts? Not handwriting, that
is definitely too difficult. Some of the process can be automated and some can
be human fueled (corrections to computer estimates, what other aspects need to
be noted. But how to judge that?
5/20/15
It was still hard to
say, I couldn't read some of the comics related articles and I only found one I
could read at the literal end of the day. It would probably be done in a .net
windows form or something. In c#.
Major project
desires:
1.
assist with wearable
tech things Halley is doing
2.
cheaper faster
personal Braille printers
3.
developing and
testing more Braille learning games
4.
comics idea might be
main thing if not other things
5/21/15
I have to finish my
research plan outlines for tomorrow, today went pretty well
5/22/15
Worked from home,
finished preparing 6 ideas.
WEEK OF 5/26 - 5/29
THIS WEEK'S GOAL: to be determined (tbd)
5/26/15
Idea #2 was picked
and I started working on Arduino stuff. Also I need
human trial training or something like that.
5/27/15
Did
more Arduino tutorials, ended on 10.
5/28/15
Did the rest of the Arduino tutorials, and met with Shaun and Ariel to talk
about buttons.
5/29/15
Completed
IRB human research subject training.
WEEK OF 6/1 - 6/5
THIS WEEK'S GOAL: prototyping
6/1/15
Read about serial
communication.
6/2/15
Read about I2C
and SPI communication.
6/3/15
Out sick.
6/4/15
Read more tutorials
and papers and sketched more ideas.
6/5/15
Read tutorials for
the ESP8266 Thing.
WEEK OF 6/8 - 6/12
THIS WEEK'S GOAL: prototyping
6/8/15
Tried to get SparkFun ESP8266 Thing to work, couldn't.
6/9/15
Tried again after
more fiddling, couldn't, and went home early.
6/10/15
New parts should be
coming in today; in the meanwhile I attended and took notes on an extremely
good webinar from ACM called "Making the Digital and Physical Worlds
Accessible for People with Disabilities."
6/11/15
Joined slack, did
stuff that didn't feel like much. Figured out we needed to mess with WiFi stuff and Mac addresses. Toured Halley's lab in the
basement and saw a Baxter, then soldered some!
6/12/15
Got Yun working with Temboo example!
I don't know how long that's going to last. I have more resources to read over
in a new document that was made. I know I have to go back to programming with
the Thing (not much progress but now I can read the serial output using Realterm), But I am very excited and confident in the Yun's ability to be a prototyping platform.
WEEK OF 6/15 - 6/18
THIS WEEK'S GOAL: prototyping
6/15/15
Tried to get thing
board to work, a little bit ago maybe last week we were able to finally read
serial messages by using Realterm. You got to the
port tab and un-open the connection, go to pins, clear the DTR pin, go back to
the port stuff, set the baud rate to 115200, set the com to 5 (/vpo) or something like that, and then open the connection
again.
6/16/15
Soldered 2 other
ESP8266 Thing boards, and it finally worked! The first one got up to a point
where it was making a WiFi connection but nothing was
able to connect to it. It was a bad board. The other two boards are good now.
6/17/15
Went out to lunch
with many people to black pepper pho, and also programmed the 3rd thing board
(58 not 82 which is the second one) to be able to ding a website Shaun made
yesterday that increases a number on the page every time it's visited. I also
wrote another program where you can press a button and then press the make a
connection button and it will change the url
to the new one. It just dings the same website though, even though the end of
the url changed. There is
also occasional random reconnecting which is bothersome, what if it does that
all the time? I couldn't make a php web page like
that yet. I also got to make it blink when it connects (I'm going to have it
blink the same # of times you pressed the yellow button to change the url, or make it do that when you
press the red button.) Family also visited which was nice.
6/18/15
Demo'd
the 3 tests for Shaun, started investigating platforms to build phone apps on.
Downloaded Xamarin, and I requested free student license for it. Tried running
a basic Android app, worked in XamarinStudio but not VisualStudio. Needs to be fixed.
WEEK OF 6/23 - 6/26
THIS WEEK'S GOAL: PROTOTYPING
6/23/15
Missed volunteering
at center for the blind in Denver on the 20th because of my cousin's
wedding. I want to see if there is another opportunity to volunteer before I
leave... Performed successful basic programming projects in
both Xamarin and Visual Studio. Started a
build up project, where I am using the old speech synth
but I test when the button is pressed and I compare previous and current #s
gathered from the web page. Thought about using php
code to clear and/or log data the end of every day. Need to check out
(literally, go find the stuff you bookmarked weeks ago) ways to use WiFi or just directly program/communicate between Arduino and android app.
6/24/15
Mostly just
programmed more in Xamarin android and read a bunch
of resources.
6/25/15
Made a mistake, was
working on Xamarin android things when I needed to be
working on Xamarin.Forms things (cross platform was
the big point). To get adjusted faster I translated a huge example, the forms
gallery.
6/26/15
Read examples for
local databases and stuff and started working on the first Xamarin.Forms
test project, which is just basic text to speech, and no memory.
WEEK OF 6/29 - 7/2
THIS WEEK'S GOAL: PROTOTYPING
6/29/15
I fixed this problem
with the WebClient.DownloadString method by using the
dependency service like I did with the text to speech (TTS). Figured
out how to run listening in the background (later moved up tier because the
method was being called by every on TTS object and increasing the number super
fast.)
6/30/15
Programmed in the
ability to manipulate pitch and speed of the TTS but struggled with getting a
local database example to work.
7/1/15
Reproduced local
database stuff and started combining/updating it with the last project's
breakthroughs. Later in the day I was able to get it to a level of basic
functionality and it didn't hit up the webpage so often now. Now, whichever
first one in the list is on, that's the one that's going to be spoken.
7/2/15
Tried working on
getting the conversion from a string to an xml document to work, but the parse
method just kept getting this same error and I couldn't figure out what it was.
Later I worked on image choosing (at the end of the day I think I found a good
example, so I'm going to read that ASAP). I also looked at how people can
record audio, it looks a little complicated but probably workable if we use the
dependency service again.
WEEK OF 7/6 - 7/10
THIS WEEK'S GOAL: PROTOTYPING
(finishing up?)
7/6/15
The problem with
parse was that the string already needed to contain xml code to be able to be parsed
and converted into an Xdocument.
7/7/15
I updated the Arduino code! Might need slightly more reverb room... Also
got all these materials to make button covers, and started the app update and
stuff...
7/8/15
Did a whole big
goose chase and finally figured out how to translate json
files to objects in c#. Dug myself into a hole later because
I waited a long time to test after changing a lot of things in the structure of
the app.
7/9/15
What's going on:
almost done with the app update, just have to make it listen for triggers; then
I have to clean up the code some, then I will make another button or two and I
think tomorrow Halley and I are going to look at making button covers.
7/10/15
We made an outline
of a button cover, but the app runs too slow so I have to look at how to run
the looping thing in the background.
WEEK OF 7/13 - 7/17
THIS WEEK'S GOAL: PROTOTYPING
(finishing up)
7/13/15
Worked on the timer
delay problem and also the tiny ESP board was finally
programmed. I have to work on the audio recording and fix the dang timer
problem...
7/14/15
Worked on time delay
problem more
7/15/15
Fixed the delay and
duplicate problem with the button response time; Also I attended the Dr. Warren
Allen -- "Beyond Platform-centric Social Media Research: Challenges and
Opportunities" talk that happened right outside the office. I started
reading about the audio stuff and doing some initial testing.
7/16/15
Got halfway through
audio stuff and worked on some bugs before I had to go home sick.
7/17/15
I finished adding
audio recording stuff for the android side only. It would likely be really
difficult to program without actually testing on an iPhone,
but you need a Mac to do that.
WEEK OF 7/20 - 7/24
THIS WEEK'S GOAL: PROTOTYPING
(finishing up)
7/20/15
Worked
on adding the chat client in the morning, finished for the most part. I
have to edit my code so it's nice and efficient and not cluttered and well
documented. Then I might want to work on aesthetics, but I am pretty happy with
it now. I have to prepare a to-do list for exiting the operation.
7/21/15
Worked on the schedule
for the last two weeks, solved a paperwork problem, and documented code.
7/22/15
In the morning I
finished up documenting the C# code... and started a code notes document for
it. Had a meeting about case design for the button, figured out we needed to do
a bit more testing with sleep and heat stuff
7/23/15
The parts didn't
come and it was hard just fiddling around with Arduino
code. I visited the BTU lab and picked up some wood stuff with Halley.
7/24/15
The parts came today
and I finished documenting all the code and I put it on a
usb drive (that was hard!). I am refocusing and
trying to prioritize my time better. I started working on the next steps of
scripting and 3-d model building, and I am going to try to get the script(s)
done this weekend.
WEEK OF 7/20 - 7/24
THIS WEEK'S GOAL: PROTOTYPING
(finishing up), doing light usability testing, getting feedback from an OT,
exiting the internship and completing DREU milestones
7/27/15
So far I have tested
Halley's hookup of the board and it works great and I resized my 3D modeled
case to be just the right size. Worked on milestones and GitHub stuff. Checked out a Google
Glass. I also printed something on the 3D printer! It was the button's
box, and it turned out that it was not quite the right size! I had to make it
taller and thinner. Started thinking about how to implement the video. We are
doing some light testing with users tomorrow.
7/28/15
I performed user
testing meetings/interviews with two students this morning, a grad and an
undergrad. I printed off the updated 3D model, and the button fits really well.
I transcribed the notes. I got to try out an oculus rift, it was really
cool and it didn't make me sick.
7/29/15
I performed more
user testing stuff, went out to lunch to talk about PhD programs, and
interviewed the occupational therapist with Halley.
7/30/15
Last day! I transcribed
the interviews, cleaned up my station and made a PowerPoint presentation to
serve as a demo of the app and button.
FINAL REPORT
The final
report for this project can be found here: Wearable
AAC Final Report