High School Students Built This iPhone App for the Visually Impaired
Members of a high school machine-learning club have developed an inexpensive smartphone application for those with low or impaired vision.
Eugene Choi, Raffu Khondaker, Irfan Nafi, and Pranav Ravella are seniors at the Thomas Jefferson High School for Science and Technology, in Alexandria, Va.
We noticed that the computer-vision field has gotten to a point where it could detect objects with really good accuracy and describe a scene just like a human would," Choi says. The immediate application we thought of was a visual aid for people with low or no vision."
The student team's Atheia is a mobile app that makes observations about a user's environment to improve spatial awareness and safety. Its facial-recognition feature can identify family and friends. The built-in text reader scans words and reads them out loud. The scan feature recognizes, counts, and pinpoints objects in the user's field of view. The application also tracks its observations, so users can ask where an object was most recently located; Atheia responds with the location, the time it was last identified, and nearby objects.
In addition, the search feature directs users to objects in their environment through haptic feedback and audio instructions. The app's voice assistant verbally describes the identity of people and objects in a user's environment.
We were inspired to improve the accessibility of technology and to further our passion to help those in need."
Atheia can answer open-ended questions about the person's environment, such as the shape and size of objects or the time of day.
Users who feel they are in danger can activate the app's sentry mode through a voice command or by pressing a button. Atheia will start recording video, text live updates to emergency contacts, and provide the contacts with the user's location.
The students began working on the project in 2019. They tried many iterations before they had a working prototype. The app is now being tested by volunteers at Blind Industries and Services of Maryland, a not-for-profit organization in Baltimore that provides training and career resources to the state's visually impaired.
The students' advisor is IEEE Member Pamela Ahn, director of the school's electronics research lab. She is the founder and president of the IEEE Richmond (Va.) Section Women in Engineering affinity group.
These four kids want to make an impact in the world," Ahn says. They want to help other people, and it's from a very unselfish standpoint. They have brilliant minds and are hard workers. They are very diligent and focused."
The Atheia team (from left): Eugene Choi, Irfan Nafi, Raffu Khondaker, and Pranav Ravella.Yeounhee Cha
BUILDING THE TECHNOLOGY
The students developed the app, their own information processing pipeline, and search and obstacle-avoidance algorithms to perform user requests with low latency and high accuracy. The app uses the latest computer-vision models and machine learning techniques to increase the accuracy of how it classifies objects. It also includes a leading object-detection model, a multimodal framework for vision and language research, a visual question-answering program, and a text-recognition algorithm.
The data is processed locally on the mobile device instead of through a cloud service, in order to fulfill user requests regardless of whether they have a cellular signal.
Through our testing, we realized how critical it was to serve our users no matter where they were," Ravella says. It was an immense challenge to convert our huge, heavy computer-vision models into something an iPhone could run. But, thanks to Apple's Neural Engine, it was possible-and worth it."
The app currently is available only on iOS because iPhones are the most popular among those testing the device and the software has more accessibility features, Ravella says. It also allows the students to offload hardware development onto Apple and focus more on the software, because Apple has excellent iPhone cameras, depth technology, processing power, and battery life, he says.
Assistive devices on the market for the blind can cost as much as US $6,000, he says. Atheia users will be charged a monthly subscription fee of around $10, he says.
THE IMPORTANCE OF FEEDBACK
The students' first attempt at an assistive device was a glove with a camera-which Choi says was unsuccessful because it was designed without input from potential users.
We fell into a common pitfall that a lot of [developers] of assistive tech fall into," he says. We created a solution first and then sort of found the exact problem that matched, which isn't really how you're supposed to create these devices or approach any engineering problem."
Because they couldn't conduct in-person interviews due to COVID-19 pandemic restrictions, the students used Facebook to find people with vision problems and ask about their needs. They also consulted with Blind Industries and National Industries for the Blind, an employment placement service in Baltimore for the visually impaired.
Nafi [left] and Choi are gluing together the parts of a 3-D printed prototype bracelet that displayed the Atheia app. It was one of six prototypes the team tested but rejected.Raffu Khondaker
They were willing to give us feedback and [realized] the potential of such a device-which was really inspiring and kept us innovating," Ravella says.
Based on that feedback, the team scrapped the glove and started developing wrist- and head-worn devices and eventually the mobile app. And they made sure to get input about each of their six prototypes.
To learn about which technologies to use, Ahn helped the students get access to research articles from a variety of IEEE publications including IEEE Spectrum.
After spending hundreds of dollars of their own money to buy parts and pay for software, the students sought out partners to help cover the development costs. Their partners include Amazon Web Services, the Maximus Foundation, MIT's assistive technology department, and Ultralytics.
The team also used money it had won at contests and hackathons.
BRIGHT FUTURES
All four students intend to pursue a STEM career.
Ravella says he wants to research the role machine learning can play in cybersecurity. Combining the fields in an internship at MITRE and Columbia University demonstrated to him how easy it was for hackers to break into Internet-of-Things devices, he says.
Nafi says he would like to get involved with computer-vision research that incorporates a mixture of physics." He has worked on digital pathology at Dartmouth.
Choi says he also plans to pursue a career in computer vision. He says he enjoyed the Atheia project as well as another one using computer vision he worked on during an internship program for aspiring scientists at George Mason University, in Fairfax, Va.
Khondaker says he would like to get a degree in artificial intelligence. He liked the client-focused aspect of developing Atheia, he says, because you're meeting the people that you're designing the project for."
You can see how Atheia works from this video the students made.
Atheia Demowww.youtube.com