Lumos

Friday. November 10, 2017 - 3 mins

Summary

A lightweight aid for the visually impaired.

Inspiration

Coming from the crowded cities of Bangalore and Mumbai, we reflected on how incredibly difficult it must be for a visually challenged person to navigate the world around them.

This thought grew into an idea for a product to help the blind to interact with the environment around them, at minimal cost.

We built Lumos, an android application that assists the visually impaired by describing their surroundings in a meaningful manner.

What it does

Lumos helps visually impaired people experience the world around them, seamlessly.

The user points their camera at any object in their surroundings. Lumos records an image, analyzes it, and outputs a small description of the image in the form of audio.

If there is something that the user needs to be particularly aware of, such as traffic or a stairway, Lumos will warn them of it.

How we built it

Our front-end consists of a lightweight android application listening for a click. Once a picture is taken, the image processing and analysis is handled by our backend infrastructure.

Using the Google Cloud Vision API, we detect relevant objects in the image before assigning our own relevance score to them. Post this, we generate a relevant phrase that we predict might be useful to the user and make use of Text to Speech APIs to relay this information back to the user in the form of a short audio clip.

Accomplishments that we’re proud of

When we started the project, we decided to keep two broad concepts in mind: efficiency and simplicity.

We’re proud to say this is reflected in our entire application. Our application runs with minimal latency and has an extremely simple and intuitive user experience.

We accomplished the first by using compression algorithms for the images that are taken, thus reducing the time for API calls. We also store nothing on the user’s phone.

The second we accomplished by sticking to our minimal and simple design principles, only having one screen and no complex sequence of commands, thereby creating a lightweight and user-friendly application.

What we learned

We learned that building an application that uses external APIs and must work in real time is an incredibly challenging task. It is a careful balance between an enhanced user experience and an app that’s just to cumbersome to use.

Toeing this line, is something that we learned was not only an engineering challenge, but a design challenge as well. We learnt the importance of simplistic,logical design that meets the user’s needs.

What’s next for Lumos

We believe that Lumos has a lot of potential for growth and that is why it was so important for us to build a basic working prototype.

We plan to incorporate a more sophisticated backend infrastructure for detecting objects in images as well as rendering and organizing this information more effectively. We are also thinking about ways to pre-process data in a better way, thus keeping latency time low and preserving as much useful information in the data as possible.

Another huge addition we hope to incorporate soon, is to make Lumos more personalized to the user, detecting faces of friends and family that the user meets etc. We also want to include text detection, so a user pointing the camera at text, can have it read out to them.