Caltech researchers developed software that combines augmented reality with computer vision algorithms to literally “give voice” to objects around a blind person.
CARA, is the name of the cognitive assistant of Augmented Reality that will help blind people move in any room, as they will be able to listen to what is around them.
Professor Markus Meister, owner of the laboratory and scientist involved in the project, describes it as follows:
“Imagine you are in a world where all the objects around you have voices and can speak to you.”
“Can you imagine moving in such a world and accomplishing some tasks for which we normally use our visual system? We did it here: We gave the objects a voice.”
Yang Liu, a PhD student in Neural Computing, was responsible for leading the team of scientists that developed CARA.
CARA uses the well-known HoloLens Augmented Reality glasses from Microsoft, which can scan a person’s environment and identify individual objects such as a laptop or photo.
The system uses Computer Vision to identify objects in the environment, giving them a name and a voice. Objects “say” their name to the user’s commands every time the user points to his address or is near it.
But that’s not all, because thanks to surround sound technology, objects can sound different depending on where they are in the room. For example, if the object is too far to the left of the user, its voice sounds as if it were coming from the left. The closer the object is, the higher the pitch of your “voice” will be.
The development team programmed CARA in different ways to prevent the blind person from being overwhelmed by hearing “so many voices” at once.
Focus mode allows an object to say its name when the user looks at it from the front or directly. When the user turns his head to face multiple objects, each object says its name and the sound of the object’s voice provides an acoustic signal of its relative distance to the user. In this way, a visually impaired user can “look around” to explore his surroundings.
In scan mode, the environment scans from left to right everything HoloLens has in view as the objects say their names.
The target mode, in which the user can select one of the objects.
As we can see in the following video, the tests conducted by the scientists were promising.
However, it is a technology that is still under development and has a disadvantage because it requires an AR helmet that is not very cheap. However, the idea that blind people can move naturally in a shop or in the corridors of their home is certainly not a price constraint.