vOICe — “seeing with the sound” system

The vOICe system provides blind people with an artificial vision, thus enabling them to navigate and walk alone.

Nowadays there are approximately 280 thousand blind people in Russia. Two thirds of them are children. Every moment they have difficulties performing everyday activities such as reading labels, using microwave oven, sorting clothes or starting the washing machine. Even the ordinary tidying up is difficult, resulting in injuries, burns and cuts. Living in our world is extremely hard for them. Those who look after blind people also undergo hardship.

Solutions

Existing solutions do not provide desired capabilities.

Our goal is to enable blind people to walk without support and take care of themselves using the vOICe. Due to continuous scanning of the environment and translating images into the encoded audio signals (which the brain converts into an analog of a black‑and‑white image), it allows blind people to obtain an artificial vision. Using the vOICe they are able to identify large font and moving objects, what gives them the ability to navigate in the room.

The vOICe algorithm

1. Brightness — volume
2. Vertical line — pitch
3. Horizontal line — signal entry delay

vOICe is a unique technology of vision through sound. It does not use ultrasound and echolocation, but is based on the perception of the video camera visual information. Combined with specialized software, it allows blind people using a smartphone, equipped with a camera, to “hear” the image of the environment.

The history of creation

vOICe was developed by Peter Meijer, a scientist at Philips Research Laboratories in the Netherlands. “All things have their own unique sound, and once you understand the principle of the system, you can ‘see’ the things around you”, says Thomas. More brightly colored objects produce the sound of a higher frequency, and a built‑in color identifier name the colour of an object. As for now, the system cannot allow you to “see” fast‑moving cars or to recognize the fine print, but it gives an opportunity to recognize the building, read the signs, and even watch TV. Using the vOICe can be compared to learning a foreign language: after some time users are comfortable with the conversion of the sound into an image and are able to recognize the environment “automatically”, i.e. without hesitation. The software constantly converts images from the camera to the sound complex. For example, a bright spot of light gives a short beep. If this bright spot is on your left, you will hear a sound signal on the left, and vice versa. If the spot is moving, you hear the beep respectively moving. You get two signals for two spots, three signals for three spots and so on. The horizontal line gives a long tone as all its constituent points together produce a pure tone. The vertical line sounds like a sharp noise signal — it is a joint sound of its constituent points located at different heights.

The described method allows you to voice any image, but the richer it is, the more sounds are needed to display it. The entire picture is voiced for a second and its content is updating every second. By ear, these so‑called “soundscapes” represent visual information obtained by scanning the scene from left to right, indicating obstacles by raising the tone, and brightness by changing the volume.

Neurons and perception

The main purpose of the vOICe technology is to provide an alternative to the “raw” visual information, by giving sound complex, which carries visual information and leaving the task of recognition to the human brain.

What are we going to do with raised money?

We want to make five vOICe system prototypes and give them to blind people with the purpose of long‑term operation and system testing in Russia. We are going to conduct the necessary training and make blind people motivated to use the vOICe, help them to get used to it. This “test drive” will provide us with the feedback, helping us to make it even more comfortable for future users. Together we will prove the usefulness and functionality of the system. In the future, we want to make the vOICe a social project that will enhance the integration of the visually impaired in our society.

The complete vOICe system will consist of a pair of video glasses with a Wi‑Fi transmitter joint to a mini‑computer via radio. Mini‑PC will be attached to a belt with a compact power source. The overhead headphones that also allow hearing surrounding sounds will be used for sound transmitting.

Rewards

Taking part in the master class of using the vOICe prototype (get the opportunity to see the world through the sound)

A smart T‑shirt

vOICe mobile app

Cards for self-study

“Sound all-seeing” figurine

Who are we and why are we doing this?

Project team:

Igor Trapeznikov — a neurophysiologist, post-graduate student of Sholokhov Moscow State University for the Humanities, IT project manager and active member of Russian Transhumanist Movement;

Valerija Udalova — an engineer, sociologist, member of the Association of futurologists, expert member of the Central Committee of All Russia Public Movement “Russia — active longevity”;

Iana Artishcheva — a software engineer, winner of the “Archimedes—2009”, inventor of “AR‑Go: augmented reality navigator” device, award winner of “Lenovo—2011”;

Anatolij Deev — a psychologist, candidate of Biological Sciences, chairman of the Moscow department of the Russian Gerontological Society, professor at Moscow State Psychological‑Pedagogical University’s Medical Biophysics Department;

Project partner:

All Russia Association of the Blind

vOICe users’ reviews

A blind woman over forty years is a vOICe user. She became blind nearly twenty years ago as a result of the chemical explosion. She has been using a sound display for several months without taking off the video camera and headphones.

“As someone who was able to see before, I can say that soundscapes (sound overview) once again gave me a sense of vision. There could be a born‑blind person in my place. I was able to imagine and see, how if I had eyes, how my hall would look like, how the curtain on the left of my three windows would look like. In general, the vOICe and a webcam turned the vague “mental image” of the hall in what seems like a real three‑dimensional vision. Now I am sitting in front of my computer and see the flow of colour on the screen, but it is the vOICe that tells me that I am looking at a square object. It is like looking at the world through the stained glasses. You can see the shape and location of objects around you, but cannot distinguish tiny details. I do not know whether it is a particular human language, but it is precisely the language that tells my mind, what I see now. Call it whatever you like, but it works! I cannot do without it anymore! Thank you for this invention!”

Three months later, “I noticed two things. Firstly, as I said, I wear the device on a daily basis. So yesterday, while I was washing the dishes. I was stunned by the fact that I saw the depth of the sink as an image. I stepped back to be certain that it was not the purely mental illusion. No, the feeling of sink depth did not disappear.”

Five months later, “When I started to learn the vOICe in a scanning mode, I got a simplified sound overview, which was difficult to interpret and understand. Then I began successfully use the vOICe mobile app with a laptop. I remember that at first I was walking through the house and identifying objects by soundscapes. A few months later, I was not focused on changing sounds and perceive them as a background, translating the incoming information into the image. Gradually, I noticed that the sound overview not only informs me about the approach to some furniture, but also provides the information about the surrounding area and its depth.

I can stay in the doorway, look at the rooms and feel the depth. As I was not blind before, I remember the perception of the space. The only difference between the images that you get using the vOICe to that you get by means of your eyes is that you cannot see colours and tiny details. Now, I even do not think about what I hear — I just feel like the program is a part of me.”

Dear friend, please take part in the vOICe project, support the creating of the vision with the sound system, give blind people the opportunity of saying “Hi! I see you!”