Researchers at the Institute of Neuroinformatics are developing the neuroprostheses of the future. Their aim is to restore the vision of blind people by feeding images from a camera directly into the brain using electrodes.
Text: Stefan Stöcklin
Pictures: Diana Ulrich
Translation: Philip Isler
The human eye is a marvelous thing. A tiny pit in the center of the yellow spot on the retina allows us to see clearly and sharply – and read these words. Measuring only one-and-a-half millimeters, this microscopic little depression, or fovea, at the back of the eye is where the concentration of light-sensitive photoreceptor cells is highest. This is where some 150,000 densely packed cone cells respond to the incoming light. Each of these cones is connected to a ganglion cell, which relays the signals to the visual cortex of the brain. The nerve signals traveling to the thalamus make it possible for us to see the world with our eyes. Any damage to the fovea, or any other part of the retina with its several million cones and rods, is very likely to result in serious loss of vision.
This is where Shih-Chii Liu comes in. She wants to help people whose retina or optic nerve is impaired and who are thus no longer able to see the world around them. The neuromorphic engineer and group leader at the Institute of Neuroinformatics of UZH and ETH Zurich is heading up the European NeuraViPeR project, which aims to develop a neuroprosthesis for the blind. “We’re not yet able to electronically replicate the incredible abilities of the human eye,” the professor of neuromorphic systems and sensors readily admits. “But we hope that thanks to our work, blind people will be able to perceive at least a basic representation of their surroundings.”
Silicon Valley background
Liu’s office can be found in one of the buildings on Irchel Campus. It is stacked with computers, electrical components and piles of manuscripts, as well as a box holding computer chips whose development she oversaw. The scientist has a strong Silicon Valley background. She earned her PhD in computation and neural systems at the California Institute of Technology in 1997. One year later, she joined the newly founded Institute of Neuroinformatics at UZH and ETH, together with Tobi Delbruck, who specializes in revolutionary sensors. His event-based sensors are modeled on the human eye in that rather than capturing images at a fixed rate like conventional cameras, they measure changes in brightness as they occur, which results in an asynchronous stream of events. This highly efficient technology will also feature in the planned neuroprostheses.
Bypassing the eyes
“If the retina or optical nerve no longer work, visual data from the surroundings have to be fed directly into the cerebral cortex,” says Liu, explaining the basic principle of the planned neuroprosthesis. This is achieved through electrodes that are implanted in the visual cortex at the back of the brain. Without visual information from the eyes, the signals required for this process instead come from a camera observing the surroundings. The electrodes then stimulate the neurons according to the visual input from the camera and generate an image of the environment – bypassing the eyes and optical nerve, so to speak.
The method works, as Pieter Roelfsema of the Netherlands Institute for Neuroscience in Amsterdam, who is a partner in the project, recently demonstrated in a widely noted animal study. His team stimulated two macaques’ visual cortex using a total of 1,024 needle-like electrodes in each monkey, enabling them to recognize the shapes of letters. Rather than being placed only on the surface, these electrodes reached 1.5 millimeters into the visual cortex. This enabled Roelfsema’s group to generate perception with even only the weakest of currents. The monkeys were previously trained to recognize letter shapes on a computer screen. “Pieter’s research is very encouraging, and we’re positive our approach will work,” says Shih-Chii Liu.
It’s clear that 1,024 electrodes won’t come close to matching the resolution of the human eye, with its millions of photoreceptor cells in the retina. But this stimulation of the visual cortex does result in a simple form of artificial sight, with test subjects able to perceive dots that appear as light, also known as phosphenes. “Every electrode generates the perception of a phosphene,” says Liu. The more phosphenes there are, the more nuanced the resulting image. One important part of the project involves developing superfine biocompatible electrodes that can precisely stimulate individual neurons. These implantable electrodes need to be wireless and capable of simultaneously sending and receiving signals to provide the whole system with feedback from the cortex on the generated patterns. The electrodes are being developed by researchers at the Albert Ludwig University of Freiburg (Germany).
Shih-Chii Liu, Tobi Delbruck and their team are currently working on the components at the other end of the neuroprosthesis, as it were, where sensory data is captured by a camera and then processed. “We’re developing the neuromorphic hardware and the right algorithms,” Liu explains. Neuromorphic circuits are modeled on biological neural networks. They are both analog and digital in nature, like the human nervous system, and are energy-efficient. One of the greatest challenges facing the researchers is to quickly and efficiently transform sensory data in small circuits into simple yet recognizable stimulation patterns for the electrodes.
The visual input from the camera has to be stripped down to the essential data so that a blind person can understand it. An inner city scene of a tram, cars and people, for example, is reduced into a simple pattern of dots and squiggles, presented by the electrodes using phosphenes (see box). Liu and her team are using deep learning methods for this, training artificial neural networks so that they only extract the most important visual information from pictures of everyday situations.
The neuroprosthesis is a joint project involving research groups from six different universities and organizations from across Europe. Liu reckons it will take another one to two years until all the parts of the prosthesis have been developed and trials with blind people can kick off. These trials will likely take place in Spain, with volunteers who will be gradually prepared to recognize phosphene patterns in the brain.
Only then will it become clear whether the project can deliver on the researchers’ high expectations – restoring a rudimentary form of vision in blind people using electrical stimulation. This would be a spectacular success, even if the phosphene patterns won’t match the vision of a healthy human eye. “The retina,” says electrical engineer Liu, “is a biological marvel that we simply can’t replicate.”