Controling Robots with the Mind

September 16, 2002

Controlling Robots with the Mind

People with nerve or limb injuries may one day be able to command wheelchairs, prosthetics and even paralyzed arms and legs by "thinking them through" the motions

By Miguel A. L. Nicolelis and John K. Chapin

Belle, our tiny owl monkey, was seated in her special chair inside a soundproof chamber at our Duke University laboratory. Her right hand grasped a joystick as she watched a horizontal series of lights on a display panel. She knew that if a light suddenly shone and she moved the joystick left or right to correspond to its position, a dispenser would send a drop of fruit juice into her mouth. She loved to play this game. And she was good at it.
Belle wore a cap glued to her head. Under it were four plastic connectors. The connectors fed arrays of microwires--each wire finer than the finest sewing thread--into different regions of Belle's motor cortex, the brain tissue that plans movements and sends instructions for enacting the plans to nerve cells in the spinal cord. Each of the 100 microwires lay beside a single motor neuron. When a neuron produced an electrical discharge--an "action potential"--the adjacent microwire would capture the current and send it up through a small wiring bundle that ran from Belle's cap to a box of electronics on a table next to the booth. The box, in turn, was linked to two computers, one next door and the other half a country away.

In a crowded room across the hall, members of our research team were getting anxious. After months of hard work, we were about to test the idea that we could reliably translate the raw electrical activity in a living being's brain--Belle's mere thoughts--into signals that could direct the actions of a robot. Unknown to Belle on this spring afternoon in 2000, we had assembled a multijointed robot arm in this room, away from her view, that she would control for the first time. As soon as Belle's brain sensed a lit spot on the panel, electronics in the box running two real-time mathematical models would rapidly analyze the tiny action potentials produced by her brain cells. Our lab computer would convert the electrical patterns into instructions that would direct the robot arm. Six hundred miles north, in Cambridge, Mass., a different computer would produce the same actions in another robot arm, built by Mandayam A. Srinivasan, head of the Laboratory for Human and Machine Haptics (the Touch Lab) at the Massachusetts Institute of Technology. At least, that was the plan.

If we had done everything correctly, the two robot arms would behave as Belle's arm did, at exactly the same time. We would have to translate her neuronal activity into robot commands in just 300 milliseconds--the natural delay between the time Belle's motor cortex planned how she should move her limb and the moment it sent the instructions to her muscles. If the brain of a living creature could accurately control two dissimilar robot arms--despite the signal noise and transmission delays inherent in our lab network and the error-prone Internet--perhaps it could someday control a mechanical device or actual limbs in ways that would be truly helpful to people.

Finally the moment came. We randomly switched on lights in front of Belle, and she immediately moved her joystick back and forth to correspond to them. Our robot arm moved similarly to Belle's real arm. So did Srinivasan's. Belle and the robots moved in synchrony, like dancers choreographed by the electrical impulses sparking in Belle's mind. Amid the loud celebration that erupted in Durham, N.C., and Cambridge, we could not help thinking that this was only the beginning of a promising journey.

In the two years since that day, our labs and several others have advanced neuroscience, computer science, microelectronics and robotics to create ways for rats, monkeys and eventually humans to control mechanical and electronic machines purely by "thinking through," or imagining, the motions. Our immediate goal is to help a person who has been paralyzed by a neurological disorder or spinal cord injury, but whose motor cortex is spared, to operate a wheelchair or a robotic limb. Someday the research could also help such a patient regain control over a natural arm or leg, with the aid of wireless communication between implants in the brain and the limb. And it could lead to devices that restore or augment other motor, sensory or cognitive functions.

The big question is, of course, whether we can make a practical, reliable system. Doctors have no means by which to repair spinal cord breaks or damaged brains. In the distant future, neuroscientists may be able to regenerate injured neurons or program stem cells (those capable of differentiating into various cell types) to take their place. But in the near future, brain-machine interfaces (BMIs), or neuroprostheses, are a more viable option for restoring motor function. Success this summer with macaque monkeys that completed different tasks than those we asked of Belle has gotten us even closer to this goal.

From Theory to Practice
Recent advances in brain-machine interfaces are grounded in part on discoveries made about 20 years ago. In the early 1980s Apostolos P. Georgopoulos of Johns Hopkins University recorded the electrical activity of single motor-cortex neurons in macaque monkeys. He found that the nerve cells typically reacted most strongly when a monkey moved its arm in a certain direction. Yet when the arm moved at an angle away from a cell's preferred direction, the neuron's activity didn't cease; it diminished in proportion to the cosine of that angle. The finding showed that motor neurons were broadly tuned to a range of motion and that the brain most likely relied on the collective activity of dispersed populations of single neurons to generate a motor command.

There were caveats, however. Georgopoulos had recorded the activity of single neurons one at a time and from only one motor area. This approach left unproved the underlying hypothesis that some kind of coding scheme emerges from the simultaneous activity of many neurons distributed across multiple cortical areas. Scientists knew that the frontal and parietal lobes--in the forward and rear parts of the brain, respectively--interacted to plan and generate motor commands. But technological bottlenecks prevented neurophysiologists from making widespread recordings at once. Furthermore, most scientists believed that by cataloguing the properties of neurons one at a time, they could build a comprehensive map of how the brain works--as if charting the properties of individual trees could unveil the ecological structure of an entire forest!

Fortunately, not everyone agreed. When the two of us met 14 years ago at Hahnemann University, we discussed the challenge of simultaneously recording many single neurons. By 1993 technological breakthroughs we had made allowed us to record 48 neurons spread across five structures that form a rat's sensorimotor system--the brain regions that perceive and use sensory information to direct movements.

Crucial to our success back then--and since--were new electrode arrays containing Teflon-coated stainless-steel microwires that could be implanted in an animal's brain. Neurophysiologists had used standard electrodes that resemble rigid needles to record single neurons. These classic electrodes worked well but only for a few hours, because cellular compounds collected around the electrodes' tips and eventually insulated them from the current. Furthermore, as the subject's brain moved slightly during normal activity, the stiff pins damaged neurons. The microwires we devised in our lab (later produced by NBLabs in Denison, Tex.) had blunter tips, about 50 microns in diameter, and were much more flexible. Cellular substances did not seal off the ends, and the flexibility greatly reduced neuron damage. These properties enabled us to produce recordings for months on end, and having tools for reliable recording allowed us to begin developing systems for translating brain signals into commands that could control a mechanical device.

With electrical engineer Harvey Wiggins, now president of Plexon in Dallas, and with Donald J. Woodward and Samuel A. Deadwyler of Wake Forest University School of Medicine, we devised a small "Harvey box" of custom electronics, like the one next to Belle's booth. It was the first hardware that could properly sample, filter and amplify neural signals from many electrodes. Special software allowed us to discriminate electrical activity from up to four single neurons per microwire by identifying unique features of each cell's electrical discharge.


A Rat's Brain Controls a Lever
In our next experiments at Hahnemann in the mid-1990s, we taught a rat in a cage to control a lever with its mind. First we trained it to press a bar with its forelimb. The bar was electronically connected to a lever outside the cage. When the rat pressed the bar, the outside lever tipped down to a chute and delivered a drop of water it could drink.

We fitted the rat's head with a small version of the brain-machine interface Belle would later use. Every time the rat commanded its forelimb to press the bar, we simultaneously recorded the action potentials produced by 46 neurons. We had programmed resistors in a so-called integrator, which weighted and processed data from the neurons to generate a single analog output that predicted very well the trajectory of the rat's forelimb. We linked this integrator to the robot lever's controller so that it could command the lever.

Once the rat had gotten used to pressing the bar for water, we disconnected the bar from the lever. The rat pressed the bar, but the lever remained still. Frustrated, it began to press the bar repeatedly, to no avail. But one time, the lever tipped and delivered the water. The rat didn't know it, but its 46 neurons had expressed the same firing pattern they had in earlier trials when the bar still worked. That pattern prompted the integrator to put the lever in motion.

After several hours the rat realized it no longer needed to press the bar. If it just looked at the bar and imagined its forelimb pressing it, its neurons could still express the firing pattern that our brain-machine interface would interpret as motor commands to move the lever. Over time, four of six rats succeeded in this task. They learned that they had to "think through" the motion of pressing the bar. This is not as mystical at it might sound; right now you can imagine reaching out to grasp an object near you--without doing so. In similar fashion, a person with an injured or severed limb might learn to control a robot arm joined to a shoulder.

A Monkey's Brain Controls a Robot Arm
We were thrilled with our rats' success. It inspired us to move forward, to try to reproduce in a robotic limb the three-dimensional arm movements made by monkeys--animals with brains far more similar to those of humans. As a first step, we had to devise technology for predicting how the monkeys intended to move their natural arms.

At this time, one of us (Nicolelis) moved to Duke and established a neurophysiology laboratory there. Together we built an interface to simultaneously monitor close to 100 neurons, distributed across the frontal and parietal lobes. We proceeded to try it with several owl monkeys. We chose owl monkeys because their motor cortical areas are located on the surface of their smooth brain, a configuration that minimizes the surgical difficulty of implanting microwire arrays. The microwire arrays allowed us to record the action potentials in each creature's brain for several months.

In our first experiments, we required owl monkeys, including Belle, to move a joystick left or right after seeing a light appear on the left or right side of a video screen. We later sat them in a chair facing an opaque barrier. When we lifted the barrier they saw a piece of fruit on a tray. The monkeys had to reach out and grab the fruit, bring it to their mouth and place their hand back down. We measured the position of each monkey's wrist by attaching fiber-optic sensors to it, which defined the wrist's trajectory.

Further analysis revealed that a simple linear summation of the electrical activity of cortical motor neurons predicted very well the position of an animal's hand a few hundred milliseconds ahead of time. This discovery was made by Johan Wessberg of Duke, now at the Gothenburg University in Sweden. The main trick was for the computer to continuously combine neuronal activity produced as far back in time as one second to best predict movements in real time.


As our scientific work proceeded, we acquired a more advanced Harvey box from Plexon. Using it and some custom, real-time algorithms, our computer sampled and integrated the action potentials every 50 to 100 milliseconds. Software translated the output into instructions that could direct the actions of a robot arm in three-dimensional space. Only then did we try to use a BMI to control a robotic device. As we watched our multijointed robot arm accurately mimic Belle's arm movements on that inspiring afternoon in 2000, it was difficult not to ponder the implausibility of it all. Only 50 to 100 neurons randomly sampled from tens of millions were doing the needed work.

Later mathematical analyses revealed that the accuracy of the robot movements was roughly proportional to the number of neurons recorded, but this linear relation began to taper off as the number increased. By sampling 100 neurons we could create robot hand trajectories that were about 70 percent similar to those the monkeys produced. Further analysis estimated that to achieve 95 percent accuracy in the prediction of one-dimensional hand movements, as few as 500 to 700 neurons would suffice, depending on which brain regions we sampled. We are now calculating the number of neurons that would be needed for highly accurate three-dimensional movements. We suspect the total will again be in the hundreds, not thousands.

These results suggest that within each cortical area, the "message" defining a given hand movement is widely disseminated. This decentralization is extremely beneficial to the animal: in case of injury, the animal can fall back on a huge reservoir of redundancy. For us researchers, it means that a BMI neuroprosthesis for severely paralyzed patients may require sampling smaller populations of neurons than was once anticipated.

We continued working with Belle and our other monkeys after Belle's successful experiment. We found that as the animals perfected their tasks, the properties of their neurons changed--over several days or even within a daily two-hour recording session. The contribution of individual neurons varied over time. To cope with this "motor learning," we added a simple routine that enabled our model to reassess periodically the contribution of each neuron. Brain cells that ceased to influence the predictions significantly were dropped from the model, and those that became better predictors were added. In essence, we designed a way to extract from the brain a neural output for hand trajectory. This coding, plus our ability to measure neurons reliably over time, allowed our BMI to represent Belle's intended movements accurately for several months. We could have continued, but we had the data we needed.

It is important to note that the gradual changing of neuronal electrical activity helps to give the brain its plasticity. The number of action potentials a neuron generates before a given movement changes as the animal undergoes more experiences. Yet the dynamic revision of neuronal properties does not represent an impediment for practical BMIs. The beauty of a distributed neural output is that it does not rely on a small group of neurons. If a BMI can maintain viable recordings from hundreds to thousands of single neurons for months to years and utilize models that can learn, it can handle evolving neurons, neuronal death and even degradation in electrode-recording capabilities.


Exploiting Sensory Feedback
Belle proved that a bmi can work for a primate brain. But could we adapt the interface to more complex brains? In May 2001 we began studies with three macaque monkeys at Duke. Their brains contain deep furrows and convolutions that resemble those of the human brain.

We employed the same BMI used for Belle, with one fundamental addition: now the monkeys could exploit visual feedback to judge for themselves how well the BMI could mimic their hand movements. We let the macaques move a joystick in random directions, driving a cursor across a computer screen. Suddenly a round target would appear somewhere on the screen. To receive a sip of fruit juice, the monkey had to position the cursor quickly inside the target--within 0.5 second--by rapidly manipulating the joystick.


The first macaque to master this task was Aurora, an elegant female who clearly enjoyed showing off that she could hit the target more than 90 percent of the time. For a year, our postdoctoral fellows Roy Crist and Jos� Carmena recorded the activity of up to 92 neurons in five frontal and parietal areas of Aurora's cortex.

Once Aurora commanded the game, we started playing a trick on her. In about 30 percent of the trials we disabled the connection between the joystick and the cursor. To move the cursor quickly within the target, Aurora had to rely solely on her brain activity, processed by our BMI. After being puzzled, Aurora gradually altered her strategy. Although she continued to make hand movements, after a few days she learned she could control the cursor 100 percent of the time with her brain alone. In a few trials each day during the ensuing weeks Aurora didn't even bother to move her hand; she moved the cursor by just thinking about the trajectory it should take.

That was not all. Because Aurora could see her performance on the screen, the BMI made better and better predictions even though it was recording the same neurons. Although much more analysis is required to understand this result, one explanation is that the visual feedback helped Aurora to maximize the BMI's reaction to both brain and machine learning. If this proves true, visual or other sensory feedback could allow people to improve the performance of their own BMIs.

We observed another encouraging result. At this writing, it has been a year since we implanted the microwires in Aurora's brain, and we continue to record 60 to 70 neurons daily. This extended success indicates that even in a primate with a convoluted brain, our microwire arrays can provide long-term, high-quality, multichannel signals. Although this sample is down from the original 92 neurons, Aurora's performance with the BMI remains at the highest levels she has achieved.

We will make Aurora's tasks more challenging. In May we began modifying the BMI to give her tactile feedback for new experiments that are now beginning. The BMI will control a nearby robot arm fitted with a gripper that simulates a grasping hand. Force sensors will indicate when the gripper encounters an object and how much force is required to hold it. Tactile feedback--is the object heavy or light, slick or sticky?--will be delivered to a patch on Aurora's skin embedded with small vibrators. Variations in the vibration frequencies should help Aurora figure out how much force the robot arm should apply to, say, pick up a piece of fruit, and to hold it as the robot brings it back to her. This experiment might give us the most concrete evidence yet that a person suffering from severe paralysis could regain basic arm movements through an implant in the brain that communicated over wires, or wirelessly, with signal generators embedded in a limb.

If visual and tactile sensations mimic the information that usually flows between Aurora's own arm and brain, long-term interaction with a BMI could possibly stimulate her brain to incorporate the robot into its representations of her body--schema known to exist in most brain regions. In other words, Aurora's brain might represent this artificial device as another part of her body. Neuronal tissue in her brain might even dedicate itself to operating the robot arm and interpreting its feedback.

To test whether this hypothesis has merit, we plan to conduct experiments like those done with Aurora, except that an animal's arm will be temporarily anesthetized, thereby removing any natural feedback information. We predict that after a transition period, the primate will be able to interact with the BMI just fine. If the animal's brain does meld the robot arm into its body representations, it is reasonable to expect that a paraplegic's brain would do the same, rededicating neurons that once served a natural limb to the operation of an artificial one.

Each advance shows how plastic the brain is. Yet there will always be limits. It is unlikely, for example, that a stroke victim could gain full control over a robot limb. Stroke damage is usually widespread and involves so much of the brain's white matter--the fibers that allow brain regions to communicate--that the destruction overwhelms the brain's plastic capabilities. This is why stroke victims who lose control of uninjured limbs rarely regain it.


Reality Check
Good news notwithstanding, we researchers must be very cautious about offering false hope to people with serious disabilities. We must still overcome many hurdles before BMIs can be considered safe, reliable and efficient therapeutic options. We have to demonstrate in clinical trials that a proposed BMI will offer much greater well-being while posing no risk of added neurological damage.

Surgical implantation of electrode arrays will always be of medical concern, for instance. Investigators need to evaluate whether highly dense microwire arrays can provide viable recordings without causing tissue damage or infection in humans. Progress toward dense arrays is already under way. Duke electronics technician Gary Lehew has designed ways to increase significantly the number of microwires mounted in an array that is light and easy to implant. We can now implant multiple arrays, each of which has up to 160 microwires and measures five by eight millimeters, smaller than a pinky fingernail. We recently implanted 704 microwires across eight cortical areas in a macaque and recorded 318 neurons simultaneously.

In addition, considerable miniaturization of electronics and batteries must occur. We have begun collaborating with Jos� Carlos Pr�ncipe of the University of Florida to craft implantable microelectronics that will embed in hardware the neuronal

pattern recognition we now do with software, thereby eventually freeing the BMI from a computer. These microchips will thus have to send wireless control data to robotic actuators. Working with Patrick D. Wolf's lab at Duke, we have built the first wireless "neurochip" and beta-tested it with Aurora. Seeing streams of neural activity flash on a laptop many meters away from Aurora--broadcast via the first wireless connection between a primate's brain and a computer--was a delight.

More and more scientists are embracing the vision that BMIs can help people in need. In the past year, several traditional neurological laboratories have begun to pursue neuroprosthetic devices. Preliminary results from Arizona State University, Brown University and the California Institute of Technology have recently appeared. Some of the studies provide independent confirmation of the rat and monkey studies we have done. Researchers at Arizona State basically reproduced our 3-D approach in owl monkeys and showed that it can work in rhesus monkeys too. Scientists at Brown enabled a rhesus macaque monkey to move a cursor around a computer screen. Both groups recorded 10 to 20 neurons or so per animal. Their success further demonstrates that this new field is progressing nicely.

The most useful BMIs will exploit hundreds to a few thousand single neurons distributed over multiple motor regions in the frontal and parietal lobes. Those that record only a small number of neurons (say, 30 or fewer) from a single cortical area would never provide clinical help, because they would lack the excess capacity required to adapt to neuronal loss or changes in neuronal responsiveness. The other extreme--recording millions of neurons using large electrodes--would most likely not work either, because it might be too invasive.

Noninvasive methods, though promising for some therapies, will probably be of limited use for controlling prostheses with thoughts. Scalp recording, called electroencephalography (EEG), is a noninvasive technique that can drive a different kind of brain-machine interface, however. Niels Birbaumer of the University of T�bingen in Germany has successfully used EEG recordings and a computer interface to help patients paralyzed by severe neurological disorders learn how to modulate their EEG activity to select letters on a computer screen, so they can write messages. The process is time-consuming but offers the only way for these people to communicate with the world. Yet EEG signals cannot be used directly for limb prostheses, because they depict the average electrical activity of broad populations of neurons; it is difficult to extract from them the fine variations needed to encode precise arm and hand movements.


Despite the remaining hurdles, we have plenty of reasons to be optimistic. Although it may be a decade before we witness the operation of the first human neuroprosthesis, all the amazing possibilities crossed our minds that afternoon in Durham as we watched the activity of Belle's neurons flashing on a computer monitor. We will always remember our sense of awe as we eavesdropped on the processes by which the primate brain generates a thought. Belle's thought to receive her juice was a simple one, but a thought it was, and it commanded the outside world to achieve her very real goal.




--------------------------------------------------------------------------------
Miguel A. L. Nicolelis and John K. Chapin have collaborated for more than a decade. Nicolelis, a native of Brazil, received his M.D. and Ph.D. in neurophysiology from the University of S�o Paulo. After postdoctoral work at Hahnemann University, he joined Duke University, where he now co-directs the Center for Neuroengineering and is professor of neurobiology, biomedical engineering, and psychological and brain sciences. Chapin received his Ph.D. in neurophysiology from the University of Rochester and has held faculty positions at the University of Texas and the MCP Hahnemann University School of Medicine (now Drexel University College of Medicine). He is currently professor of physiology and pharmacology at the State University of New York Downstate Medical Center.


� 1996-2002 Scientific American, Inc. All rights reserved.
Reproduction in whole

uh hmmm,
Did anyone here see the movie "Firefox" with clint eastwood? its a plane controlled with the mind, same thing in macross plus.

I think it would be really possible to adapt that to a human and make him controll an airplane, say, a F-22!

Imagine an F-22 fitted with a hydrogen fusion driven turbine, a Brain Control Device and nanotechnology avionics!

It would be easily 90 years in advance!

neural induction is a great thing... it must be 20 years before it will have some real life applications for humans though

Steven Hawking must be watching these developments veeeeeery closely...

If ever there was a technology capable of being abused this is it.... I don't know it is cool and all but it is also scary. I mean yeah prosthetics and all but you know that is not where it is going to stop.

We can only hope they dont stop.... Link your mind to all kinds of things Cars, boats, computers, games, Tv, radio, anything, And giant killer robots. moohahahah...

Just make sure the user has a way to switch it all off inside his mind. So people cant force a link with them.

also would have to have a some way to control voltage spikes well. Nothing would be as bad if the system worked on like 1.5 volts and some how a hacker chunked 9 volts down the line and heated up the wires in your brain. Whats that smell?..... aaaa cra* my brain is on fire...

No need for electrode interfaces anymore. It has come out of the "dark ages" according to this article.

www.betterhumans.com...
Lasers may allow brain-machine interfaces that far exceed the capabilities of existing devices, permitting people to control artificial limbs through brain-linked fiber optic cables.

So suggest researchers at Vanderbilt University in Nashville, Tennessee who have used a low-intensity infrared laser light to successfully bring nerves to life in rats without touching actual nerve cells.

"This technique brings nerve stimulation out of the Dark Ages," says Anita Mahadevan-Jansen, an assistant professor of biomedical engineering at Vanderbilt.

nice very nice


but I would also love to have it so that you could get feedback inside your brain. or have a image pop up in your vision or hear a phone call inside your head... 8)

Originally posted by sardion2000
No need for electrode interfaces anymore. It has come out of the "dark ages" according to this article.

www.betterhumans.com...
Lasers may allow brain-machine interfaces that far exceed the capabilities of existing devices, permitting people to control artificial limbs through brain-linked fiber optic cables.

So suggest researchers at Vanderbilt University in Nashville, Tennessee who have used a low-intensity infrared laser light to successfully bring nerves to life in rats without touching actual nerve cells.

"This technique brings nerve stimulation out of the Dark Ages," says Anita Mahadevan-Jansen, an assistant professor of biomedical engineering at Vanderbilt.

This is pretty cool. Actually, I went to the opening of the Arizona Biodesign Institute last night. There is a researcher there who has monkey's controlling the cursor of a mouse via an implant. The fiber optic thing is really cutting edge though. I have to admit being impressed by the ability of the monkey to control this cursor with it's brain via this tiny implant. They've actually got a prosthetic device that they are preparing to try out in the very near future.

Yes I have seen that on TV. The demonstration was amazing I really would like to find out what the capabilities of these fiber optics are on brain matter. What are the limitations?

Originally posted by sardion2000
Yes I have seen that on TV. The demonstration was amazing

I really would like to find out what the capabilities of these fiber optics are on brain matter. What are the limitations?

What are the limitations? You've got me swinging, but maybe size. The ideal thing would be to place FO cables right at the synaptic junctions. If the fiber optics were 'tuned' to detect neurotransmitter release, it's conceivable that they could function as effectively as neural receptors. To a certain extent I suppose you're limited by the properties of FO's themselves. Your limited to chemistries and detection or signaling systems based on or otherwise involving light... maybe not even necessarily light, but EM energy... not too sure of the properties of FO's though. Of course turning fiber optics into feedback generating systems would be much more complex. For example, how could we restore a sense of touch in a prosthetic? Is it even conceivable? I would say yes it probably is... at least at a rudimentary level. Generating neural information and transmitting it could be one limitation of such a technology. Of course, we may be able to combine FO neural networks coupled with some sort of 'nanodevice' that simulates or stimulates feedback from the prosthetic arm... It's probably doable, but a ways off.