Eric Leuthardt believes that in the near future we will allow doctors to insert electrodes into our brains so we can communicate directly with computers and each other.
It’s the Monday morning following the opening weekend of the movie Blade Runner 2049, and Eric C. Leuthardt is standing in the center of a floodlit operating room clad in scrubs and a mask, hunched over an unconscious patient.
“I thought he was human, but I wasn’t sure,” Leuthardt says to the surgical resident standing next to him, as he draws a line on the area of the patient’s shaved scalp where he intends to make his initial incisions for brain surgery. “Did you think he was a replicant?”
“I definitely thought he was a replicant,” the resident responds, using the movie’s term for the eerily realistic-looking bioengineered androids.
“What I think is so interesting is that the future is always flying cars,” Leuthardt says, handing the resident his Sharpie and picking up a scalpel. “They captured the dystopian component: they talk about biology, the replicants. But they missed big chunks of the future. Where were the neural prosthetics?”
It’s a topic that Leuthardt, a 44-year-old scientist and brain surgeon, has spent a lot of time imagining. In addition to his duties as a neurosurgeon at Washington University in St. Louis, he has published two novels and written an award-winning play aimed at “preparing society for the changes ahead.” In his first novel, a techno-thriller called RedDevil 4, 90 percent of human beings have elected to get computer hardware implanted directly into their brains. This allows a seamless connection between people and computers, and a wide array of sensory experiences without leaving home. Leuthardt believes that in the next several decades such implants will be like plastic surgery or tattoos, undertaken with hardly a second thought.
“I cut people open for a job,” he notes. “So it’s not hard to imagine.”
But Leuthardt has done far more than just imagine this future. He specializes in operating on patients with intractable epilepsy, all of whom must spend several days before their main surgery with electrodes implanted on their cortex as computers aggregate information about the neural firing patterns that precede their seizures. During this period, they are confined to a hospital bed and are often extremely bored. About 15 years ago, Leuthardt had an epiphany: why not recruit them to serve as experimental subjects? It would both ease their tedium and help bring his dreams closer to reality.
“At the pace at which technology changes, it’s not inconceivable to think that in a 20-year time frame everything in a cell phone could be put into a grain of rice,” he says. “That could be put into your head in a minimally invasive way, and would be able to perform the computations necessary to be a really effective brain-computer interface.”
Leuthardt began designing tasks for them to do. Then he analyzed their brain signals to see what he might learn about how the brain encodes our thoughts and intentions, and how such signals might be used to control external devices. Was the data he had access to sufficiently robust to describe intended movement? Could he listen in on a person’s internal verbal monologues? Is it possible to decode cognition itself?
Though the answers to some of these questions were far from conclusive, they were encouraging. Encouraging enough to instill in Leuthardt the certitude of a true believer—one who might sound like a crackpot, were he not a brain surgeon who deals in the life-and-death realm of the operating room, where there is no room for hubris or delusion. Leuthardt knows better than most that brain surgery is dangerous, scary, and difficult for the patient. But his understanding of the brain has also given him a clear-eyed view of its inherent limitations—and the potential of technology to help overcome them. Once the rest of the world understands the promise, he insists—and once the technologies progress—the human race will do what it has always done. It will evolve. This time with the help of chips implanted in our heads.
“A true fluid neural integration is going to happen,” Leuthardt says. “It’s just a matter of when. If it’s 10 or 100 years in the grand scheme of things, it’s a material development in the course of human history.”
Leuthardt is by no means the only one with exotic ambitions for what are known as brain-computer interfaces. Last March Elon Musk, a founder of Tesla and SpaceX, launched Neuralink, a venture aiming to create devices that facilitate mind-machine melds. Facebook’s Mark Zuckerberg has expressed similar dreams, and last spring his company revealed that it has 60 engineers working on building interfaces that would let you type using just your mind. Bryan Johnson, the founder of the online payment system Braintree, is using his fortune to fund Kernel, a company that aims to develop neuroprosthetics he hopes will eventually boost intelligence, memory, and more.
These plans, however, are all in their early phases and have been shrouded in secrecy, making it hard to assess how much progress has been made—or whether the goals are even remotely realistic. The challenges of brain-computer interfaces are myriad. The kinds of devices that people like Musk and Zuckerberg are talking about won’t just require better hardware to facilitate seamless mechanical connection and communication between silicon computers and the messy gray matter of the human brain. They’ll also have to have sufficient computational power to make sense out of the mass of data produced at any given moment as many of the brain’s nearly 100 billion neurons fire. One other thing: we still don’t know the code the brain uses. We will have to, in other words, learn how to read people’s minds.
But Leuthardt, for one, expects he will live to see it. “At the pace at which technology changes, it’s not inconceivable to think that in a 20-year time frame everything in a cell phone could be put into a grain of rice,” he says. “That could be put into your head in a minimally invasive way, and would be able to perform the computations necessary to be a really effective brain-computer interface.”
Scientists have long known that the firing of our neurons is what allows us to move, feel, and think. But breaking the code by which neurons talk to each other and the rest of the body—developing the capacity to actually listen in and make sense of precisely how it is that brain cells allow us to function—has long stood as one of neuroscience’s most daunting tasks.
In the early 1980s, an engineer named Apostolos Georgopoulos, at Johns Hopkins, paved the way for the current revolution in brain-computer interfaces. Georgopoulos identified neurons in the higher-level processing areas of the motor cortex that fired prior to specific kinds of movement—such as a flick of the wrist to the right, or a downward thrust with the arm. What made Georgopoulos’s discovery so important was that you could record these signals and use them to predict the direction and intensity of the movements. Some of these neuronal firing patterns guided the behavior of scores of lower-level neurons working together to move the individual muscles and, ultimately, a limb.
Dr. Eric Leuthardt explaining the process to help “fix” a broken brain by rewiring a patient’s hand to his brain pic.twitter.com/6T4HYiEKTk
— TEDxGatewayArch (@TEDxArch) November 7, 2015
Using arrays of dozens of electrodes to track these high-level signals, Georgopoulos demonstrated that he could predict not just which way a monkey would move a joystick in three-dimensional space, but even the velocity of the movement and how it would change over time.
It was, it seemed clear, precisely the kind of data one might use to give a paralyzed patient mind control over a prosthetic device. Which is the task that one of Georgopoulos’s protégés, Andrew Schwartz, took on in the 1990s. By the late 1990s Schwartz, who is currently a neurobiologist at the University of Pittsburgh, had implanted electrodes in the brains of monkeys and begun to demonstrate that it was indeed possible to train them to control robotic limbs just by thinking.
Leuthardt, in St. Louis to do a neurosurgery residency at Washington University in 1999, was inspired by such work: when he needed to decide how to spend a mandated year-long research break, he knew exactly what he wanted to focus on. Schwartz’s initial success had convinced Leuthardt that science fiction was on the verge of becoming reality. Scientists were finally taking the first tentative steps toward the melding of man and machine. Leuthardt wanted to be part of the coming revolution.
He thought he might devote his year to studying the problem of scarring in mice: over time, the single electrodes that Schwartz and others implanted as part of this work caused inflammatory reactions, or ended up sheathed in brain cells and immobilized. But when Leuthardt and his advisor sat down to map out a plan, the two came up with a better idea. Why not see if they might be able to use a different brain recording technique altogether?
“We were like, ‘Hey, we’ve got humans with electrodes in them all the time!’” Leuthardt says. “Why don’t we just do some experiments with them?”
… Many were skeptical that the electrodes would yield enough information to control a prosthetic. To help find out, Leuthardt recruited Gerwin Schalk, a computer scientist at the Wadsworth Center, a public-health laboratory of the New York State Department of Health. Progress was swift. Within a few years of testing, Leuthardt’s patients had shown the capacity to play Space Invaders—moving a virtual spaceship left and right—simply by thinking. Then they moved a cursor in three-dimensional space on a screen.
In 2006, after a speech on this work at a conference, Schalk was approached by Elmar Schmeisser, a program manager at the U.S. Army Research Office. Schmeisser had in mind something far more complex. He wanted to find out if it was possible to decode “imagined speech”—words not vocalized, but simply spoken silently in one’s mind. Schmeisser, also a science fiction fan, had long dreamed of creating a “thought helmet” that could detect a soldier’s imagined speech and transmit it wirelessly to a fellow soldier’s earpiece.
Leuthardt recruited 12 bedridden epilepsy patients, confined to their rooms and bored as they waited to have seizures, and presented each one with 36 words that had a relatively simple consonant-vowel-consonant structure, such as “bet,” “bat,” “beat,” and “boot.” He asked the patients to say the words out loud and then to simply imagine saying them—conveying the instructions visually (written on a computer screen), with no audio, and again vocally, with no video, to make sure that he could identify incoming sensory signals in the brain. Then he shipped the data to Schalk for analysis.
Schalk’s software relies on pattern recognition algorithms—his programs can be trained to recognize the activation patterns of groups of neurons associated with a given task or thought. With a minimum of 50 to 200 electrodes, each one producing 1,000 readings per second, the programs must churn through a dizzying number of variables. The more electrodes and the smaller the population of neurons per electrode, the better the chance of detecting meaningful patterns—if sufficient computing power can be brought to bear to sort out irrelevant noise.
“The more resolution the better, but at the minimum it’s about 50,000 numbers a second,” Schalk says. “You have to extract the one thing you are really interested in. That’s not so straightforward.”
Schalk’s results, however, were surprisingly robust. As one might expect, when Leuthardt’s subjects vocalized a word, the data indicated activity in the areas of the motor cortex associated with the muscles that produce speech. The auditory cortex, and an area in its vicinity long believed to be associated with speech processing, were also active at the exact same moments. Remarkably, there were similar yet slightly different activation patterns even when the subjects only imagined the words silently.
Schalk, Leuthardt, and others involved in the project believe they have found the little voice that we hear in our mind when we imagine speaking. The system has never been perfect: after years of effort and refinements to his algorithms, Schalk’s program guesses correctly 45 percent of the time. But rather than attempt to push those numbers higher (they expect performance to improve with better sensors), Schalk and Leuthardt have focused on decoding increasingly complex components of speech.
In recent years, Schalk has continued to extend the findings on real and imagined speech (he can tell whether a subject is imagining speaking Martin Luther King Jr.’s “I Have a Dream” speech or Lincoln’s Gettysburg Address). Leuthardt, meanwhile, has attempted to push on into the next realm: identifying the way the brain encodes intellectual concepts across different regions.
The data on that effort is not published yet, “but the honest truth is we’re still trying to make sense of it,” Leuthardt says. His lab, he acknowledges, may be approaching the limits of what’s possible using current technologies.
YOU JUST THOUGHT, ‘I’D LIKE TO READ MORE.’ MAYBE THIS COMPUTER UNDERSTOOD YOU