Fixed or Flexible? Study Shows Vision-Related Neurons Can Rapidly Switch Codes

For many years, a dominant view in neuroscience was that neurons in the inferotemporal (IT) cortex—a critical center in the brain for the recognition of objects—represent the world through fixed tuning functions. Doris Tsao (BS '96), who has been studying how the brain processes visual information for her entire career, believed this to be a fact. But now, with a team of other scientists that includes recent Caltech graduate Yuelin Shi (PhD '26), she has found evidence that the opposite is true.

In a recently published paper in Nature, the team shows that neurons in the IT cortex rapidly switch from a neural code that supports face detection to one optimized for face identification—all within roughly 20 milliseconds after a face is viewed. The results suggest a previously unrecognized neural mechanism that may help explain how the brain flexibly extracts different kinds of information from an image.

"The inferotemporal cortex has been the poster child for saying that the brain works like a deep network in machine learning—in which neurons or 'nodes' have set functions—and we thought we had figured everything out," says Tsao, chief scientist for neuroscience at the Astera Institute, a nonprofit research organization. Tsao was a Caltech faculty member from 2009 to 2021 and is a corresponding author on the paper. "It was really surprising to find that this code the cells are using is changing completely on this very fast timescale. It's still a little hard for me to accept."

The project began with a simple but fundamental question for the researchers: Could they predict how a face cell responds to faces using a model trained only on non-face objects? If so, that would suggest that neurons in the IT rely on a general-purpose code that works across categories, much like units in deep neural networks. If not, it would suggest that these neurons use specialized coding mechanisms for specific categories, making them domain specific. This idea speaks directly to a long-standing debate in neuroscience about whether the IT cortex is organized around general principles that apply to all visual categories or contains category-specific mechanisms. To answer this question, Shi and Tsao studied single neurons in the IT cortex of nonhuman primates as the animals viewed faces and non-face objects.

Building on earlier work by Tsao that found the mechanism that the brain uses to represent facial identity and discovered the mathematical system used by the brain to organize visual objects, the team recorded activity in face-selective neurons in specialized regions of the IT cortex known as face patches while study subjects were shown a large image set of faces or non-face objects. With the help of an artificial neural network architecture developed for image-classification tasks, the researchers were able to reveal how different neurons were responding to faces versus objects.

"We saw populations of face cells transition from detection to discrimination by switching from an object-general code to a face-specific one in a very short period of time," says Shi, who is first author on the paper and has been working with Tsao for nearly a decade.

Shi says the response of the individual neurons can be thought of like a ruler in which each cell is trying to measure different kinds of features for an object, for example, and giving different weights to these features.

"We are looking at the cells' rulers in a temporal way, which is something that's never been done before," Shi explains. "We did this by checking neuron activity for every 20-millisecond interval to see how the ruler changes for each cell over time."

Initially, a cell's face model and object model—its internal ‘ruler' for responding to faces and non-face objects—are aligned, suggesting that the cell starts with a general code for representing all categories, Shi says. But when the stimulus is a face, the cell switches its face model rapidly and begins to weigh more specialized facial features. If the stimulus is an object, the cell does not switch codes and continues using the general one.

"Our work shows that the brain is computing through dynamics and we, as a field, have to accept this new picture," Tsao says. "The idea that cells have these static tuning functions is simply not true."

She adds that the findings could also matter for the design of future computational systems, especially as researchers search for more energy-efficient approaches to AI and machine learning. If neurons can shift their tuning over time, then a single neuron may be able to support multiple different computations, potentially offering a much more efficient way to process information.

"This suggests a new way of thinking about computation in the inferotemporal cortex," Shi says. "It also solved this longstanding debate in the field about what's the fundamental logic of object recognition, and it actually matches with a lot of psychophysics or behavior studies."

In addition to having applications for technology, Shi and Tsao say the work sheds light on how our brains decide what is a face and what is not.

"If we see that most cells in this area change their code in this characteristic way, then the brain itself is signaling that it is representing a face; it is not a label imposed by the experimenter," Shi explains. "That gives us a concrete metric for identifying when a brain area starts treating a stimulus as a face and handing it off to a more face-specialized code."

In the future, this information may help people who have gone blind see the faces of their loved ones again, Tsao says. By identifying this previously unknown computation in the IT cortex, researchers may eventually learn how to stimulate the brain in ways that better match its intrinsic processing, potentially improving efforts to restore face perception. The team now wants to understand how this code shift contributes to a broader range of face-related cognitive functions. They plan to do this by showing test subjects more ambiguous images, including partial faces and objects that resemble faces, and asking them to report what they see, so the researchers can determine how shifts in neural coding relate to changes in perception and behavior.

The Nature paper, titled "Rapid concerted switching of the neural code in the inferotemporal cortex," was published on March 25. Janis K. Hesse (PhD '20) and Frank Lanfranchi (PhD '25), two former Caltech graduate students are co-authors of the paper. Additional co-authors from UC Berkeley are Dasheng Bi and Shi Chen. Funding was provided by the National Science Foundation, the National Institutes of Health, the Office of Naval Research, and the Howard Hughes Medical Institute.