Background knowledge: a neural network, conceptually, has billions of little things called neurons. One goal of so-called "interpretability" would be to be able to look at each neuron and say what it does for the network.
However, there is an observed phenomenon called "polysemanticity" wherein it seems that sometimes a single neuron seems to be for multiple different things. An early example of this was a finding that a single neuron in this image model seems to be for both animals and cars: Dragons and Cars …
An incendiary post from a prominent researcher suggests that AI is actually more predictable than we thought.
Boiled down, Cammarata’s main contention is that generative-AI models are not actually black boxes, contrary to the common perception that these systems are so big and labyrinthine as to be nearly impenetrable. In fact, Cammarata argues, there is a relatively clear path toward understanding them. If that’s the case, the material significance could be tremendous: It would provide a way to map AI models’ inner workings and, in turn, perhaps prevent them from wreaking all sorts of harm—autonomously hacking companies and sovereign nations, encouraging users’ delusions, appearing to deceive AI researchers themselves. There is an entire field of research, called “mechanistic interpretability,” devoted to the study of just this.
The argument is just as important as the person making it. Cammarata, a former OpenAI employee, is a respected figure in his field. Years ago, he played a key role in establishing the opposite claim, which is now common wisdom: that neural networks, the software architecture that generative AI is built on, are impossible, or at least very difficult, to understand. In this new essay, Cammarata, who now works at the AI-research company Goodfire, essentially says that he and his entire field have been wrong ever since. “The claims here are viscerally shocking,” one observer wrote. Or, as another wrote on X, they are “big big big big.”
Neural networks, inspired by the human brain, consist of layered connections among individual nodes, or “neurons,” that activate and send information to one another when responding to a query. Earlier research, including Cammarata’s, established that individual machine neurons can activate in response to many different, unrelated topics—for instance, one neuron can correspond to images of cat faces, cat legs, and the front of cars. Nobody was sure why this happened, but one explanation was that, because AI models contain far more information than they have neurons, they have to repurpose these nodes. Regardless, the basic units of an AI model having seemingly arbitrary, multitudinous functions would make it very difficult to understand what is going on under the hood.
Cammarata now thinks that individual neurons can be understood and that each has a self-consistent function. Neural networks are not “tangled knots,” he wrote, but “clean.” He found that the cat-face and car-face neuron, for instance, was actually responding to things with a V-shape—including V-shaped flames and jets of water. This would make AI’s internal mechanisms relatively straightforward and predictable rather than mysterious and nebulous. And if we can harness that understanding, some people in the industry believe, AI will be much less likely to go haywire and, say, kill us all. “This is truly one of the greatest cases for optimism in years!” the OpenAI researcher Adam Majmudar wrote on X.
Not everyone agrees with Cammarata. For one, his 26,000 words were merely a “working hypothesis” for a “forthcoming technical paper” and were thus hard, on a technical level, to evaluate. And the neural networks he studied most closely were vision models, which are generally simpler and better understood than the language models underlying ChatGPT and Claude. Cammarata could be mistaken, or he could be overstating his case. The blog post isn’t peer-reviewed, and the forthcoming paper probably won’t be, either, as is typical in the field. The research itself is something of a black box. (Cammarata, who did not immediately respond to a request for comment, wrote on X earlier today that he appreciates all of the discussion and that his approach “seems promising so far” for “bigger modern language models.”)
Discussions of Cammarata’s essay have revealed philosophical rifts. “This is interesting work, but I don’t think it’s a significant update to the big picture,” Neel Nanda, who leads mechanistic interpretability at Google DeepMind, wrotein his appraisal of Cammarata’s findings. Even if researchers could approximate the function of every neuron in Claude Mythos 5.1 or GPT-6 Astra, that’s not the same as understanding how they connect and how the network works as a whole. Physicists have good models of atoms and interatomic interactions, but they cannot from first principles explain or predict every physical event (hence unpredictable extreme weather or, for that matter, the notion of free will). You know all of your friends from school, various workplaces, and the local softball league, but you can’t always know whether they’ll get along at your birthday party. More to the point, insights into the inner workings of an AI model will not necessarily produce insights into how to make the technology more reliable. Interpretability may be necessary to build safer AI models, but it is by no means sufficient.
If generative-AI models are actually legible under the hood, that would undoubtedly help make the technology safer. But many people in Silicon Valley, and especially employees at OpenAI, are discussing Cammarata’s blog post with a breathless enthusiasm that echoes Silicon Valley’s long track record of easy technical solutions and grand promises—an attitude that belies its mixed record on following through.
Any research finding that pertains to AI safety will require an industry chasing multitrillion-dollar public offerings to slow down and take a long, hard look at itself. Perhaps even more concerning than an unintelligible superintelligence is the fact that a group of companies openly state that they don’t really know what they are building—and then continue to push that technology to extremes anyway. If Cammarata is correct, AI companies will have to work with one another, outside auditors, and governments to figure out how to translate such a breakthrough into safe and reliable products. And right now, absent any serious regulation, nobody can decide to take such action other than the companies themselves. All of which suggests that the real black boxes are OpenAI and Anthropic themselves.
