“We’ve been looking at it all wrong”: A Goodfire researcher proposes a new perspective on neural network interpretability, sending shockwaves through the AI community.
Goodfire researcher Nick Cammarata published a 26,000-word paper arguing that the internal mechanisms of large language models (LLMs) may be far more structured and comprehensible than the academic community has previously assumed. His central thesis hinges on a reinterpretation of “polysemous neurons” in models such as Inception v1—suggesting that neurons associated with both cats and cars, for instance, actually respond to V-shaped visual patterns rather than exhibiting disordered overgeneralization. This view has yet to undergo systematic validation on modern LLMs and has been challenged by scholars including Chris Potts of Stanford University and Jacob Hilton of the Alignment Research Center, who question its universality and technical depth.