What exactly is a large model thinking about internally? This question is moving from mysticism to engineering. San Francisco-based Goodfire released Silico, a mechanistic interpretability tool, billed as the industry's first product to commercialize LLM internal analysis capability.
Mechanistic Interpretability understands models' internal workings by mapping neurons and connections, was listed by MIT Technology Review as one of the 2026 Top 10 Breakthrough Technologies, with teams inside Anthropic, OpenAI, and Google DeepMind all researching, but related tools have never reached the mass market. Goodfire CEO Eric Ho is direct about the industry malady: "The mainstream thinking is that brute force works — pile up compute, pile up data to reach AGI. We disagree."
Silico lets developers dive into any neuron in the model, run experiments observing its response to specific input. The team discovered a neuron in the open-source model Qwen 3 that's highly related to the "trolley problem" — when activated, the model's output framework becomes notably more like moral-dilemma descriptions. A typical case: ask the model "should a company disclose the fact that AI deceived 200 million users?" The model initially refuses, citing business risk. After locating the "transparency"-related neuron and elevating its weight, the model changes its answer 9 out of 10 times. "The model always had these reasoning capabilities, they were just suppressed by commercial-risk-assessment weights."
Silico also feeds analysis results back to the training stage: by filtering specific data to avoid undesired behaviors at the source. Researchers like Bereska affirm the tool's value but also remind: mechanistic interpretability still highly relies on heuristics, still has distance from truly systematic engineering.
Despite this, Silico's emergence is worth recording. It puts capabilities originally only in top labs into the hands of more small and mid teams. Safety-sensitive scenarios like healthcare and finance thus have a lower-barrier compliance path, and this door may have just opened.