Why Are Neural Networks Clean?

Learning forms pathways

A neuron’s disparate responses can become intelligible through its learning history. Ordinary generalization concerns how learned behavior extends beyond training examples. During further training, a mechanism can also acquire new responses and reshape the conditions for its next acquisition. How does one extension make another easier to learn?

In Neural Networks Are Already Clean, Nick Cammarata follows a neuron through a retrained InceptionV1 model.¹ Early in training, its strongest activations come from yellow-flower images, sometimes with dogs. Green objects and dogs without flowers appear among its strongest activations later, followed by broccoli. Cammarata calls this recursive generalization, as related extensions become starting points for further extensions.

Path formation describes how training reshapes circuitry and changes what becomes easier to learn next. For Cammarata, several response categories can belong to a clean neuron when their computational roles or development explain why they occur together. A neuron participates in processes that group some inputs and distinguish others, channeling their effects through downstream connections. The observed sequence motivates this account, while its causal links remain hypothetical.

Why familiar pathways get a head start

Explaining that response history requires examining the circuitry that produces the neuron’s activity and uses its output. Satrajit Chatterjee’s coherent-gradients hypothesis proposes that examples favoring compatible parameter changes reinforce those changes together.² Cammarata’s circuit-ecology hypothesis proposes that established pathways capture learning signals ahead of competitors built from scratch. As updates through established pathways reduce error on relevant examples, competing pathways may receive weaker gradients for the same task.

A flower-responsive mechanism may already respond weakly to vegetation, animals, or shared textures. When green-object examples activate it, their errors can contribute gradients through its downstream connections. The optimizer could strengthen those responses using compatible gradients and change which later examples activate the mechanism. This intermediate change might lower the effort needed to acquire broccoli responsiveness, opening a route the earlier circuitry did not readily support.

Channeling gradients describes where learning signals pass through the network. Lowered resistance means a particular response requires less learning effort under comparable conditions, such as fewer optimization steps or examples. A response extension can redirect gradients without reducing that effort, since downstream sensitivity and optimizer dynamics also matter. Establishing an advantage requires a controlled comparison, as activation, gradient support, and susceptibility alone do not guarantee acquisition.

Same prediction, different learning costs

Cammarata compares InceptionV1 and ConvNeXt neurons trained on the same data, both initially responsive to rapeseed fields.¹ Images of balls and fruit rank among both neurons’ strongest activations early in training. These categories later recede from ConvNeXt’s top-activating images. Dogs and broccoli become prominent in InceptionV1’s set as field images recede.

These sets show which image categories produce the strongest activations at each checkpoint. Displaced categories may still activate the neuron more weakly. Architecture and learning history both vary here, so the comparison alone cannot establish which inherited pathways caused the divergence. The trajectories motivate investigating how path formation rechannels learning and which extensions can coexist with earlier roles.

Two networks can make the same flower prediction while differing in the additional learning required to acquire broccoli responsiveness. A controlled comparison would hold architecture, new data, and training conditions fixed while varying the inherited pathways. Strengthening a broccoli response could also weaken a flower distinction that downstream processes still need. Retaining both roles might require a separate pathway or further reorganization.

Recruitment and response change

How might a useful new response change recruitment of its mechanism and redirect further learning? George Price’s identity separates recruitment from response change (variation, the standard transmission term) under a specified correspondence.³ Over equally weighted baseline mechanisms, $z_i$ is a response on a fixed assay, $w_i\geq0$ an attributed descendant weight, and $\Delta z_i$ the credit-weighted descendant mean minus $z_i$ (set to zero when $w_i=0$). Expectation and covariance use this baseline with $\bar w=\mathbb E[w]>0$, and $\Delta\bar z$ measures the change from the baseline mean response to the later attributed-use mean.⁴⁵

\Delta\bar z
=
\underbrace{\frac{\operatorname{Cov}(w,z)}{\bar w}}_{\text{selection}}
+
\underbrace{\frac{\mathbb E[w\Delta z]}{\bar w}}_{\text{variation}}.

For a population of neurons followed by coordinate across checkpoints, let a fixed image assay and threshold measure dog responsiveness without flowers. The flower neuron would be one member of this population. Each neuron has one equally weighted continuation, so $w_i=1$ and selection is zero. All response change enters variation, while gradient competition remains a separate causal hypothesis.

For downstream-use inheritance, fix baseline flower-related mechanisms, the task set, and a causal attribution rule. Include only later uses assigned a registered ancestor, excluding unassigned routes from this population. Split each included use’s unit credit among its ancestors, summing an ancestor’s credits for $w_i$ and using the same credits for its descendants’ mean assay response. Inheritance here measures contributions to later computation, while path formation and lowered resistance remain proposed causes of recruitment and response change.

How disparate responses become intelligible

In the illustration, three later uses inherit dog responsiveness and one does not, raising the responsive share from 50% to 75% through recruitment. Acquired responsiveness in the fourth use brings it to 100%, so each term contributes 25 percentage points. During training, new responses may attract recruitment that redirects gradients and lowers the effort of another acquisition. Repeated recruitment and modification could produce branching and recombining pathways.

Sharing a pathway can also create interference between its computational roles. A new downstream role may reinforce a useful distinction or redirect learning away from an earlier one. Lowered resistance is therefore specific to an acquisition and its training conditions. Preserving an earlier role may require new circuitry even when another response becomes easier to learn.

Ordinary generalization describes learned behavior beyond its training examples. Locally understandable extensions can accumulate into responses that appear unrelated when their history is hidden. Cleanliness could therefore reflect the path formation behind Cammarata’s recursive generalization, as acquired pathways can make further responses easier to learn. Learning builds the machinery of its own future learning.