Clean Learning and What Remains

What makes a feature useful?

Every acquisition changes what a neural network learns next. But if neural circuitry itself is a moving target, what is the conserved nature of a learned mechanism?

In Neural Networks Are Already Clean, Nick Cammarata notes the importance of identifying when mechanisms are used in later development.

When I say neural networks are already clean, I mean something deeper than neuron monosemanticity, although even by that standard, I think things look much better than the field expects. I mean that a network’s parts are built to be read, and built upon, by its downstream readers.¹

Cammarata follows a neuron in a retrained InceptionV1 model whose strongest responses initially come from yellow-flower images, some with dogs. Green objects and dogs without flowers appear later, followed by broccoli. This developmental history traces how a mechanism’s responses branch into related domains.

Cammarata calls the branching of responses across related inputs “recursive generalization.” His proposed explanation is that existing mechanisms give related learning a head start.

Cammarata traces how a mechanism develops. In this post, I build on this and ask what is conserved through that development. I use controlled experiments to examine which responses persist, which readers continue to use them, and whether their usefulness survives changes in the circuitry supplying them. These experiments distinguish forms of continuity and examine how earlier acquisitions shape later learning.

One path leads to another

In Cammarata’s circuit-ecology hypothesis, established mechanisms and their readers can learn related tasks faster than competitors starting fresh. As their predictions improve, less error remains to train competing pathways. He proposes that repeated reuse can produce recursive generalization.¹

Applied to Cammarata’s flower neuron, the hypothesis suggests a possible mechanism. A flower-responsive neuron might also respond weakly to vegetation. Training on green objects could strengthen that response into a green signal for a later broccoli detector. This intermediate acquisition may make broccoli responsiveness easier to learn than it was from the original flower response.

A broccoli detector that uses the green signal would give the earlier mechanism a new role. This is the causal interpretation explored below, rather than a measured dependency in Cammarata’s flower neuron.

A reader is a downstream computation using pathway outputs. I built a synthetic model to test when an earlier acquisition helps later learning and when it becomes a constraint.² The model receives binary flower, green, and texture cues. Supplying these distinctions directly lets us study how feature use develops without explaining a mechanism’s initial discovery.

The broccoli task requires green and texture together. Flower discrimination means correctly classifying inputs with and without the flower cue, even when response strength changes.

I measure learning pressure as the negative loss gradient on a parameter. For ordinary gradient descent, its sign and magnitude determine the proposed update per unit learning rate. Constraints can also alter the update. A shared capacity limit, for example, can weaken flower responsiveness without a direct flower-loss gradient.

In the first experiment, I trained two readers to predict a green-dependent target using competing pathways. Training updates both the feature coefficients and the weights readers use to combine those features. The established route receives stronger gradients, which decline as prediction error falls.³

Readers reinforce one another when their errors favor the same update to a shared feature. Opposing updates can cancel, leaving weak net pressure despite strong individual contributions.

The weights carrying signals to downstream readers also learn. For the same prediction error, a stronger input produces a larger gradient on the weight carrying that signal.

An established green signal gives the broccoli reader a useful input before its own training begins. Whether this reduces acquisition cost depends on how subsequent updates affect the other responses that share the pathway.

Two paths can represent the same distinction

Cammarata compares neurons in InceptionV1 and ConvNeXt that initially respond to yellow-flower fields. The ConvNeXt neuron stays focused on fields, while the InceptionV1 neuron’s strongest responses shift toward dogs and broccoli. To examine how learned pathways affect later acquisition, I compared two toy networks under matched training conditions.

The networks have the same architecture and initially classify flowers equally well. Both receive the same later training, with a capacity limit on each pathway’s combined flower and green responses.

World A uses an established green response in the flower pathway to reach the broccoli target sooner. Strengthening green weakens its flower response. World B learns a separate green route, which takes longer but preserves the flower signal.

The advantage reverses when acquisition requires a higher broccoli score. The shared pathway must fit a stronger green response within the same capacity limit, slowing further learning and weakening flower. The separate route then reaches the target in fewer updates.⁴

A distinction can persist despite weaker activation. Both routes preserve flower classification, but the weaker signal may affect other downstream uses.

Across a population of pathways, increased use of a flower-responsive mechanism can coincide with loss of flower responsiveness in its descendants. Measuring only the final responsive share would miss these opposing changes.

How roles and responses change

Across inherited pathways, an unchanged average can conceal a redistribution of the mechanisms supplying a distinction. How much change comes from ancestors supplying more pathways, and how much comes from inherited responses changing?

I initialized four pathways with the weights of two earlier mechanisms and trained them toward specified responses. Initially, $\mathsf{A}$ responds to flowers, while $\mathsf{B}$ supplies texture without a flower response on the same assay. Three pathways inherit $\mathsf{A}$’s weights and one inherits $\mathsf{B}$’s. Two $\mathsf{A}$ descendants retain flower responsiveness, while the third learns green and loses its flower response.

The flower-responsive share stays at ½, even though $\mathsf{A}$ now supplies ¾ of the pathways. Its increased representation is offset by a change in what one descendant detects.

Trained networks can develop overlapping representations along separate pathways,⁵ while shared parameters can receive opposing learning signals.⁶ The toy isolates these ingredients by prescribing ancestry and choosing changes that cancel exactly. This makes the accounting ambiguity visible without establishing how often such cancellation occurs in ordinary training.

Price accounting has previously been applied to evolutionary algorithms and neural learning.⁷ Here, I use it to measure how differential representation and response change contribute to the population mean. The equation separates selection (which mechanisms contribute more descendants) from variation (how those descendants differ from their ancestors).⁸⁹

Under a fixed assay, $z$ is one for a flower-responsive pathway and zero otherwise. The descendant count $w$ records how many later pathways are attributed to each earlier mechanism, with average $\bar w$.¹⁰ For each ancestor, $\Delta z$ is its descendants’ mean response minus its own. The change in the population mean is¹¹

\Delta\bar z
=
\underbrace{\frac{\operatorname{Cov}(w,z)}{\bar w}}_{\text{selection}}
+
\underbrace{\frac{\mathbb E[w\Delta z]}{\bar w}}_{\text{variation}}.

$\operatorname{Cov}(w,z)$ measures whether initially responsive mechanisms supply more descendants, and $\mathbb E$ averages over the initial mechanisms. Dividing by $\bar w$ expresses both contributions per descendant.

Counting each descendant by its ancestor’s response gives three responsive pathways out of four. This reweighting raises the share from ½ to ¾, giving selection +¼. One $\mathsf{A}$ descendant loses the flower response, contributing −¼ through variation. The two changes cancel, leaving the final share at ½.

Each copy represents an inherited use with known ancestry. Pathways count equally, so three nearly unused copies outweigh one heavily used pathway in this calculation. A persistent mechanism can also serve more readers without being copied, which motivates the allocation experiment below. Whether readers depend on those signals requires testing their use.

Ancestry records identify which responses are retained or acquired. Price accounting quantifies how changing representation and inherited responses contribute to the population mean, exposing changes that cancel in the total. The identity does not itself assign causes to those changes.

Explaining the changes requires examining training dynamics. Direct learning pressure and competition for shared capacity can each weaken an earlier response.¹² Here, ancestry and targets are prescribed, so the construction illustrates the accounting without explaining why branching would emerge.

The global terms do not identify which readers receive retained flower responses and which receive newly acquired ones.

Readers reveal inherited roles

I constructed two histories with identical global Price terms but different ancestry for the flower readers’ inputs. Each has three descendants of $\mathsf{A}$ and one of $\mathsf{B}$. Both finish with two flower pathways, a green pathway, and a texture pathway supplying identical readers.

In History I, both flower pathways descend from $\mathsf{A}$ and retain its flower response. In History II, only one flower pathway descends from $\mathsf{A}$. The other descends from $\mathsf{B}$ and acquires its flower response during training. The remaining pathways fit the same prescribed green and texture targets in both histories, giving the broccoli reader matching signals.

Their global Price terms are identical, selection +¼ and variation −¼. Final reader outputs also agree on all eight toy inputs. Tracing each reader group’s ancestry shows flower responses that were inherited versus acquired during training.¹³

Each group is compared with the full initial population, whose flower-responsive share is ½.

Averaging the equally sized groups recovers the global terms in either history. In History I, the flower readers use two retained responses. In History II, they use one retained response and one newly acquired response.

The matched histories prescribe which pathways supply each reader. In the next experiment, reader errors train both the responses of two persistent routes and the weights each reader assigns to them. These routes start afresh, with flower and green responses in A and zero responses in B.

Each reader learns nonnegative routing shares that sum to one, starting with equal shares. The broccoli reader combines its routed signal with a fixed texture input.¹⁴

The first Price analysis counts inherited pathway uses, while this one measures allocation among persistent routes. Selection can therefore change here without creating any new pathways, simply because a reader assigns more weight to one route.

The trait is now flower contrast, the change in a route’s response when the flower cue is added while other cues stay fixed. This continuous value can be negative or exceed one.

Selection measures changes in allocation to the initially flower-responsive route. Variation measures changes in the routes’ flower contrasts during training. Both terms come from the same training trace as the reader gradients. Equivalent computations can yield different decompositions under different route identities, signal representations, or weighting rules.¹⁵

The table weights each route’s flower contrast by its reader’s allocation. It shows change after 800 updates from an initial mean of 0.50. Both reader groups count equally in the global row. Values are rounded to two decimal places, so displayed terms may not sum exactly.

After 800 updates, the flower reader assigns about 72% of its routing share to A, while the broccoli reader assigns about 27% to A. These opposing allocation changes nearly cancel in the global selection term. Freezing both readers at 50% per route fixes selection at zero and leaves broccoli accuracy at 75%. Letting routing learn raises that accuracy to 100% under the same training conditions.¹⁶

Removing the initial green response before training leaves final classification unchanged but reduces mean squared reader error by about 99% after 800 updates. Restoring the green response reproduces the original trajectory.¹⁴ A starting point that makes one acquisition cheaper can impede joint learning when several readers and representations adapt together.

At update 400, the flower reader assigns about 75% to A without the precursor, compared with 56% with it. The broccoli reader assigns about 75% to B without the precursor, compared with 55% with it. B also acquires a stronger green response without the precursor. The two readers’ Price terms differ more strongly in both allocation and flower-response change.¹⁷

The precursor therefore changes the course of joint learning as well as its final error. Earlier differentiation is consistent with the improvement, but the comparison does not isolate which allocation or response changes cause it.

A reader’s allocation shows how strongly it weights a route, but does its classification depend on that signal? I tested this by zeroing each route after training while holding all learned weights and routing shares fixed.¹⁸

Under this intervention, perfect flower classification requires A and perfect broccoli classification requires B. Yet B began without the green precursor. Each reader has come to depend on a different route, although either route can still affect the other reader’s numerical output.¹⁸

These endpoint dependencies leave open whether the readers kept classifying correctly while the routes changed.

What learning carries forward

A distinction may remain encoded while its signal becomes too weak for an existing reader to classify correctly. Even Price accounting for a reader’s inputs cannot show whether it kept classifying correctly between checkpoints.

I tested whether a reader could keep classifying correctly while one pathway takes over another’s flower response. A fixed reader averages the two signals and classifies at threshold 0.45. Initially, $\mathsf{A}$ has response strength one and $\mathsf{B}$ zero. Both schedules train $\mathsf{B}$ toward one and $\mathsf{A}$ toward zero, with identical update budgets.¹⁹

In the continuous handoff, $\mathsf{B}$ learns the flower response before $\mathsf{A}$ is trained to lose it. This overlap keeps the reader classifying correctly. Reversing the order causes temporary failure, even though the flower signal remains positive. Both schedules end with the same pathway parameters and reader outputs.

Each pathway has one equally weighted descendant, giving zero selection. The unchanged binary responsive share gives zero net variation. The reader’s training trace shows an interruption that endpoint accounting misses.

Throughout the continuous schedule, the score for flower-present inputs stays at least 0.05 above the reader’s threshold, while flower-absent inputs score zero. At each tested threshold from 0.001 to 0.45, the continuous schedule preserves classification and the reversed schedule briefly loses it.¹⁹

Both schedules reach the same state, so further training under identical conditions would produce matching updates. Their difference lies in what the reader could do along the way. The earlier experiments show how retained circuitry can shape later learning; this experiment shows that continuity of use is a separate property.

Together, these experiments distinguish a response’s persistence, a reader’s dependence on it, and uninterrupted access to it. A response can survive without remaining usable, and a useful response can pass between mechanisms. Functional conservation therefore requires specifying what remains usable, by which reader, and over what interval.

Protein function offers a familiar analogy. Related proteins can retain functional structures despite differences in their amino-acid sequences.²⁰ Likewise, preserving a learned capability need not require preserving every component that first supplied it.

Cammarata’s clean-network account explains mechanisms through their use by downstream readers. This account of functional conservation follows that relationship through change. Matching endpoints alone cannot establish its continuity. What is conserved is the reader’s ability to use a learned distinction, even as the mechanism supplying it changes.

Thank you to Toby Shorin and Cullen Miller for your support and feedback while writing this post.