Superposition in Neural Networks: Toy Models, SAEs and Feature Merging

Storing more than there's room for

Picture a neural network whose hidden layer has just two dimensions, so every input is represented by two numbers, a point (x, y) in a flat plane. Now suppose the network has to track five different features. It sounds impossible. Two dimensions should hold two features, one per axis.

Superposition is the way out. A network can represent more features than it has dimensions by storing them along overlapping directions in the same space. Rather than "dimension 1 holds feature A and dimension 2 holds feature B", each feature gets its own direction, and several features can be present together in the same pair of dimensions. The open questions are why this works and how the network tells overlapping features apart afterward.

A layer holds several active features at once by adding their directions together.

In the equation above, the activation h is the layer's output for one input, a vector of d numbers. In the toy model d is 5, and for GPT-2 small at layer 8 it's 768. Each feature i has a strength x_i of zero or more, and there are n features in total, 20 in the toy model and unknown for GPT-2. Each feature also has a fixed unit direction W_i in the d-dimensional space, and all the directions together form a d × n matrix W whose column i is W_i.

In plain terms, every active feature pushes the representation along its own direction. The layer stores only the final summed vector h, with no separate slot for any feature.

To read one feature back, take the dot product of h with that feature's direction.

Going from the first line of that derivation to the second uses the fact that a dot product distributes over a sum. Dotting W_i into h produces one term per feature, and the j = i term gets pulled out while the sum over j ≠ i stands for all the rest. The first part is a unit vector dotted with itself, which is 1, so that term is exactly x_i, the true strength. The second part collects every other feature, each weighted by how closely its direction lines up with feature i.

That one equation drives the whole subject, and it splits into two cases:

• When n ≤ d, there are no more features than dimensions, so every direction can sit at a right angle to every other. All the dot products W_i · W_j are zero, the interference half disappears and readout is exact. The ceiling is d features.
• When n > d, that's impossible. In d dimensions at most d vectors can be mutually perpendicular, so some pair must overlap, and the interference half is never zero.

Superposition means accepting the second deal on purpose: more than d features, paid for with some interference.

Why the interference is tolerable

Interference only matters when the other features are on. If feature j is off, x_j is zero and its contribution vanishes. If each feature fires with probability p, the expected number of interfering features is about (n − 1)p.

With n = 20 features and p = 0.001, that's 19 × 0.001 = 0.019 expected interferers. Around 98% of inputs have no other feature active at all, so readout is exact. The interference is real, but it's rarely met.

High-dimensional spaces help further. Only d directions can be exactly orthogonal, but there's room for a great many nearly orthogonal ones. Random unit vectors have an expected squared alignment of 1/d, so a typical alignment is about 1/√d.

In GPT-2's 768 dimensions that works out to 0.036, an angle of about 87.9°. Two directions picked at random are already nearly perpendicular by accident. Big spaces forgive in ways small ones can't.

Actual networks add a ReLU after the readout, defined as ReLU(z) = max(0, z): positive values survive and negative ones become zero. Since strengths are never negative, a negative interference term drags the reading under zero, and the ReLU then clamps it to 0, which happens to be correct for a feature that was off. So negative interference is free, and only the positive kind hurts.

The toy model

The setup follows the Toy Models of Superposition paper (Elhage et al., 2022): take 20 features, force them through 5 dimensions and ask for them back.

The diagram shows the whole model, which squeezes 20 numbers into 5 and then tries to recover all 20.

• x is the truth, 20 strengths that get generated.
• W is a 5 × 20 matrix whose column i is direction W_i.
• h is the 5-number compressed form.
• x̂ is the reconstruction.
• b holds 20 extra offsets that get added ahead of the ReLU.
• Training minimizes the mean squared gap between the truth and its reconstruction.

The transpose Wᵀ is the same matrix turned on its side. The model has no separate matrix for reading, which is exactly what makes column i the direction of feature i and gives "the geometry of the features" real meaning. The count is 5 × 20 = 100 numbers in W plus 20 in b, or 120 trainable parameters.

Nothing in the features has a name. Feature 7 isn't "the plural feature"; it's merely column 7. This is a feature of the design and not a flaw. With no language going in, nothing coming out can be a statement about language, so any pattern has to be about compressing sparse signals, and that is why the same pattern ought to show up inside a real model.

Sparsity is the main dial. Each strength is drawn independently, zero with probability S and uniform on 0 to 1 otherwise. Ten runs step S upward from 0 through 0.5 and 0.9 to 0.999, with 0.95, 0.99 and 0.997 in between and 0.3, 0.7 and 0.8 filling the lower range.

The architecture is a single matrix with no depth and no attention. W compresses on the way in and decompresses on the way out, with one ReLU at the end. Calling h a hidden layer overstates it, as it's simply W times x. All twenty runs together finish in about a minute on a CPU.

Reading WᵀW

WᵀW isn't a diagnostic added afterward. It appears on its own when h = Wx is substituted into the readout.

WᵀW is a 20 × 20 table showing the operator the model actually applies to the truth, with W_i · W_j as entry (i, j). One picture exposes the whole solution. Down the diagonal you see how firmly each feature is stored, with a value near zero meaning the model gave it up. Off the diagonal you see which features bleed into which, and a table of zeros means no interference at all. The ideal is the identity matrix, and superposition is the look of a model that won't accept the ideal.

The same model is shown at four sparsities. At the left there are five dark diagonal cells, fifteen features abandoned and no off-diagonal color, since there's no interference to show. At the right there are twelve dark diagonal cells and a scatter of off-diagonal color. That color is the price of the seven extra features.

The linear control is the same experiment with the ReLU deleted and nothing else changed.

Exactly five features get stored at every sparsity level, even 0.999, and that held in all ten runs. Deleting the nonlinearity therefore does more than weaken superposition; it kills it. The claim moves from "the ReLU helps" to "the ReLU is a precondition", since without the free clean-up of negative interference the extra features never pay for themselves.

The left panel shows the phase change. The climbing curve is the ReLU model going from 5 features to 12, and the flat line is the linear control that never moves. The middle panel shows the price, with mean dimensionality falling from 1.0 to 0.38. The right panel shows the conservation law, with the total sitting on 5 in all twenty runs.

What dimensionality measures

The middle and right panels use per-feature dimensionality, meaning how much of a dimension one feature has to itself.

The top of that formula is how strongly feature i is stored, squared, and the bottom is its total squared overlap with everything, itself included. A feature alone on its own axis has nothing overlapping it, so the bottom equals the top and D_i = 1. With k features sharing an axis, the bottom picks up k equal terms and D_i = 1/k.

Add up the values across all features and every one of the twenty runs lands between 4.94 and 5.00, for both models and every sparsity. At S = 0.9 nine features are stored with 0.5556 of a dimension apiece, and 5 ÷ 9 = 0.5556. Superposition never adds capacity. It slices a fixed amount into finer pieces.

Does a mean of 2/3 prove a triangle?

The middle panel reaches a mean dimensionality of 2/3, and that looked like proof that three features had formed a triangle. It wasn't, and the mistake turned out to be the most useful part of the project.

A mean D of 2/3 would follow from three features 120° apart, but it follows from other layouts too, because D only totals the squared overlap and says nothing of the geometry producing it. Measuring the angles directly settled it: of 30 runs with 3 features in 2D, 29 gave no triangle. The typical result had one feature on its own and two sharing an axis in opposite directions, with dimensionalities of 1.0, 0.5 and 0.5 and angles of 90°, 90° and 180°. The mean is the same 2/3 and the shape is entirely different. Whether this layout is the genuine optimum or a product of how the loss is weighted remains open.

What shape do the features settle into?

To make the directions drawable on paper, the model drops to 2 dimensions and the feature count varies from 3 to 6. The question "what shape?" can then be answered by looking. Six sparsities × five seeds × four feature counts makes 120 runs.

Every feature direction appears as an arrow on a circle, with sparsity increasing across panels. A zero-length arrow marks a dropped feature. At higher sparsity more arrows show up and spread out evenly, and that spacing is how the model answers the interference equation: put the directions as far apart as possible so that each pairwise overlap shrinks.

Is the arrangement reliable as features multiply?

Not at all, and the drop is steep. The check used 30 seeds per feature count, with the same setup but different random starting weights, and counted how often runs landed on the same shape.

A digon means two features sharing a line while pointing opposite ways. At 3 features the result is almost forced, whereas at 6 the runs scatter over several competing layouts. Crowding more features into the space multiplies near-equivalent solutions, so the final geometry leans more heavily on the initialization. There's one limit: all runs are 2D, so it can't be said whether the effect follows the raw feature count or the ratio of features to dimensions. Telling those apart would need a dimension sweep that wasn't done.

The chart sorts all 120 runs by what they found. The tall bar in each group is the winner, and it shrinks as the feature count rises.

Can a sparse autoencoder read the features back?

Everything so far was a model writing features into a cramped space. The reverse problem is the one that interpretability work cares about: given only h, recover what went in.

In the diagram, f is the wide sparse code, 40 numbers here, nearly all zero. The decoder matrix W_dec has the atoms as its columns, the autoencoder's estimates of the true directions, and those are the only output that counts.

TopK holds on to the k biggest values and zeroes everything else (Gao et al., 2024). The alternative is an L1 penalty on the loss that lets sparsity emerge. TopK suits this job since it fixes sparsity directly, while a penalty weight tuned on one problem tells nothing about another. Comparing two models at matched sparsity is awkward with L1 and trivial with TopK. A decoder bias is subtracted ahead of encoding and restored afterward, so atoms model the shape of the data and aren't wasted on its offset from the origin.

The autoencoder never sees the true directions. Grading happens afterward by comparing each true direction against the atoms, in two ways, and the difference between the two is the result.

Scoring recovery

The obvious score asks, for each true feature, how good the closest thing the autoencoder found is.

Each true feature chooses its best atom on its own, and nothing prevents two features from choosing the same atom. Suppose the autoencoder welded features a and b into a single atom along their sum. It lands midway between them; with the features 36° apart it's 18° from each, giving cos(18°) = 0.951. Both features score about 0.95, so the outcome looks great despite two features having become one. A merged atom truly does match both well.

Matched recovery fixes it. Every true feature is given a distinct atom in a way that maximizes total similarity, via scipy.optimize.linear_sum_assignment. When two features merged, just one receives the shared atom and the other gets pushed onto a poorer match, so the merge shows. The difference between the two scores measures merging, since either score alone can mask it.

A case where recovery is impossible in principle

A line of algebra shows that the failure ahead isn't the autoencoder's fault. Take features a and b that always fire together with identical strength, so x_a = x_b for every input. They then contribute a single shared term to the activation.

No method can undo this, because the activations carry only the sum of the two directions. The individual W_a and W_b are never observable. The information isn't concealed; it's missing. A stronger autoencoder, extra compute or another loss won't bring it back. This is non-identifiability, a limit of the data and not of the technique.

Change a single detail so that the two features still always fire together but with strengths that vary independently, and the picture reverses. Varying x_a while x_b stays fixed exposes W_a, which makes both directions recoverable. "Correlated features break sparse autoencoders" is therefore too blunt to be true: co-firing is harmless, locked strengths are fatal, and only the second of those is a real problem.

Results on the toy model

The grid crossed sparsity, co-firing rate and strength coupling over 15 conditions with 3 seeds each, making 45 runs with 20 true features, 5 dimensions and 40 atoms.

Twelve of the fifteen conditions show exactly 0% merging. Only the three combining full co-firing with fully locked strengths break, at 20%, 50% and 40%, while half co-firing with full strength coupling stays perfectly clean. There's a cliff here rather than a slope, and it sits exactly where a pair stops being identifiable.

The worst condition is shown through four metrics. The autoencoder isn't failing here. It recovers each merged pair's sum direction at scores of 0.878, then 1.000 twice, over the three broken conditions. What it found is real structure, only not the structure tagged as ground truth, because the missing distinction isn't present in the data.

Reproducibility doesn't catch it either. Agreement across seeds stays at 0.878 to 0.925 even where merging occurred, because the runs keep landing on one answer. That demonstrates a stable optimization and says nothing about whether the answer is correct.

Merged atoms land squarely between the two features they absorbed, which explains their high scores against each one.

Does it survive in a real language model?

A fair objection is that fitting 20 features into 5 dimensions is absurdly cramped, so naturally they fuse. With 768 dimensions of real language the pressure might disappear, since random directions there already sit 87.9° apart by chance. Testing this hits a wall, because neither setting can answer it alone.

The table above lays out what each setting offers. The toy model knows the ground truth but isn't realistic, while a real model is realistic but its true features are unknown.

Synthetic injection

Synthetic injection bridges the two. Take real GPT-2 activations, plant a handful of features that were designed in advance, train an autoencoder on the mixture without saying which directions are the planted ones, and grade it on those only. It works like a spike-in control. A metal detector can't be tested in a field of unknown contents, but burying a few coins and checking that it finds them works, and the rest of the field can stay unknown.

In the setup, h_real stands for a genuine 768-number activation from GPT-2 small at layer 8 while it reads real text. The 24 directions v_k are random unit vectors fixed beforehand, and the strengths a_k are under control, including which pairs are tied together. Three conditions share the same activations with four pairs each: pairs firing at unrelated moments (the control), pairs that always co-fire with free strengths, and pairs that always co-fire with locked strengths. Only the last should break.

GPT-2 small ran over 2 million tokens, and the residual stream after layer 8, about halfway through the network, was saved as 768-dimensional vectors. That gave 2 million vectors, around 2.9 GB, in roughly seven minutes on a laptop GPU. Position 0 was thrown out since it acts as an attention sink with a norm near 3,100 against about 128 everywhere else, around 24 times larger, which would swamp the statistics.

The random baseline

Each score reported is the best match from a big pile, and a best-of-many wins partly by luck. So there has to be a baseline for what meaningless directions would earn. For a single pair, two random unit directions in d dimensions have an expected squared alignment of 1/d, so a typical value is 1/√768 = 0.036. For the best of many, thousands of candidates are compared and only the top is kept.

Theory and measurement differ slightly. At N = 2,543 atoms the formula predicts 0.143 and simulation gives 0.126; at N = 31,295 it predicts 0.164 against a simulated 0.145. The formula sits about 15% high, so every baseline here uses the simulated figure. What the formula does get right is the shape, a baseline rising with the square root of the log of the pile size. A dictionary twelve times bigger nudges the bar from 0.126 to 0.145 and no more, so size alone doesn't buy a high score.

Result

The run used 1M tokens, 3,072 atoms, TopK with k = 32 and 3 seeds, graded only on the 24 planted features.

The failure reproduces precisely. All locked pairs fused under all seeds, with matched recovery of 0.416, 0.418 and 0.417. Moving to 768 dimensions of real language changes nothing, so the issue isn't missing room; what's missing from the data is the information that would separate the two features.

This is fusion and not collapse. Each pair's combined direction was recovered at 0.998 cosine, equal to the control, meaning the autoencoder captured real structure almost perfectly and merely welded two features into one. The naive metric would have missed it, reporting 0.702 against 0.417 matched, which looks mediocre without being alarming, because both features can claim the same merged atom.

The left panel shows recovery per condition, with the locked bar collapsing while the others hold. The middle shows every locked pair fusing under every seed. The right shows the score returning to 0.998, level with the control, once the combined direction is the target.

A direct search for merge suspects

A separate audit examined a released OpenAI autoencoder for GPT-2, hunting for feature pairs that are never seen apart. It covered the 1,024 most frequent features over 2 million tokens, 476,408 pairs in all. The algebra above gives two danger signs, namely that the pair is bound and that the strengths match.

• Only 3 pairs are nearly inseparable.
• Strengths are strongly correlated in 17,403 pairs.
• No pair meets both conditions.
• The 200 most tightly bound pairs have a median strength correlation of just 0.024.

Real text yields the benign kind of co-occurrence: items that turn up together but whose intensities vary independently.

The scatter shows the danger quadrant, high on both axes, completely empty. Pairs cluster along the edges, either bound but independent in strength, or correlated in strength but not always together, and never both.

The search has a blind spot, and it's the reason the injection experiment exists. After an autoencoder fuses two features there's only one atom left and no pair to detect, so a pair search walks right past a merge that already happened. An empty danger zone is comforting about features that stayed apart and silent about those that didn't.

Do two autoencoders find the same features?

All the autoencoders so far saw doctored activations containing the 24 planted features. So another was trained on clean activations with 1 million tokens, 3,072 atoms, TopK k = 32 and two seeds, four minutes each. It was compared with the released OpenAI dictionary, which has 31,295 live atoms and about a billion training tokens behind it.

Is the small one healthy enough?

Reconstruction is graded inside the model rather than in activation space, because squared error treats all 768 directions as equally important and GPT-2 does not. The reconstruction is spliced into the forward pass and the test checks how much next-token prediction degrades.

The chart plots cross-entropy, the model's prediction loss in nats, and the clean model sits at 3.302. Both seeds end up in the same place, recovering 93.76% and 93.97% of the loss, explaining 78.9% and 79.0% of variance, and keeping 2,543 and 2,597 of 3,072 atoms live. The released dictionary recovers 97.1%. The small one is undertrained by field standards but adequate for a comparison.

This chart shows the mean best-match cosine, counting live atoms only, each scored against its own random baseline.

At left each score is shown against its own random baseline, and the baselines differ because the pile sizes differ. At right the same numbers appear as angles. The bars shrink dramatically, which is the point, since cosine is generous where angle is honest.

The two seeds matched on every health metric yet disagreed about what a feature is: 93.76% against 93.97% loss recovered, 78.9% against 79.0% variance explained, and 2,543 against 2,597 live atoms. Their dictionaries lay 38.6° apart, and no standard quality check noticed.

Is the bigger dictionary just splitting ours?

The released dictionary has 12 times more atoms. The usual story is feature splitting, where extra capacity lets one coarse feature divide into finer ones, such as a quotation feature turning into opening quotes, dialogue attribution and scare quotes. If so, two things should hold: released atoms should cluster around the small dictionary's features, and each feature should be rebuildable as a positive mixture of nearby atoms, because a coarse feature should combine its refinements without signed cancellation.

In the test, q is one feature direction from the small dictionary. The first measure counts released atoms within cosine 0.4 of it, which is 66.4° and deliberately generous. The second takes the 16 nearest released atoms as the columns of a matrix B, fits the best mixture with no negative coefficients allowed, and reports the fraction of q explained, where R² = 1 is a perfect rebuild and 0 is nothing.

Both measures run twice, once on real features and once on 400 random unit directions. The second run is needed because atoms picked for being near a direction will fit it somewhat whatever it is, which is selection bias. Only the gap between the columns carries meaning.

Result

Each condition used 400 query directions against the 31,295 live released atoms. Clustering was decisive: a real feature has around six released atoms close by, whereas the 400 random directions had none within 66°. The two dictionaries evidently point into the same regions.

Spanning was only partial. The nearest 16 released atoms rebuild 0.696 of the features, against 0.495 for the best single atom, leaving about 30% unexplained. That remainder might be a real refinement the released dictionary lacks, or noise from undertrained features. Separating the two would take a sweep at 8×, 16× and 32× capacity, which was designed and not run.

The left panel counts released atoms near each query, with real features ranging from about one to fifteen and the random column a lone spike at zero. The right panel compares rebuild quality from the single best atom with that from the nearest sixteen.

Main findings

• The ReLU is required: with it deleted, the linear control kept exactly 5 features whatever the sparsity.
• Superposition divides existing capacity and never creates more, as total dimensionality held at 4.94 to 5.00 in all 20 runs.
• Pairs that can't be told apart fuse even in 768-dimensional GPT-2 activations, because their individual directions aren't in the data.
• A simple recovery score can conceal merging, since a merged atom matches both features well. Matched recovery shows 0.702 against 0.417 in GPT-2.
• Reproducibility says nothing about truth: seeds repeatedly reached the same wrong answer, which shows only that optimization is repeatable.

Putting one feature in one dimension, it turns out, was too boring for a neural network.

The experiments draw on the toy-model paper for the setup and the dimensionality measure, on earlier monosemanticity and sparse-autoencoder interpretability studies for the autoencoder recipe, and on the 2024 scaling study for the TopK formulation. They amounted to 185 toy training runs plus 2 million GPT-2 activations.