Patterning the Toy Model of Superposition

Neural networks are programmed by their training data. This is particularly visible in AI alignment where common techniques, from RLHF to constitutional AI, act by reshaping the training distribution in order to reshape the model. At present this is largely a craft, guided by human intuition and empirical iteration. In this post we explore patterning, a principled approach to shaping models by shaping data.

In training a neural network the objective is to minimize loss, but there is rarely a unique way of doing so. Different minimizers may achieve low loss in different ways, for example by realizing different algorithms in their weights. A nice example comes from recent work by Li et al. (2025), who train a transformer to map strings of ( and ) to True or False depending on whether they are valid nestings (for example, (()()) is True while (() is False). They choose the training distribution so that at least two distinct algorithms achieve zero loss; the details aren’t important here, so let’s call them A and B. Both correspond to zero-loss parameters in the loss landscape, and both are found by transformers during training. Loss alone does not pick between them.

Alignment is, at bottom, a problem of this kind: aligned and misaligned models can both achieve low training loss, and something other than loss must do the selecting. Before tackling the hard version, shaping LLM-based agents to act in aligned ways, it makes sense to ask whether we can solve the easy version: shaping transformers to choose B over A in the above task. When this shaping is done via changes in the data distribution informed by measuring susceptibilities, we call the technique patterning. In our first paper on patterning we showed that this particular task can be achieved, by a nontrivial re-weighting of datapoints in the training set (Wang & Murfet 2026).

Patterning works. But it isn’t obvious why it works, or how the chosen re-weighting of the data serves to shape the model in the targeted way. At the scale of language models, whose internal structure is intrinsically hard to study, this question is hard to answer directly. In this post we study patterning in a simpler setting, the Toy Model of Superposition (TMS) of Elhage et al. (2022), where the weight vector can be visualized as a configuration of points in the plane and local minima of the loss correspond to regular polygons.

In this post we

• Provide a visual exploration of patterning in the setting of TMS. The intent is to help the reader build intuition for what patterning does by watching it happen, and interacting with it (Controlling the Transition).
• Demonstrate multi-step patterning where a complex patterning target is achieved by performing the patterning step multiple times (Multi-Step Patterning).

Toy Model of Superposition

We begin by explaining the Toy Model of Superposition (TMS) and how the local minima of its loss function correspond to regular polygons (focusing on 4-gons and 5-gons). Learning trajectories under stochastic gradient descent proceed from local minima corresponding to 4-gons to one of several different 5-gons which are derived from the 4-gon by “learning a feature”. The main goal of this post is to explain how patterning may be used to choose which 5-gon we end up at, and to understand how this works.

Inputs are binary vectors $x \in \{0,1\}^d$ drawn from a distribution $q(x)$. In $x = (x_0,\ldots,x_{d-1})$ we refer to each $x_i$ as a feature, and say the feature is active if $x_i = 1$. The distribution $q(x)$ is defined by saying that each feature is active with an independent probability $p=1-S$, where $S$ is the "sparsity".

The weights of the model $W \in \mathbb{R}^{m \times d}$ first encode inputs with representation $r = W x$, then reconstruct them as:

We train against the feature-weighted MSE,

where $x$ ranges over $\{0,1\}^d$ and $\ell_x(W) = \sum_{i=0}^{d-1} (x_i - \hat{x}_i)^2$. We sometimes refer to $x$ as a feature vector. In this post we take $m = 2, d = 5$ throughout.

The weight matrix $W$ has the shape $2 \times 5$ and maps each feature (meaning one of the standard unit vectors) to one of the $5$ columns of $W$ in $\mathbb{R}^2$. We refer to this vector as the representation of the feature. We visualize $W$ by drawing each vector colored by features zero (red), one (blue), two (green), three (orange) and four (purple).

Critical points and k-gons. At high sparsity, critical points of $L$ (that is, points where the gradient vanishes) are weight matrices $W$ whose columns form regular $k$-gons, where $k < 5$ occurs when $5 - k$ columns are zero. For precise statements see Elhage et al. (2022) and Chen et al. (2023). The figure below shows the ideal digon, triangle, square, and pentagon forms of the model.

Note that $W$ contains more information than just the polygon's shape: rotating all feature vectors by the same angle gives a different $W$ with identical loss, making the absolute orientation a redundant degree of freedom. There are also discrete symmetries: for example, the pentagon has a $D_5$ symmetry where permuting the feature labels or reflecting the polygon leaves the loss unchanged.

From gon to gon, down the landscape. Weight matrices that represent more features (that is, have more nonzero columns) achieve lower reconstruction error and thus lower loss $L$. This leads to one of the striking observations of Elhage et al. (2022): the learning trajectory, when initialized near the origin, passes through $k$-gons with increasing $k$ and in particular from $4$-gons to a global minimum, which consists of $5$-gons with all features represented:

Note that in the bottom row there are several distinct 5-gons, each representing a continuous space of global minima (obtained from the visualized weight matrix by the continuous rotational symmetries). These represent the 5-gons that can be reached from the particular 4-gon shown in the first row by learning a feature.

Patterning TMS. Earlier we said that patterning is about shaping models by shaping data. Here we have a perfect test case: can we change the data distribution such that, starting near the 4-gon, the training trajectory leads to a chosen 5-gon? Changing the data distribution means replacing $q$ by $q + \alpha \, dh$ for some $\alpha > 0$ and perturbation $dh$ with $\sum_x dh(x) = 0$, with a corresponding change in the loss to

where we assume $\alpha$ is small enough that the $q(x) + \alpha \, dh(x)$ are still in $[0,1]$. We think of the perturbation $dh$ in the data leading to a tilt or deformation of the loss landscape from $L(W)$ to $L(W;dh)$. In the following $\alpha = 0.3$.

So the question becomes: can we choose $dh$ such that training on $L(W; dh)$ from near the 4-gon ends up at a chosen 5-gon? Yes we can:

Note that since we train on a deterministic loss, the trajectory taken over training from near the 4-gon (and thus the final 5-gon reached) is determined by the initialization and the loss. Let us identify a binary vector $x$ with a subset of the features $\{ 0, 1, 2, 3, 4 \}$ in the obvious way. We notice a pattern in the data perturbations $dh$: to grow the new feature between represented features $i$ and $j$ we downweight feature vectors $\{ i \}$ and $\{ j \}$ and upweight feature vectors $\{ k \}$ with $k \notin \{ i,j \}$ and k not the new feature. Other feature vectors $x$ receive minimal changes.

In the next section we explain how this is achieved. More generally, we explain how to start with a patterning target (in this case the chosen 5-gon) and a current state of the network (in this case the 4-gon) and derive a principled tangent vector $dh$ in the space of probability distributions such that training on $L(W;dh)$ leads from the current state in the direction of the target. This is what we call patterning.

Controlling the Transition

In this section we explain how to use patterning to control the transition from the 4-gon to the 5-gon over training in the TMS.

We begin by formally introducing the notion of a patterning target: a characterization of the desired structure of the final trained model in terms of expectation values of observables under the local Bayesian posterior (Defining the Target). We introduce susceptibilities, which measure how these expectation values change under infinitesimal reweightings of the data distribution (Susceptibilities). Given a target change $d\mu$ in observables, we can invert the susceptibility matrix to get the data perturbation $dh$ that moves us toward it. This is the patterning step (The Patterning Step). We close by applying this to the 4-to-5-gon bifurcation and showing that a single patterning step at the bifurcation point is enough to steer the new feature into a chosen gap (Return to Patterning TMS).

Defining the Target

The goal of patterning is to produce a model with low loss on the original training distribution which satisfies some additional constraints on how that model generalizes (both in how it generalizes from the training data to the full training distribution, and also out-of-distribution). In this section we explain the “language” in which these targets are expressed, which is the language of expectation values of observables $\phi: \mathcal{W} \rightarrow \mathbb{R}$ where $\mathcal{W}$ is the space of weights for our neural network. In this section we consider a general neural network and write $w \in \mathcal{W}$ for the weight vector; recall that in the setting of TMS we write $W$ for the weight matrix.

Observables are suitably differentiable functions $\phi: \mathcal{W} \rightarrow \mathbb{R}$ and their expectation values are with respect to a distribution $p(w) \propto \exp(-\beta L(w))$, that is

where $Z_\beta = \int_{\mathcal{W}} \exp(-\beta L(w)) dw$ and $\beta > 0$ is called the inverse temperature. The distribution $p$ is called a Boltzmann distribution in physics (Gordon et al. 2026) and is a population version of the posterior distribution in statistics. We just refer to it as the posterior in this post (technically the tempered posterior because we allow $\beta \neq 1$).

Next we briefly motivate why control of such expectation values is a way to control local geometry.

Let $w^*$ be a local minimum of some (population) loss $L$ defined for some neural network and data distribution (not necessarily TMS). The local loss landscape means the function $L$ defined on some small open neighborhood around $w^*$. The geometry of this landscape controls generalization, in various ways that are spelled out in Watanabe’s work on singular learning theory. Since the goal of patterning is to control generalization, this means that at a high level the target of patterning is some condition on the geometry of the local loss landscape at $w^*$.

How do we impose conditions on geometry? The geometry of $L$ at $w^*$ is captured by its partial derivatives or, more canonically, by certain matrices and tensors of such. Of these the first is the gradient $\nabla L(w^*)$ which of course vanishes since $w^*$ is a local minimum, and the second is the Hessian $H = \nabla^2 L(w^*)$. So a special case of patterning would be to ask: how to find a global minimum of $L$ which satisfies some constraint on its Hessian matrix?

The general solution is complicated by two factors: firstly, we may need to control more than second order information (the Hessian) and secondly, we want to control the geometry of the population loss but only have access to a random variable (the empirical loss) whose mean is the population loss. This is not an issue in TMS where we have direct access to the population loss, but as we wish to exhibit the general principles we use the approach that generalizes.

To motivate the next step we recall a theorem. For smooth functions $\phi, \psi$ which vanish at $w^*$ the covariance with respect to $p$ is defined to be

It is well-known that when $L$ is nondegenerate at $w^*$ (meaning that the Hessian $H$ is invertible)

That is, the expansion of the covariance in powers of $\beta$ starts at order $\beta^{-1}$ and the coefficient “pairs” the gradient of $\phi, \psi$ via the symmetric matrix $H^{-1}$. By varying $\phi, \psi$ we can extract $H^{-1}$ and thus $H$. That is, the asymptotic expansion in $\beta$ of $\operatorname{Cov}_{w \sim p}\big[ \phi, \psi \big]$ knows the geometry of the loss landscape to second order through the coefficient of $\beta^{-1}$. Thus, to constrain the Hessian is to constrain this covariance.

This is the Way! To put constraints on the geometry of the local loss landscape, we need only put constraints on expectation values $\mathbb{E}_{w \sim p}[ \phi ]$ for some collection of $\phi$’s, that is, for some collection of observables (noting that the covariance is such an expectation value). That leaves two problems:

• The specification problem: how do we translate our informal patterning target (say a 5-gon with some particular cyclic ordering of features) to a formal patterning target, written in terms of constraints on such expectation values?
• The control problem: how do we steer the learning process to a local minimum $w^*$ where the local loss landscape satisfies those constraints?

In general, such as for applications to AI alignment, the specification problem remains difficult and unsolved. In TMS it is trivial, as we explain in the next section. The control problem is solved in principle by susceptibilities, the pseudo-inverse and a patterning control loop.

Observables for Patterning TMS

Recall that the weight matrix $W \in \mathbb{R}^{m \times d}$ where $m = 2$ and $d = 5$. We write $W = (W_0 \cdots W_{d-1})$ for the columns of $W$. We define two sets of observables:

It is intuitive that these 15 observables (10 dot products + 5 norms) should be able to, via their expectation values, distinguish the local minima of $L$ corresponding to 5-gons with different cyclic orientations (using the normalized dot product observables) and to distinguish the 4-gon from the 5-gon (using the norm observables). And indeed, using the angles between features to specify the cyclic permutation and the norms to specify that all features should be represented, we will succeed in patterning towards our desired 5-gon.

Susceptibilities

With $\mathcal{W}$ the space of weights for the TMS network, $W \in \mathcal{W}$ a weight, and $\phi: \mathcal{W} \rightarrow \mathbb{R}$ a differentiable function (an observable) we define $\mathbb{E}_{W \sim p}[\phi(W)]$ to be the expectation of the observable under the Boltzmann distribution $p(W) \propto \exp(-\beta L(W))$.

We can perturb the loss by an amount $\alpha > 0$ in a direction $dh$ by perturbing the weight given to each feature, and thus also perturb $p$:

where $x$ ranges over feature vectors $x \in \{0,1\}^d$.

The derivative of the expectation with respect to $\alpha$ (up to a constant multiple) is known as a susceptibility:

We can define a susceptibility matrix $\chi$ whose entry $\chi_{k,x}$ in row $k$ (indexing an observable from a list $\phi_1, \ldots, \phi_k, \ldots$) and column $x$ (indexing a datapoint) is the susceptibility $\chi(\phi_k,dh)$ which measures how the expectation value of observable $\phi_k$ responds when we infinitesimally upweight $x$, that is, we take $dh$ to be the direction which upweights $x$ and equally downweights all other samples.

This derivative has a more practical alternative form. The susceptibility can be expressed as a covariance under the unperturbed posterior,

We can learn how the system would respond to a perturbation by observing how it fluctuates in the absence of one. Fluctuations encode the local geometry of the loss landscape; responses encode how that geometry shifts under changes in data. Susceptibilities connect the two.

Methodologically, the covariance form of the susceptibility is what makes it possible to use susceptibilities in practice. Without it, computing susceptibilities would require independently varying the data distribution for each perturbation $h_i$. Instead, we can estimate susceptibilities by sampling from the posterior. In particular, we are concerned with the local posterior — the set of models around the model we are studying. By taking $T$ samples $W_1,\ldots,W_T$ from the local posterior, susceptibilities are estimated as:

where $\Delta L(W) = \ell_x(W) - L(W)$ and bars denote sample means.

The Patterning Step

Susceptibilities measure how expectation values respond to changes in the data. Patterning inverts this to ask what change to the data distribution would be needed to produce a desired change in expectation values. The basic idea of optimizing the data mixture rather than model weights alone is not new (see discussion on bilevel optimization and curriculum learning in Related Works). What’s new is that susceptibilities provide a principled way to compute the right change.

Given a target change $d\mu_{\text{target}}$ in the vector of expectation values

we want the minimum-norm sample reweighting $dh$ such that $\chi\, dh \approx d\mu_{\text{target}}$. The solution is:

when we assume $\chi$ has full row rank. In practice, we form the Gram matrix $G = \chi \chi^T$ and add ridge regularization $G_{\text{reg}} = G + \lambda I$. We solve for $v = G_{\text{reg}}^{-1} d\mu_{\text{target}}$ and project back: $dh_{\text{opt}} = \chi^T v$. The per-sample scores $s_i = (\chi^T v)_i$ tell us how much to upweight ($s_i > 0$) or downweight ($s_i < 0$) each training sample.

From $dh$ to training weights. The patterning equation produces a vector $dh \in \mathbb{R}^n$ that can take arbitrary sign and magnitude. To convert it into valid per-sample training weights, we start from the natural data weights $h_0$ (the Bernoulli base distribution over input types, normalized to mean 1), add the perturbation, floor at a small $\varepsilon > 0$ to prevent zero or negative weights, and renormalize:

The scalar $\alpha$ controls the overall perturbation scale. Normalization to mean 1 ensures the effective learning rate is unchanged. Note that the flooring and normalization modify the actual applied perturbation: the prediction $\delta\mu_{\mathrm{pred}} = \chi \cdot (\hat h - h_0)$ is the effective change in observables, not the raw output of the pseudoinverse. For small $\alpha$ these coincide; for large $\alpha$ the floor clips extreme negative entries.

The additive clipping scheme above is not the only option. We find that other methods of converting $dh$ into valid weights work comparably well, including exponentiating the scores ($\hat h_i \propto \exp(\alpha \cdot dh_i)$), which avoids the flooring issue entirely. In general, the patterning step is not particularly sensitive to this choice (at least in this toy model); most reasonable schemes produce similar results, though some tuning of the perturbation scale $\alpha$ is needed. For the ridge parameter, we find values around $10^{-3}$ to work well; the interpretation of this choice in terms of the mode structure of $\chi$ is discussed in mode-structure section (forthcoming).

Training on the reweighted data. The patterning step produces a set of weights $h$; we then train the model for multiple gradient steps on the reweighted loss $L(w; \hat h) = \frac{1}{n}\sum_x \hat h_x \ell_x(w)$ before re-estimating $\chi$. How many gradient steps to take after a patterning step is a design choice that we return to in The Patterning Controller.

Interpreting the Transition

In Susceptibilities we introduced the covariance form of the susceptibility matrix $\chi$, as well as how we can use this form to empirically estimate it. In this section we visually motivate the covariance form of the susceptibility, walk through the susceptibilities of the 5-gon, then study the susceptibilities of a model naturally undergoing the 4 to 5 gon transition (without any patterning taking place).

Digon

Triangle

Square

Pentagon

By coloring the samples from the posterior by the loss on some particular input (or set of inputs), we can see where in parameter space that input's loss is high and where it is low. This is the geometry of that input's local loss landscape. Below we do this for three inputs— each is colored by the loss at that sampled model on the indicated input. Darker regions of the clouds have a higher loss.

In the leftmost glyph, we color the samples by their loss on $(1,0,0,0,0) = e_0$, indicated by the first in the vector of little squares being colored. For this input, when $e_1$’s representation moves radially away from the origin (it grows) the loss on $e_1$ increases. The same thing happens for $(0,1,0,0,0) = e_1$ — when $e_1$’s representation (red) moves outwards the loss on $e_1$ increases.

The third example, $(1,1,0,0,0)$, is a little different. Here $e_0$ and $e_1$ are firing at the same time, meaning they are both active in the input. As before, the representations of both $e_0$ and $e_1$ have regions of higher loss and lower loss. However, this time the high loss zones aren’t perfectly aligned with the stems of their features, but rather are at an angle to them. This means that loss will increase if the features grow at a particular angle.

In particular, the fluctuation-dissipation theorem says that the direction the posterior would shift if we upweighted input $x$ is encoded in the correlation between the feature positions and the loss on $x$. For each feature $i$, this gives a displacement vector:

Plotting these arrows on top of the sample-clouds for our previous examples, we see that they point in the opposite direction of the highest loss regions of parameter space.

Each dot has a position (where the feature landed in this posterior draw) and a shade (the loss on input $x$ at that draw). The arrow compresses this into two numbers: the covariance of the feature's position with the loss on $x$, computed separately for each coordinate. If the x-coordinate of feature $i$ tends to be large when the loss on $x$ is high, that coordinate gets a large (negative, due to the sign convention) entry.

These arrows are the susceptibilities of the position of each feature to input $x$, or, in other words, how $\chi$ predicts the model would change if input $x$ were infinitesimally upweighted. By examining the colorings of the above point clouds, or more specifically the correspondence between directions features can move in and per-input losses, we were already looking at susceptibilities.

Below we show the sample-cloud and susceptibility arrows for each of the $32=2^5$ inputs.

We now track susceptibilities across a full 4 to 5-gon transition. At each epoch, we run SGLD at the current weights and estimate the susceptibility of each observable to each input.

In biological development, these transitions are where morphogens act. A cell at a bifurcation is maximally sensitive to molecular signals in its environment, meaning a small concentration gradient applied at the right moment is enough to commit a tissue to its fate. The same signal during a stable phase would do almost nothing. Susceptibility peaks tell us both how to intervene (which samples to reweight) and when (at transitions, when the model is responsive).

Return to Patterning TMS

Starting from a perfect 4-gon with a single ‘dead’ feature at the origin, there are four gaps the new feature can grow into. In the language of observables, placing f4 between f0 and f1 means asking for $\hat W_0 \cdot \hat W_4 = \hat W_1 \cdot \hat W_4 = \cos(\pi/4)$. In the language of observables, we target $\phi_{\text{dot}[0,4]} = \phi_{\text{cos}[1,4]} = 0.707$ (an interior angle of 45°), leaving the angles between $W_4$ and non-neighbors unconstrained.

Using the patterning equation, we compute the reweighting at the bifurcation point. Then, we train the model on the re-weighted distribution $h$ until the new feature $W_4$ has oriented itself into the target quadrant. At this point the target has been achieved, and so we switch back to the natural training distribution q.

The vanishing gradient means SGD cannot move $W_4$​ from the origin on its own. We therefore displace $W_4$ into a non-target quadrant before training begins. The displacement is a mechanical necessity, as gradient descent needs a nonzero gradient to act on, and is an informationally separate step from the patterning step: the data weights $h$ are computed at the origin and encode the target gap, while the displacement merely starts the flow. Regardless of the initial quadrant $W_4$ is placed in, training on the re-weighted data will cause it to swing into the desired quadrant within a small number of steps before it starts significantly increasing in norm.

Columns of the susceptibility matrix $\chi_i$ tell us how some input, if upweighted, would shift the expected value of our observables. When we pseudoinvert $\chi$, we are finding the minimum norm solution $dh$ such that $\sum_x {\chi_x \cdot dh_x} = d\mu$. In this way, our $dh$ is a linear combination of the per-input susceptibilities. Let us look again at the susceptibilities of the studied 4-gon with the fifth feature at the origin.

Earlier, in Toy Model of Superposition, we saw that the patterning on the target of placing the new feature into gap 0 (between $W_0$ and $W_1$, so in the upper right quadrant) would lead to $W_0$ and $W_1$ being downweighted, while $W_2$ and $W_3$ were upweighted. Looking at these susceptibilities, it is somewhat clear which sorts of inputs should receive which sort of weights. Solving the patterning, we can see it gives the weightings you would expect:

Inputs pushing $W_4$ up, to the right, or both are favoured. Inputs pushing the other way are down-weighted. In the next section we extend patterning to the multi-step regime, where we take multiple patterning steps in order to gain more coherent control.

Multi-Step Patterning

With patterning we aim to control the learning process by controlling the data distribution. In the previous sections we provided a simple example, by demonstrating and visualizing control over the 4-gon to 5-gon transition in TMS. This was a particularly simple patterning target, which requires only a “single step” of patterning: we measured susceptibilities at the 4-gon, derived our data re-weighting $dh$, trained forward on $L(W;dh)$ and arrived at our target.

In general this patterning loop of measuring susceptibilities at the current parameter and re-weighting the data distribution must be repeated multiple times in order to achieve the desired target. The simplest version of this loop is as follows:

Rewriting Minima

Only a single patterning step was needed to control bifurcations. This is somewhat due to the nature of this toy model, where once a gap is chosen via patterning training descends uninterrupted to the target minima. But what about moving between minima — taking a model that has already settled and pushing it somewhere else?

We take a trained 5-gon and ask: can we rearrange its features into a specified configuration? The target has the same loss as the original (by symmetry of the polygon), but there is no low-loss path between the two under the natural data distribution. Indeed each is a global minimum.

In this section we do three things. First, we classify all possible target permutations of the pentagon into three different groups, and show that within these groups all patterning targets are equivalent. Secondly, we demonstrate patterning on targets from these groups, studying the dynamics given by the patterning compiler. Thirdly, we step back and consider what patterning is doing geometrically: tracing a trajectory not through weight space but through the space of loss landscapes.

Classifying targets

Our observables are the ten pairwise cosine similarities $\hat W_i \cdot \hat W_j$ and the five feature norms $\|W_i\|$. Because every target is a rearrangement of the same regular pentagon, all norms are identical across targets — the targets differ only in which features are adjacent and which are not.

On a regular pentagon, the cosine between two features takes one of just two values, determined by whether the features are adjacent (gap 1, one edge apart) or non-adjacent (gap 2, two edges apart) on the original ring. A target is therefore fully specified by its gap pattern: for each pair of positions on the ring, are the two features placed there gap-1 or gap-2 apart?

Many different-looking rearrangements produce the same gap pattern. For instance, the 3-cycle $(f_1\;f_2\;f_3)$ and the double swap $(f_1\;f_2)(f_3\;f_4)$ place different features at every position, yet at every position pair the gap between the features placed there is the same — so the two rearrangements define the same target. This happens because one arrangement can be obtained from the other by relabeling all features via a rotation of the pentagon, which preserves all gaps.

Working out which rearrangements are observationally distinct, we find exactly 12 gap patterns — the 120 permutations of five features, modulo the 10-element dihedral group $D_5$ of rotations and reflections. One is the identity. The remaining 11 non-trivial targets are naturally organized by how many of the five original neighbor pairs they destroy — the break count.

2-breaks. Two features of the starting polygon exchange positions. The rest of the polygon is undisturbed — three of the five original neighbor pairs survive. Any single swap between features falls in this class, and all are equivalent under $D_5$. For example, swapping f2 ↔ f3 and swapping f4 ↔ f1 are the same task: both swap a pair of adjacent features, and the starting polygon's rotational symmetry maps one to the other.

3-breaks. Only two of the five original neighbor pairs survive. Each 3-break target can be described equally well as a 3-cycle rotating three features or as two simultaneous swaps — these very different operations produce the same gap pattern at every position pair, so they define the same observable target. All five 3-break targets are equivalent under $D_5$.

A rearrangement can break 0, 2, 3, or 5 of the five original neighbor pairs. Break counts of 1 and 4 are impossible: four intact edges of the pentagon necessarily form a path whose two endpoints are adjacent, but the single remaining step would need to connect non-adjacent features — a contradiction. (By a duality that swaps gap-1 and gap-2 steps, the impossibility of 1-break implies the impossibility of 4-break.)

The dihedral symmetry of the starting pentagon makes all targets with the same break count equivalent patterning problems: a $D_5$ rotation or reflection maps any one to any other, so the susceptibility structure, data reweightings, and patterning trajectory are identical up to relabeling of features. Using this target-taxonomy, we are able to succinctly study all possible ways to pattern between pentagons. We do so in this next section.

Trajectories in the Space of Data Distributions

Over the course of patterning a trajectory is traced through the space of data distributions. This trajectory inherently has a dual — the trajectory of the model through parameter space. As the model's internal structure evolves, its susceptibilities change, and so interpretable structure visible in susceptibilities is carried through to interpretable structure in the perturbations the patterning compiler is making to the data distribution.

What does this trajectory look like? We study our break classes, starting with the 2-break, swapping features 0 and 1.

The figure below shows the effective weights $\hat h$ (see The Patterning Step ) computed by the patterning equation, over the course of patterning on different break types (as well as the raw perturbations $dh$, if you click the button to the top left of the heatmap). Each row is an input type (labelled by which features are active); each column is a patterning step; color indicates the accumulated sample weight $\hat h_x^{(t)}$. During each patterning step we take 10 gradient steps on the re-weighted loss $L(w, \hat h)$. By mousing-over the plot, you can see the exact configuration of the model at each epoch.

The line plot above shows the natural, unweighted loss (dashed), and the patterning, weighted loss (solid). Note that we remove the Bernoulli base distribution from the weights so that emphasis is placed instead on how patterning is scaling these base weights (factoring in the base distribution would leave the rare inputs with an imperceptibly faint coloring). The different break types can be viewed by the buttons on the bottom right; we will first focus on the dynamics of the 2-break.

The 2-break

Looking at the 2-break, we can clearly see several phases of weightings as the feature crossing progresses. We now give a transcript of the major phases.

Initially (step zero), is heavily pushed up, alongside and . As we approach the crossover (step 68), and increase nearly in tandem. Once through the crossing, if we were to return to the unweighted loss the model would naturally settle at the permuted target. However, even after crossing we see significant weightings, with and increasing gradually before rounding off around step 150, where the target has been reached. After hitting the target (or at least very near it), the weights oscillate around in an attempt to keep the model stable.

These weightings are interesting but hard to interpret. Why up-weight at the start? Why the others?

Step Zero

This 2-break is asking us to swap $W_0$ and $W_1$. At epoch zero, we want $\delta\mu$ that moves $W_0$ toward $W_1$​'s position and $W_1$ toward $W_0$​'s. The patterning equation finds the linear combination of input-susceptibilities whose sum points in that direction. Let’s examine what $\chi$ ‘thinks’ its data weights are going to do to the model. We can check this by, as described in Interpreting the Transition, visualizing the susceptibilities of the model to each of these inputs at each point in training. At epoch zero, the susceptibility of the model to looks like: 1.Arrow lengths on each glyph show the raw per-feature susceptibility $\chi_W \cdot e_i$ — the direction and relative magnitude of the displacement each input would induce, independent of its weight. Arrows are normalized so that the largest susceptibility across all inputs and all epochs fills the glyph radius, making directions comparable across the full trajectory. The weight $\hat{h}_i$ shown above each glyph is a separate scalar multiplier applied by the patterning equation.1, where $W_1$ and $W_2$ move out, while $W_4$ moves in towards the center, getting shorter (note that this movement out causes the angle between them to decrease). and have symmetric effects — their susceptibilities look like and , respectively.

Taking a linear combination of just these susceptibilities, weighted as they were by the patterning equation at epoch 0, you would get:

Further including the top nine highest weighted inputs in the sum:

The sum reveals that the patterning equation, at step zero, is weighting inputs as to move $W_0$ and $W_1$ towards each other, as we would expect. Notice the weights given to and are nearly the same — 1.72 to the former, 1.71 to the latter. These inputs have a symmetric susceptibility, under reflection across the line drawn by $W_3$ (the orange feature). Patterning is making a symmetric perturbation to solve an inherently symmetric task. Similarly and are nearly equally weighted — these differences in weighting are due to the fact that we are estimating $\chi$ via samples from the posterior, which will never quite give numerically perfect values like an analytic solution would.

Step 38

The features are approaching one another. Things are less symmetric — the first step, taken on the slightly noisy weights, broke symmetry, and since then small differences have accrued. $W_0$ is now very slightly shorter than $W_1$. is predicted to have a similar effect on the model as at step 0, although smaller in magnitude. and are trying to pull the features across one another. $W_3$ (orange) has come in towards the origin slightly

Step 68

By now the features have nearly crossed. $W_0$ is now more significantly shorter than $W_1$. Modulo magnitude, and are again roughly symmetric, both having the effect of pulling $W_1$ (blue) and $W_0$ (red) across into the new gap, respectively. $W_0$ and $W_1$ are predicted to be pulled slightly down as they cross.

Step 145

The patterning target has essentially been reached — $\chi$ is now tasked with keeping it there. From here on out the weights oscillate slightly: this is a generic property of feedback loops, where each correction overshoots the target, and the next correction overshoots back in the opposite direction. There are standard fixes from the literature we could apply here, but for simplicity we leave these to future work.

Rather than walk through the 3 and 5-breaks in a similar fashion, we present the data for all breaks and encourage the reader to explore. Instead of displaying the given weight for each input numerically, as above, we factor the weights ($\hat h$) directly into the length of each arrow. This shows in one snapshot which inputs are being upweighted, and what their susceptibilities are.

Above we show the same patterning runs as before, except now the corresponding susceptibilities and their weightings over patterning rather than just the weights. Representations are colored according to the corresponding features present in each input.

The purpose of changing the data distribution is to change the loss landscape. In the next section we examine the structure of the loss landscapes defined at each patterning step.

Loss Landscaping

A data distribution defines a loss landscape. When patterning changes the per-input weights $h_i$, the surface the model trains against — $L(W; h) = \frac{1}{n} \sum_x h_x \ell_x(W)$ — changes with it. So the trajectory through data space we just examined has a dual: a trajectory through the space of loss landscapes.

The full landscape lives in 10 dimensions (two coordinates per feature, five features). To visualize it we take 2D slices: for each feature, sweep its position in the hidden plane while holding all other features fixed at their current trained values, recording the loss at each point. These slices show exactly the local force on each feature: the gradient $\nabla_{W_i} L(W; h)$ at the current configuration equals the gradient of feature $i$'s slice at its current position. So when a slice's basin sits to the left of the feature, that is where gradient descent is pulling it. The one thing the slices don't show is cross-feature curvature, so a configuration where every feature sits in a slice-basin is a critical point of the full loss but not automatically a stable minimum.

Even though we are only viewing slices of the loss landscape, we can see quite clearly how patterning is iteratively shifting the (per-feature posterior). In particular, as the features move to cross, the minima under the re-weighted loss are always in front of the current position of the features.

Under the unweighted Bernoulli base distribution, the slices are close to a faithful picture of the full landscape rather than a 2D projection of it. Single-feature inputs $e_i$ dominate the base distribution at high sparsity (their probability is $p(1-p)^{d-1}$, larger than any other cardinality), and a single-feature input's loss is nearly diagonal in the per-feature decomposition: $\ell_{e_i}$ depends primarily on $W_i$, with cross-feature dependence entering only through small interference terms. So, when single-feature-active inputs dominate the loss, the slice is close to the loss feature $i$ actually experiences. Still, care is required in interpreting these slices — the full, 10 dimensional landscape certainly is complex in ways not visible in 2D.

This is visible in particular for $W_0$ and $W_1$ here, where the posterior mass quickly switches to have the features crossed — $W_0$ and $W_1$ sort of ‘surf’ the wave into their target positions. In order to have control over the response of the model we need to ensure the perturbations we make are only slightly shifting the minima at which the model sits. We discuss this more in The Patterning Controller

Recall that and are getting majorly up-weighted in order to make the features cross — these are what give the $W_0$ and $W_1$ slices their ‘lima bean’ shape. We can see this by looking only at the loss on these inputs, or the per-feature per-input loss landscapes:

When we look at the linear combination of landscapes, exactly as we did in Trajectories in the Space of Data Distributions, we can see that

We can go further, and decompose the total landscape into per-input contributions. Each of the 32 input types contributes its own loss surface, and the weighted landscape is their sum. By re-weighting we are sort of ‘landscaping’ the topography of the loss, contouring by re-scaling the sub-landscapes that make it up. If you click the animated glyph for a given feature, the per-input view will open, and makes this decomposition visible: each small panel is one input's contribution to the landscape for a given feature, with opacity proportional to its current weight.

In the next section we draw on our new intuitions about the loss landscape of this model, connect ideas from control theory to build some scaffolding around our patterning loop.

The Patterning Controller

The idea that statistical observations of a system's behaviour can provide the information needed to control it — without requiring a mechanistic model of its internals — is among the founding insights of modern control theory (Wiener, 1948). The framework is general: we are trying to steer a system, and so we use a controller to observe its output, compare it to a target, and compute an intervention. The gap between target and output is the tracking error. The loop closes when the intervention changes the output, which changes the tracking error, which changes the next intervention.

The patterning loop wraps the susceptibility computation in a feedback controller. At each step, the controller observes the current structure (measuring observables $\mu$), estimates the state of the system (computing $\chi$), computes an intervention (solving for $dh$), applies it (training on the reweighted loss), and repeats.

The goal of control is to keep your system near a minimum at all times. In particular, perturbations need to be calibrated such that the predicted response of the model is the realized response of the model. Too large of a perturbation and you can take your system out of equilibrium. Being out of equilibrium means there are no nearby minima — the response of your system will be determined by the global properties of your energy landscape, rather than the measured local properties. This is a catastrophe in the technical sense: a qualitative change in the critical-point structure of the parameterized loss. From the controller's perspective it is also catastrophic in the colloquial sense.

Assuming that your controller is well calibrated, in that it avoids catastrophic perturbations, the system (in our case, the model) should coast between local minima without ever being taken out of equilibrium. Ideally we would not re-perturb the data before the model has equilibrated — doing so would be a sort of ‘eager’ action, as there would still be gas left in the tank of the previous weighting.

The loop as written re-estimates $\chi$ on a fixed schedule. This is suboptimal, as we sometimes recompute mid-flight while the previous step is still resolving, other times leaving the model stalled at a new equilibrium with no fresh intervention. Instead, we can re-estimate only when needed. The model equilibrating shows up as a drop in the first derivative of the weighted loss. When this number drops enough, the current weights have done all they can, and we need to re-estimate $\chi$ and compute a fresh perturbation. This is event-triggered control (Aström 2008).

By doing so we obtain a controller which is more efficient, in terms of total compute usage. The above figure showcases this adaptive method of recomputation, compared against the fixed recompute patterning regime used in prior sections.

When patterning on a 3-break target, a fixed schedule of one susceptibility estimate every 10 epochs takes about 37 patterning steps to reach the correct permutation: reaching the same target can take as few as three steps with the adaptive method. This works because the recompute, when properly calibrated, fires precisely at structural boundaries: when features begin to move, when they cross, and when they need to settle into their new positions.

Slow Data, Fast Model

Patterning is a procedure on two coupled spaces. The model parameters $w$ live in weight space $\mathcal{W}$, and the data weights $h$ live in data space $\mathcal{D}$. Together they form a joint state $(w, h) \in \mathcal{W} \times \mathcal{D}$, on which the patterning loop is a dynamical system: training moves $w$ under the gradient of the current weighted loss, and the controller moves $h$ via the patterning equation. Writing this out as a continuous-time flow:

The two equations are coupled: the model moves under a loss that depends on $h$, and the controller's action depends on $\chi$ measured at the current $w$. The parameter $\epsilon > 0$ is a timescale ratio, and captures how fast data evolves relative to the model.

Studying this joint system in full generality is hard. As physicists do, we can simplify the problem by taking limits. The natural limit here is $\epsilon \to 0$: data evolves slowly enough that the model can always catch up. The model equilibrates instantly between patterning steps, and the joint dynamics simplify dramatically. The model's parameters are no longer an independent variable; they are determined by the current data weighting through the equilibration map $w^*(h) = \arg\min_w L(w; h)$. Now the entire dynamics live on data space:

The right-hand side is the patterning step, evaluated at the equilibrated configuration. The left-hand side is the rate at which the controller moves through data space. Solving this differential equation traces out a trajectory $h(t)$, and the corresponding model trajectory $w^*(h(t))$ follows automatically.

We call this the slow-data, fast-model limit. The event-triggered recomputation technique described above lets us realise this limit in practice. By waiting for the weighted loss to plateau before re-perturbing, the controller ensures that $h$ only changes when $w$ has equilibrated, which satisfies the condition $w \approx w^*(h)$ that the slow-data limit assumes. For this reason, we see this limit is a natural setting for a theory of multi-step patterning, where now we can be primarily concerned with the structure of our data and how it couples with our observables and targets.

Related Works

This work sits at the intersection of several research areas: singular learning theory, data attribution and optimization, constrained optimization, and alignment. We survey each below, organized around how patterning relates to and differs from existing approaches.

Singular Learning Theory and Developmental Interpretability

Patterning comes most directly out of a research agenda we have been developing over the past three years [Hoogland et al. (2023), Murfet (2023)], applying singular learning theory Watanabe (2009) to interpretability and alignment.

What is singular learning theory? First, some background: many classical results in learning theory (including asymptotic normality, the Bayesian Information Criterion, the Bernstein–von Mises theorem) assume regular models, that is, models whose loss landscapes have isolated, locally quadratic minima. For neural networks, this assumption fails. The loss landscape is degenerate, with non-isolated minima with Hessians that have zero eigenvalues. The classical results that were built on regularity do not apply.

Singular learning theory relaxes the regularity assumption and generalizes many of the classical results to the singular setting. It is a theory of Bayesian learning, for which the learning process is governed by Bayes' rule as the dataset size $n$ grows, rather than by gradient descent dynamics in increasing timestep $t$. Watanabe's central result (Watanabe 2009) derives closed-form asymptotics for (in-distribution) generalization error in the large-data limit, replacing the parameter count in the BIC with a geometric invariant known as the real log canonical threshold (RLCT) or, alternatively, the learning coefficient.

The developmental interpretability agenda. The developmental interpretability agenda (Hoogland et al. 2023) began with several key hypotheses. First, that Watanabe's Bayesian theory could serve as a productive idealized model of SGD-based training. This has borne out empirically across a series of works spanning toy models and now increasingly larger language models [Wei et al. (2023), Lau et al. (2025), Chen et al. (2023), Hoogland et al. (2025), Wang et al. (2025)]. Second, that SLT could be extended beyond the asymptotic, in-distribution, Bayesian setting into a more general theory of deep learning, even at finite dataset sizes and in other limiting regimes, even under distribution shifts, and even for non-Bayesian learning algorithms.

Susceptibilities [Baker et al. (2025), Wang et al. (2025), Gordon et al. (2026), Kreer et al. (2025), Adam et al. (2025), Lee et al. (2025)] are the first step towards the second extension: they measure the first-order correction to posterior expectations under a vanishing shift in the data distribution, and form the basis of our approach to interpretability, which we call “Spectroscopy”. This provides a principled route towards understanding how model internals and behaviors respond to changes in the data distribution.

Patterning, introduced by Wang & Murfet (2026) and applied to the TMS here, provides evidence for the last extension.

Influence Functions and Data Attribution

Susceptibilities are a generalization of influence functions. Influence functions [Cook & Weisberg (1980), Koh & Liang (2017)] measure the sensitivity of model predictions to changes in training data importance. This can be seen as a special case of a susceptibility where the perturbation is a per-sample weight (the same as considered here) and the posterior can be approximated as Gaussian (Kreer et al. 2025):

where $H=\nabla^2 L_n(w)$ is the Hessian at the local minimum.

Influence functions face two major problems, one practical and one theoretical. The practical problem is that Hessian inversion is intractable for large models. The theoretical problem is the Hessian inverse is not even well-defined for neural networks. Susceptibilities solve both.

Susceptibilities bypass the Hessian inversion. Many methods within the field of training data attribution (TDA) can be understood as approximations to the full Hessian inverse. This includes EK-FAC (Grosse et al. 2023), TRAK (Park et al. 2023), DataInf (Kwon et al. 2024), and even the simple inner product of gradients. Estimating susceptibilities bypasses the Hessian inversion step by directly computing statistics over the local loss landscape.

Susceptibilities measure influence at all orders. Classical influence functions assume that the model is at an isolated local minimum with an invertible Hessian $H$. These are precisely the same regularity assumptions we discussed previously that break down for neural networks. As a consequence of this breakdown, the empirical Hessian inverse is technically ill-defined. This can be “fixed” by regularizing the Hessian, $H \to H + \lambda \mathbb 1$, but this is not enough: at the 4-to-5-gon bifurcation, the Hessian is exactly zero and contains no information about which branch the model will take. Regularizing $H$ makes the matrix invertible but does not conjure any new signal.

Understanding the response of the model in general, especially at bifurcations, requires going to fourth order in loss landscape geometry. This is exactly the information that is available to susceptibilities.

Appendix

Analytically Solving the 4 to 5 gon Bifurcation

At the 4-to-5-gon transition, the dead feature $W_4$ sits at the origin. The total loss near this point turns out to be purely quartic in $|W_4|$: the quadratic term vanishes exactly due to a cancellation between interference costs and self-reconstruction benefits. The Hessian at the origin is the zero matrix.

This has two consequences. First, any method that relies on at most second-order information (including adaptive stochastic optimizers like Adam and classical influence functions) gets no signal at this point, and will struggle even in the vicinity of this point: the loss gradients with respect to $W_4$ are zero, so there is nothing for the Hessian to invert. Second, the angular structure that determines which gap the feature grows into lives entirely in the per-sample losses, not the total loss. This is the structure that susceptibilities detect and that patterning exploits.

At the transition point, the system sits at a critical point—a saddle in the loss landscape. Near this saddle, the loss has the form:

where $\ell = |W_{\text{new}}|$ is the magnitude of the new feature. When $a < 0$, the saddle becomes unstable and the feature spontaneously grows—but the critical point doesn't specify which branch. That choice depends on which valley the system descends into.

Four active features form a regular square at unit norm, with the dead feature parametrised in polar coordinates: $W_4 = r(\cos\theta, \sin\theta)$. By parameterising in this way, the total loss turns out to be remarkably clean:

The quadratic term in $r$ vanishes exactly, for all angles $\theta$. This cancellation has a satisfying origin: the interference that a growing $W_4$ causes on the four existing features contributes $+r^2$ to the loss (summing over the four quadrants via the tight-frame property of the square), while the self-reconstruction benefit of having a nonzero $W_4$ contributes $-r^2$ (from $\ell_4 = (1-r^2)^2 + r^2$). These cancel perfectly, leaving only the quartic. The Hessian at the origin is the zero matrix.

The posterior at this point factorizes in a useful way: the angle $\theta$ is uniformly distributed and independent of $r$, while $r$ follows the quartic distribution $p(r) \propto r,e^{-\beta r^4}$. While the total loss is independent of $\theta$, the individual per-feature losses are not. Each feature's loss fills exactly one quadrant of the angle:

Feature 0's loss is largest when $W_4$ points toward $W_0$, because that is where the interference is worst. This angular structure — invisible in the total loss but present in each per-sample loss — is what allows data reweighting to steer the direction of growth.

On the right is the analytically computed susceptibility matrix $\chi_{\mathrm{dot}[j,4], x}$. Each row is a dot-product observable $\phi_{\mathrm{dot}[j,4]}$; each column is a per-sample weight $h_x$. The matrix has anti-diagonal structure with entries $\pm\tau$ (where $\tau = \frac{2}{3\pi^{3/2}}\beta^{1/2} \approx 0.847$) connecting each feature to its diametrically opposite partner. The $h_4$ column is zero: the dead feature's own data weight does not bias the angular direction. On the left is the same matrix, but computed using SGLD to sample the posterior. This structure directly encodes the branching geometry of the bifurcation.

Computing the covariances under the quartic posterior gives the susceptibility of the dot product observables in closed form, depicted above. The susceptibilities tell us that increasing the weight on feature $i$'s data pushes the dead feature away from feature $i$ and toward the diametrically opposite feature, because the added weight penalizes interference in that direction. Orthogonal features have zero effect on each other's angular susceptibility. The dead feature's own weight $h_4$ does not bias the angular direction at all — it is a uniform destabilizer, controlling whether the feature grows but not where.

Visualizing these data weightings, we can see that $\chi$ is predicting that the new feature will grow into the gap.

The moments where developmental control matters most are precisely the singular points where classical methods fail. This is not a limitation of a particular implementation; it is a structural feature of bifurcations. The signal that patterning needs lives in higher-order geometry that the Hessian cannot see. Hence the need for more powerful techniques like patterning.