Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory

Let $\mathcal{X}$ denote a sample space and let $q^{(n)}\in\Delta(\mathcal{X}^{n})$ be an unknown data-generating distribution on the space of $\mathcal{X}$-sequences $\mathbf{x}^{\left(n\right)}=(x_{1},\dots,x_{n})\in\mathcal{X}^{n}$ of length $n\in{\mathbb{N}}$. We assume that $\mathcal{X}$ is finite (e.g., the token vocabulary for modern transformer language models). Any distribution $p^{(n)}$ on $\mathcal{X}^{n}$ gives rise to a prefix-free (thus uniquely decodable) encoding, $\llbracket\mathbf{x}^{\left(n\right)}\rrbracket_{p^{(n)}}$, for any $\mathbf{x}^{\left(n\right)}\in\mathcal{X}^{n}$ with code length given by $\mathfrak{len}\left(\llbracket\mathbf{x}^{\left(n\right)}\rrbracket_{p^{(n)}}\right)=-\log p^{(n)}\left(\mathbf{x}^{\left(n\right)}\right)$. Conversely, every prefix-free encoding can be used to define such distributions (Kraft, 1949; McMillan, 1956), which we shall call an encoding distribution.

A central observation of the MDL principle is that any statistical pattern or regularity in $q^{(n)}$ can be exploited to compress samples $\mathbf{x}^{\left(n\right)}$ of $q^{(n)}$. If a learning algorithm can extract these regularities through only samples $\mathbf{x}^{\left(n\right)}$, then it has implicitly learned a good compression of $\mathbf{x}^{\left(n\right)}$. This is the oft-invoked principle of “learning as compression”. Throughout, we will consider learning machines with a finite-dimensional parameterized statistical models, denoted as ${\mathcal{M}}=\left\{p^{(n)}_{w}\in\Delta(\mathcal{X}^{n})\mid w\in W\right\}$ where $W\subset{\mathbb{R}}^{d}$ is a compact $d$-dimensional parameter space. An important example for this work is the case of modern auto-regressive language models where $\mathbf{x}^{\left(n\right)}$ are the token sequences and the model $p^{(n)}_{w}$ takes the form

$$p^{(n)}_{w}(\mathbf{x}^{\left(n\right)})=p_{w}(x_{1})p_{w}(x_{2}|x_{1})p_{w}(x_{3}|x_{1},x_{2})\dots p_{w}(x_{n}|x_{1},\dots,x_{n-1})$$

for some learned sequence-to-next-token model $p_{w}$, such as a transformer (Vaswani et al., 2017).¹¹1This is an example of what is known as prequential code in MDL literature. For exposition, we will focus on the case where both the data and model are independent and identically distributed (i.i.d., see discussion of assumptions in Appendix F). This means that, for every $n\in{\mathbb{N}}$, the data distribution and model distribution on $\mathcal{X}^{n}$ are respectively given by

$\displaystyle q^{(n)}\left(\mathbf{x}^{\left(n\right)}\right)=\prod_{i=1}^{n}q(x_{i})\quad\text{and}\quad p^{(n)}_{w}\left(\mathbf{x}^{\left(n\right)}\right)=\prod_{i=1}^{n}p_{w}(x_{i})$

for some unknown $q$ and model $\left\{p_{w}\right\}$. Under such assumptions, the unique minimum average code length in the long-run (large $n$) is achieved by setting the data generating distribution itself as the encoding distribution, i.e., setting $p^{(n)}=q^{(n)}$. The expected per-symbol excess length compared to this minimum is measured by the Kullback-Leibler (KL) divergence,

$$D_{\mathrm{KL}}\left(q\middle\|p_{w}\right):={\mathbb{E}}_{x\sim q}\left[\log\frac{1}{p_{w}(x)}\right]-{\mathbb{E}}_{x\sim q}\left[\log\frac{1}{q(x)}\right]=\mathcal{H}\left(q,p_{w}\right)-\mathcal{H}(q),$$

where $\mathcal{H}$ denotes the (cross-)entropy. We will call the first parameter-dependent term above the population loss and denote it as $\mathcal{L}(w)$. Note that the empirical estimate of $\mathcal{L}$ given by $\mathrm{L}_{n}(w)=-\frac{1}{n}\sum_{i=1}^{n}\log p_{w}(x_{i})$ is the usual per-token cross-entropy criterion used for training modern transformer network, also known as the average negative log-likelihood at $w$.

We will focus on the case of two-part codes to clarify the underlying geometrical phenomenon and explain the direct relevance to neural network compression. To communicate with two-part codes, the sender and receiver agree to first communicate an encoding distribution $p$ by sending some encoded representation $\llbracket p\rrbracket$, before sending the data encoded with $p$, $\llbracket\mathbf{x}^{\left(n\right)}\rrbracket_{p}$. Once $p$ is received, the receiver can reconstruct the encoding distribution and decode any message encoded with $p$. The result is a message of length

$$\mathfrak{len}(\llbracket p\rrbracket)+\mathfrak{len}(\llbracket\mathbf{x}^{\left(n\right)}\rrbracket_{p})=\mathfrak{len}(\llbracket p\rrbracket)+\sum_{i=1}^{n}\log\frac{1}{p(x_{i})}.$$

One measure of code performance, known as redundancy, is defined to be the excess code length as compared to the encoding generated using the data distribution itself as encoding distribution. So, if the data is drawn i.i.d. from $q$, then the redundancy of the two-part code is given by

$$R_{n}:=\mathfrak{len}(\llbracket p\rrbracket)+\mathfrak{len}(\llbracket\mathbf{x}^{\left(n\right)}\rrbracket_{p})-\sum_{i}\log\frac{1}{q(x_{i})}=\mathfrak{len}(\llbracket p\rrbracket)+\sum_{i}\log\frac{q(x_{i})}{p(x_{i})}.$$

(1)

Notice that if the chosen encoding distribution $p$ is sufficiently good in the sense of having small KL-divergence, ${\mathbb{E}}_{q}\left[\log\frac{q(x)}{p(x)}\right]=D_{\mathrm{KL}}\left(q\middle\|p\right)$, from $q$, then it is worth paying $\mathfrak{len}\left(\llbracket p\rrbracket\right)$ bits to obtain a cheap encoding for samples drawn from $q$.

Suppose the sender and receiver have a shared knowledge of a finite dimensional statistical model (e.g., a neural network architecture). This allow them to communicate using codes specific to the biases implicit in the model architecture. In other words, the model provides an implicit prior on the set of distributions ${\mathcal{M}}$, allocating codes of varying length depending on which hypotheses are considered simple or complex.

To the state the main theorem let ${\mathcal{M}}=\left\{p_{w}\in\mathring{\Delta}_{m}(\mathcal{X}):w\in W\subset{\mathbb{R}}^{d}\right\}$ be a statistical model of finite dimension $d\in{\mathbb{N}}$. We will assume some technical conditions on the model laid out in Appendix (E.1). In particular, we require that all distributions in our models are in the restricted simplex $\mathring{\Delta}_{m}(\mathcal{X})$ of uniformly lower bounded distributions where given $m>0$ and a distribution $p$ over $\mathcal{X}$ we say $p\in\mathring{\Delta}_{m}(\mathcal{X})$ if $\min_{x\in\mathcal{X}}p(x)\geq m$.

We refer to Watanabe (2009) for the definition of the learning coefficient $\lambda$ and multiplicity $m$.

To establish this result, we need to specify a way for the sender to communicate a specific hypothesis or distribution $p$ in the model. We note that a model generally contains uncountably many distinct distributions, yet any parameter encoding can specify at most countably many. Thus, discretization is needed. We assume the sender and receiver have a way to construct, for any $\epsilon>0$, shared finite sets $Q_{\epsilon}=\left\{p_{1},p_{2},\dots,p_{N_{\epsilon}}\right\}\subset{\mathcal{M}}$ such that any $p\in{\mathcal{M}}$ belongs to some set of the form $P_{\epsilon}(p^{*}):=\left\{p\in{\mathcal{M}}:D_{\mathrm{KL}}\left(p\middle\|p^{*}\right)\leq\epsilon\right\}$ where $p^{*}\in Q_{\epsilon}$²²2This is a finite $\epsilon$-net of the model in distribution space.. Let us define (R for Reversed)

$$V^{R}_{p}(\epsilon):=\mathrm{Vol}\left(\left\{w\in W:D_{\mathrm{KL}}\left(p_{w}\middle\|p\right)\leq\epsilon\right\}\right)\,.$$

Given $p\in{\mathcal{M}}$, we assume a consistent and shared algorithm for choosing³³3breaking ties consistently when needed $p^{*}\in Q_{\epsilon}$ such that $p\in P_{\epsilon}(p^{*})$ and $V^{R}_{p^{*}}(\epsilon)=\max\big\{V^{R}_{p^{\prime}}(\epsilon)\mid p^{\prime}\in Q_{\epsilon}\text{ and }p\in P_{\epsilon}(p^{\prime})\big\}$.

Observe that this produces a partition of the model, ${\mathcal{M}}$, with each set in the partition represented by a grid point $p^{*}$ in $Q_{\epsilon}$⁴⁴4possibly a smaller set, in which case we take $Q_{\epsilon}$ to be this non-redundant set. Ideally, we would like a tight $\epsilon$-KL-sphere packing. If ${\mathcal{M}}$ is a subset of the interior of $\Delta(\mathcal{X})$ simplex, itself, with non-empty interior, we can use construction similar to Balasubramanian (1996) to obtain such an $\epsilon$-net.. We will then assign probability⁵⁵5this is only an approximate equality because the true volume should be the volume of the partitions instead of those of the $\epsilon$-nets. However, their difference can be made small via careful selection of centers $p^{*}$ and sphere packing arguments. $\approx\frac{V^{R}_{p^{*}}(\epsilon)}{\mathrm{Vol}(W)}$ to $p^{*}$ and therefore a code of length

$$\mathfrak{len}(\llbracket p^{*}\rrbracket):=\log\frac{\mathrm{Vol}(W)}{V^{R}_{p^{*}_{n}}(\epsilon)}\,.$$

Notice that this is very different from putting the uniform distribution on ${\mathcal{M}}$ (e.g., by using the Jeffrey’s prior on $W$ if ${\mathcal{M}}$ is regular). We are deliberately assigning shorter codes to hypotheses $p^{*}\in Q_{\epsilon}$ that are simpler according to the model’s own implicit bias: a hypothesis is simpler to state relative to a given model if it takes up more parameter volume (requires lower parameter-precision to specify its distribution) up to $\epsilon$ error tolerance.

With such a construction, we can now calculate the two-part code length with respect to some model ${\mathcal{M}}$ for i.i.d. data drawn from data distribution $q$ that is realizable ($q\in{\mathcal{M}}$) and satisfies assumptions in Appendix (E.1). Let $\hat{p}=\arg\min_{p\in{\mathcal{M}}}\sum_{i=1}^{n}\log\frac{1}{p(x_{i})}$ be the maximum likelihood distribution and define $p^{*}_{n}$ be the grid point in $Q_{\epsilon}$ closest to $\hat{p}$, $p^{*}_{n}:=\arg\min_{p\in Q_{\epsilon}}D_{\mathrm{KL}}\left(\hat{p}\middle\|p\right)$. To send the data $\mathbf{x}^{\left(n\right)}$ we send the encoding of $p^{*}_{n}$ and the data encoded with this distribution. Writing $K_{n}(p)=\frac{1}{n}\sum_{i=1}^{n}\log\frac{q(x_{i})}{p(x_{i})}$ the redundancy of the code at tolerance $\epsilon$ is given by

	$\displaystyle R_{n}$	$\displaystyle=\log\frac{\mathrm{Vol}(W)}{V^{R}_{p^{*}_{n}}(\epsilon)}+nK_{n}(p^{*}_{n})$		(2)
		$\displaystyle=\log\frac{\mathrm{Vol}(W)}{V^{R}_{p^{*}_{n}}(\epsilon)}+n\underset{(\star)}{\underbrace{D_{\mathrm{KL}}\left(q\middle\|p^{*}_{n}\right)}}+n(K_{n}(p^{*}_{n})-D_{\mathrm{KL}}\left(q\middle\|p^{*}_{n}\right))\,.$		(3)

Now, we introduce a dependency of the tolerance on $n$ by $\epsilon_{n}=\frac{a}{n}$ for some $a>0$. With this assumption, both $nD_{\mathrm{KL}}\left(q\middle\|p^{*}_{n}\right)$ and $n(K_{n}-D_{\mathrm{KL}}\left(q\middle\|p^{*}_{n}\right))$ are $O_{p}(1)$ by Theorem 5 and Theorem 3 respectively. Therefore, the redundancy is given asymptotically by

$$R_{n}=-\log V^{R}_{p^{*}_{n}}(\epsilon_{n})+O_{p}(1).$$

In (3) we see a fundamental tradeoff: decreasing the error tolerance $\epsilon$ (a finer grid) decreases the excess code length $(\star)$ because we can find a grid point $p^{*}_{n}$ closer to $\hat{p}$ and thus $q$, but decreasing the tolerance will also decrease the volume $V^{R}_{p^{*}_{n}}(\epsilon)$ and thus increase the cost for communicating $p^{*}_{n}$. Similar to the case for regular models (see for example Balasubramanian (1996)), the optimal grid size for data set of size $n$ scale as $\epsilon_{n}=O\left(\frac{1}{n}\right)$: any higher order rate of decay for $\epsilon_{n}$ implies a finer distinguishability of grid points than the number of data points $n$ can justify (the MLE itself has $D_{\mathrm{KL}}\left(q\middle\|\hat{p}\right)=O_{p}(1/n)$, see further discussion in Appendix E.4).

It remains to determine the behavior of the $V^{R}_{p^{*}_{n}}(\epsilon_{n})$. One difficulty is that the center $p^{*}_{n}$ is a random variable depending on data and changes with $\epsilon_{n}$. However, it is also clear that, as $\epsilon_{n}\to 0$, $p^{*}_{n}$ approaches in KL-divergence to the data generating distribution $q$. Furthermore, the relevant volume will also be similar to that of the set of parameter where $p_{w}$ is close to $q$ defined by $V_{q}(\epsilon):=\mathrm{Vol}\left(\left\{w\in W:D_{\mathrm{KL}}\left(q\middle\|p_{w}\right)\leq\epsilon\right\}\right)$. Theorem 4 shows that there exist $C>0$ such that for any $\epsilon>0$ and $p^{*}\in{\mathcal{M}}$ such that $D_{\mathrm{KL}}\left(q\middle\|p^{*}\right)\leq\epsilon$ we get

$$V_{q}\left(\epsilon\right)\leq V^{R}_{p^{*}}(C\epsilon)\leq V_{q}\left(\frac{C}{2}(C+1)\epsilon\right).$$

(4)

This allows us to make conclusions about $V^{R}_{p^{*}}(\epsilon)$, for $p^{*}$ such that $D_{\mathrm{KL}}\left(q\middle\|p^{*}\right)\leq\epsilon$, by investigating $V_{q}(\epsilon)$. To do that, we invoke a central result of SLT:

Applying this theorem to the map $w\mapsto V_{q}(\epsilon)$ and using Equation (4), we get

$$c\epsilon^{\lambda}\left(-\log\epsilon\right)^{m-1}\leq V^{R}_{p^{*}}(C\epsilon)\leq c\left(\frac{1}{2}C(C+1)\right)^{\lambda}\epsilon^{\lambda}\left(-\log\epsilon-\log\frac{1}{2}C(C+1)\right)^{m-1}.$$

Using the fact that $(-\log\epsilon+a)^{m-1}/(-\log\epsilon)^{m-1}\to 1$ as $\epsilon\to 0$ for any $a\in{\mathbb{R}}$, we conclude there exist $c^{\prime},c^{\prime\prime}>0$ such that for sufficiently small $\epsilon$,

$\displaystyle c^{\prime}\epsilon^{\lambda}\left(-\log\epsilon\right)^{m-1}\leq V^{R}_{p^{*}}(C\epsilon)\leq c^{\prime\prime}\epsilon^{\lambda}(-\log\epsilon)^{m-1}.$

(5)

This in turn implies⁶⁶6note that Equation (5) shows that $\frac{V^{R}_{p^{*}}(C\epsilon)}{V^{R}_{p^{*}}(\epsilon)}=O(1)$ for $\epsilon\to 0$.

$$-\log V^{R}_{p^{*}}(\epsilon)=\lambda\log\frac{1}{\epsilon}-(m-1)\log\log\frac{1}{\epsilon}+O_{p}(1).$$

(6)

Finally, recalling that we took grid scale to be $\epsilon_{n}=a/n$. For sufficiently large $a>0$, this implies that $D_{\mathrm{KL}}\left(q\middle\|p^{*}_{n}\right)\leq\epsilon_{n}$ with high probability by Theorem 5. Therefore, the result about in Equation (6) applies and plugging in expression for $\epsilon_{n}$, we get

$$R_{n}=\lambda\log n-(m-1)\log\log m+O_{p}(1)$$

which concludes the proof for Theorem 1. Notice that the leading order terms above can be interpreted as model complexity: it is the code length required to communicate a sufficiently good encoding distribution $p^{*}_{n}$ in the model while maintaining an $O(1)$ excess length for the encoded message even when the number of sample $n\to\infty$.

We remark that Theorem 2 is a consequence of the celebrated theorem on the resolution of singularities by Hironaka (1964). The scaling exponent $\lambda$ is known as the real log canonical threshold (RLCT) of the analytic function $w\mapsto D_{\mathrm{KL}}\left(q\middle\|p_{w}\right)$ and $m$ is its multiplicity. Watanabe (2009) was the first to make use of the resolution of singularities and thereby connect these geometrical invariants to statistical learning, showing that $\lambda\log n$ gives the leading-order term for model complexity and $\frac{\lambda}{n}$ gives the leading-order term for generalization error in Bayesian learning. In the context of machine learning, $\lambda$ is referred to as the learning coefficient. For regular models, the sublevel sets look ellipsoidal (Figure 2 top-left), with volume $\sim\epsilon^{d/2}$ and thus the learning coefficient is $\lambda=\frac{d}{2}$ where $d$ is the parameter count. Its multiplicity is $m=1$. Indeed, there are simpler two-part code construction for regular model that achieves $R_{n}=\frac{d}{2}\log n+O_{p}(1)$ by just having regular rectangular grid in $W$ of scale $O\left(\frac{1}{\sqrt{n}}\right)$ (corresponding to KL-divergence scale of $O(1/n)$ in the space of distributions). Observe that this leading order behavior of $R_{n}$ for regular model is independent of data distribution $q$. For singular model, $\lambda<\frac{d}{2}$, which means models can potentially be much more compressible than their explicit parameter count suggests. Figure 2 (top-middle and top-right) illustrates how the sublevel sets can have complex geometry with degeneracies, resulting in larger sublevel set volume that allow for higher level of compressibility.

In the previous section we established the existence of a two-part code in which the leading term of asymptotic redundancy (excess code length compared to the encoding which we would use if we knew the true data distribution) is $\lambda\log n$ where $\lambda$ is the learning coefficient.

This is directly related to compression (Grünwald and Roos, 2019) as it tells us the number of bits needed to communicate a set of samples $\mathbf{x}^{\left(n\right)}$ between a sender and receiver who share a statistical model. This MDL perspective captures the idea that a model class which allows for simpler representations of a given data distribution (smaller $\lambda$) offers better compression of its samples. However, it remains to explain how any given practical compression scheme (e.g., quantization) fits into this story. In this section we provide a less formal argument based on the concepts introduced in the previous section which aims to explain this connection in a straightforward way.

From a mathematical perspective parameters live in a continuous space $W\subseteq{\mathbb{R}}^{d}$, but any realization in a computer uses some kind of grid with spacing $h>0$. Fix a local minimum $w^{*}$ of the population loss $\mathcal{L}$ and define the local excess loss $K(w)=\mathcal{L}(w)-\mathcal{L}(w^{*})$. We consider only parameters in a neighborhood near $w^{*}$ that is small enough that $K(w)$ is non-negative. Invoking the sublevel-set volume law (Theorem 2), there exist numbers $\lambda(w^{*})$ and $m(w^{*})$ such that

$$V(\epsilon):=\mathrm{Vol}\left(\left\{w\in W:K(w)\leq\epsilon\right\}\right)\sim c\epsilon^{\lambda(w^{*})}\bigl(-\log\epsilon\bigr)^{m(w^{*})-1}\,.$$

(7)

Here $\lambda(w^{*})$ and $m(w^{*})$ are known as the local learning coefficient (LLC) and multiplicity, introduced in Lau et al. (2024). Our in the remainder of this section is to connect the resolution $h$ to the loss tolerance $\epsilon$ through the LLC $\lambda(w^{*})$.

Consider the quantization cell $C_{h}(w)=\{u:\|u-w\|_{2}\leq h/2\}$ around a parameter $w$ with a volume proportional to $h^{d}$. To guarantee that quantization does not increase excess loss beyond $\epsilon$, it is sufficient that the cell containing $w^{*}$ be contained in the $\epsilon$-sublevel set around $w^{*}$. A surrogate for this containment is the volume condition $\mathrm{Vol}(C_{h})\leq V(\epsilon)$ or

$$h^{d}\leq\epsilon^{\lambda(w^{*})}\,\bigl(\log\tfrac{1}{\epsilon}\bigr)^{m(w^{*})-1}\,.$$

(8)

If we write $n_{q}$ for the number of intervals for each coordinate in our grid, then this behaves like $1/h$. We denote by $h^{*}$ and $n_{q}^{*}$ the level of quantization that reaches the loss tolerance $\epsilon$ and therefore makes (8) an equality. Hence, writing the per-coordinate bit budget as $b^{*}(\epsilon)=\log_{2}n_{q}^{*}$ we have

$$b^{*}(\epsilon)=\frac{\lambda(w^{*})}{d}\,\log_{2}\frac{1}{\epsilon}+O\!\Bigl(\frac{\log\log(1/\epsilon)}{d}\Bigr).$$

(9)

Thus, for a fixed loss tolerance $\epsilon$ the critical bits per coordinate grows linearly with the LLC. Intuitively with larger $\lambda$ (less degeneracy), the admissible basin is smaller, so smaller cells (finer grids, more bits) are needed to keep the entire cell inside the basin. This is illustrated in Figure 2.

We quantize models using a symmetric quantization scheme that includes $0$. Given $n_{q}\in 2\mathbb{Z}_{>0}$ and $m>0$ we divide the intervals $[0,m]$ and $[-m,0]$ into $\tfrac{1}{2}n_{q}$ intervals of length $\Delta=m/(\tfrac{1}{2}n_{q}-1)$ so that in each interval there are $\tfrac{1}{2}n_{q}$ possible values lying at the endpoints of subintervals (including $0$ and $\pm m$). Combining these to form $[-m,m]$ and accounting for double counting of $0$ there are $n_{q}$ intervals and $n_{q}-1$ possible quantized values. To quantize a parameter $w\in W\subseteq\mathbb{R}^{d}$ with $w=(w_{1},\ldots,w_{d})$ means firstly to “clamp” each $w_{i}$ to the interval $[-m,m]$ and then round these values to the nearest quantized value in this interval according to the above subdivision. More precisely we define $w^{\text{quant}}_{i}:=\mathrm{round}\left[\frac{w_{i}}{\Delta}\right]\Delta$ and $w^{\text{quant}}=(w_{1}^{\text{quant}},\ldots,w_{d}^{\text{quant}})$. Note that specifying each $w_{i}^{\text{quant}}$ requires $\log_{2}(n_{q})$ bits.

In Section 5 we treat $m$ as a free parameter and search for a value that minimizes the loss of the quantized model. This is our baseline method, inspired by a more sophisticated approach in Cheong and Daniel (2019) where they allow for non-evenly spaced quantization intervals.

The increase in loss caused by quantization is a function $\Delta\text{Loss}=\mathrm{L}_{n}(w^{\text{quant}})-\mathrm{L}_{n}(w)$ of $n_{q}$ and $w$. This is typically larger when $n_{q}$ is smaller. We measure the compressibility of a language model with parameter $w$ by finding the smallest $n_{q}$ with $\Delta\text{Loss}(w)\leq\epsilon$ and call this value the critical $n_{q}$ and denote it $n_{q}^{*}$. When $n_{q}^{*}$ is large the model is less compressible (we hit the threshold with a smaller amount of compression), and conversely when $n_{q}^{*}$ is small the model is more compressible.

In Section C.3 we show results of the cruder quantization method of setting $m$ to the largest parameter absolute value, which is equivalent to the scheme used by Kumar et al. (2025).

We consider a transformer neural network that models the conditional distribution $p(y|x;w)$ of outputs $y$ (next tokens) given inputs $x$ (contexts), where $w\in W$ represents the network parameters in a compact parameter space $W$. Given samples $D_{n}$ from a true distribution with associated empirical loss $\mathrm{L}_{n}$, we define the estimated local learning coefficient at a parameter point ${w^{*}}$ to be:

$$\hat{\lambda}({w^{*}})=n\beta\left[{\mathbb{E}}_{w|{w^{*}},\gamma}^{\beta}[\mathrm{L}_{n}(w)]-\mathrm{L}_{n}({w^{*}})\right],$$

(10)

where ${\mathbb{E}}_{w|{w^{*}},\gamma}^{\beta}$ is the expectation with respect to the Gibbs posterior (Bissiri et al., 2016),

$$p(w|{w^{*}},\beta,\gamma)\propto\exp\left\{-n\beta\mathrm{L}_{n}(w)-\frac{\gamma}{2}\|w-{w^{*}}\|^{2}_{2}\right\}\,.$$

(11)

The hyperparameters are the sample size $n$, the inverse temperature $\beta$, which controls the contribution of the loss, and the localization strength $\gamma$, which controls proximity to ${w^{*}}$. For a full account of these hyperparameters, we refer the reader to Watanabe (2013); Lau et al. (2024); Hoogland et al. (2025). Our LLC estimation procedure uses the preconditioned stochastic gradient Langevin dynamics (pSGLD) algorithm (Li et al., 2015). This combines RMSNorm-style adaptive step sizes with SGLD (Welling and Teh, 2011). For more details on LLC estimation and its uncertainties, see Appendix D.