The Local Learning Coefficient: A Singularity-Aware Complexity Measure
Authors
Edmund Lau =
University of Melbourne
Zach Furman =
Timaeus
George Wang
Timaeus
Daniel Murfet
University of Melbourne
Susan Wei
University of Melbourne
Abstract
Deep neural networks (DNN) are singular statistical models which exhibit complex degeneracies. In this work, we illustrate how a quantity known as the learning coefficient introduced in singular learning theory quantifies precisely the degree of degeneracy in deep neural networks. Importantly, we will demonstrate that degeneracy in DNN cannot be accounted for by simply counting the number of 'flat' directions.
Main contributions:
- The LLC is theoretically well-defined. Earlier, Watanabe introduced the global learning coefficient. This paper introduces a local variant and shows that this is well-defined.
- The LLC can be estimated. This paper shows that the LLC can be estimated by using SGLD-based posterior sampling combined with a Gaussian localization term.
- The estimated LLC respects ordinality. Given ground truth knowledge that one model is more complex than other, the estimated LLCs respect this ordering.
See the accompanying distillation.
Cite as
@inproceedings{lau2025local,
booktitle = {Proceedings of the 28th International Conference on Artificial Intelligence and Statistics},
series = {Proceedings of Machine Learning Research},
volume = {258},
pages = {244--252},
editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz},
month = {03--05 May},
publisher = {PMLR},
pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/lau25a/lau25a.pdf},
url = {https://proceedings.mlr.press/v258/lau25a.html},
title = {The Local Learning Coefficient: A Singularity-Aware Complexity Measure},
author = {Edmund Lau and Zach Furman and George Wang and Daniel Murfet and Susan Wei},
year = {2023},
abstract = {Deep neural networks (DNN) are singular statistical models which exhibit complex degeneracies. In this work, we illustrate how a quantity known as the learning coefficient introduced in singular learning theory quantifies precisely the degree of degeneracy in deep neural networks. Importantly, we will demonstrate that degeneracy in DNN cannot be accounted for by simply counting the number of 'flat' directions.}
}Click to copy