The Local Learning Coefficient: A Singularity-Aware Complexity Measure
Authors
Affiliations
Edmund Lau University of Melbourne Zach Furman Timaeus George Wang Timaeus Daniel Murfet University of Melbourne Susan Wei University of MelbournePublished
Aug 23, 2023Links
Abstract
Deep neural networks (DNN) are singular statistical models which exhibit complex degeneracies. In this work, we illustrate how a quantity known as the learning coefficient introduced in singular learning theory quantifies precisely the degree of degeneracy in deep neural networks. Importantly, we will demonstrate that degeneracy in DNN cannot be accounted for by simply counting the number of 'flat' directions.