Saddles and Metastability in SLT

This project aims to extend Singular Learning Theory (SLT) beyond local minima to saddle points and investigate metastability in neural networks. We’ll focus on understanding the physical meaning of LLC estimates at saddle points and explore potential extensions or alternatives to the real log canonical threshold (RLCT) for these scenarios.

Key research questions:

1. What is the physical interpretation of LLC estimates at saddle points?
2. How can we extend SLT theory to rigorously account for saddle points and metastable states?
3. Is the RLCT the appropriate geometric invariant for saddle points, or do we need an alternative?
4. How do saddle-to-saddle dynamics in deep linear networks (DLNs) relate to SLT concepts?

Methodology:

1. Develop theoretical extensions to SLT that encompass saddle points and metastable states.
2. Implement LLC estimation for saddle points in simple models, starting with deep linear networks.
3. Analyze the behavior of LLC estimates around saddle points in various network architectures.
4. Investigate the relationship between LLC dynamics and saddle-to-saddle transitions in DLNs.
5. Explore alternative geometric invariants that might better characterize saddle points in the context of SLT.
6. Develop and test hypotheses about the physical meaning of LLC estimates at saddle points.

Expected outcomes:

1. Theoretical framework extending SLT to saddle points and metastable states.
2. Characterization of LLC behavior around saddle points in various neural network architectures.
3. Insights into the relationship between SLT concepts and saddle-to-saddle dynamics in DLNs.
4. Potential development of new geometric invariants for characterizing saddle points in machine learning.
5. Improved understanding of the role of saddle points and metastability in neural network training dynamics.

This research could provide crucial insights into the optimization landscape of neural networks and potentially lead to new optimization strategies that leverage saddle point dynamics.