LLCs and Unlearning

This project aims to investigate how unlearning procedures, particularly LEACE (Linear Concept Erasure), affect the Local Learning Coefficient (LLC) of neural networks. Understanding the impact of unlearning on the LLC could provide insights into how these procedures affect model complexity and internal structure.

Key research questions:

1. How do unlearning procedures like LEACE change the LLC of a model?
2. Can changes in the LLC during unlearning reveal information about the concepts being erased?
3. How does the impact of unlearning on the LLC vary across different model architectures and sizes?
4. Can LLC analysis be used to improve or validate unlearning procedures?

Methodology:

1. Implement LEACE and other unlearning procedures for a variety of model architectures.
2. Estimate LLCs before, during, and after the unlearning process.
3. Analyze how different types of concept erasure affect the LLC.
4. Investigate whether changes in the LLC correlate with successful concept erasure.
5. Compare the impact of unlearning on the LLC across different model architectures and sizes.

Expected outcomes:

1. Empirical data on how unlearning procedures affect the LLC.
2. Insights into the relationship between concept erasure and model complexity as measured by the LLC.
3. Potential development of LLC-based metrics for evaluating the effectiveness of unlearning procedures.
4. Improved understanding of how unlearning affects the internal structure of neural networks.

This research could provide valuable insights into the nature of unlearning and its effects on model complexity, potentially leading to improved techniques for concept erasure and model editing.