Literature Record: Risks from Learned Optimization
What It Establishes
The paper introduces mesa-optimization: the case where a trained model is itself an optimizer, searching internally over a space of actions or plans. It then asks two questions — when does training produce an optimizer at all, and when it does, what objective does that inner optimizer pursue, and how does it differ from the loss it was trained under.
The answer to the second question is the load-bearing part: the inner objective is selected by the outer training pressure, not specified by it. A model can be a competent optimizer for the training objective while pursuing something else that happened to score well on the training distribution. The paper names this the inner alignment problem and distinguishes it from the outer alignment problem of specifying the right loss.
Rating Justification
- Textual Authority (B): retrieved and read at abstract level; preprint, not peer-reviewed.
- Corroboration (2): this is a theoretical paper. Its claims are analytical, so corroboration means independent theoretical and empirical uptake, which exists — LIT-011 supplies the empirical mechanism and LIT-005 supplies a constructed demonstration — but neither is a test of the paper's own predictions about real training runs.
What It Does Not Show
It does not show that mesa-optimization actually occurs in contemporary large models. The paper is explicit that it is analysing a possibility and setting a research agenda. Six years on, whether current systems contain mesa-optimizers is not settled by this paper, and treating it as settled is the error the vault's CON-004 is careful to avoid.
Relation to the Vault
This is the theoretical source for the vault's central negative claim: training is not a binding mechanism, because the objective that gets trained is not necessarily the objective that gets learned. CON-004 states this as a consequence of weights being removable; LIT-004 states it as a consequence of how learned optimizers work. The two arguments are independent and arrive at the same place, which is why MAP-001's L3 row survives the literature even though its stated mechanism does not.
It also supplies the concept the corpus lacks for INS-008: the overseer is not the only optimizer in the arrangement, and the inner one is not observable from the loss curve.