Literature Record: Goal Misgeneralization in Deep RL
What It Establishes
Goal misgeneralization is a failure of out-of-distribution generalisation in which an agent retains its capabilities while pursuing the wrong goal. The canonical illustration is an agent that continues to navigate competently but goes to the wrong place — the competence transfers, the objective does not.
This is the empirical mechanism behind LIT-004's theoretical claim. It shows that capability and objective can come apart under distribution shift, which is the observable signature of a model that learned something other than what the loss specified.
Rating Justification
- Textual Authority (B): retrieved at abstract level; preprint, not peer-reviewed.
- Corroboration (3): demonstrated across multiple RL environments and subsequently treated as an established failure mode, with LIT-004 supplying the theory it instantiates and LIT-003 supplying a language-model analogue.
Correction Notice
This record replaces a false identifier. The register previously gave arXiv:2109.10430, which resolves to a genetic-algorithm paper on web-service selection. The correct identifier is arXiv:2105.14111. See Identifier Audit.
Relation to the Vault
This supports MAP-001's L3 row — training does not bind — but for a different reason than the corpus gives. The corpus says training is defeated by deliberate circumvention. LIT-011 says training can fail without any circumvention at all, because the objective that generalises is not the objective that was specified. The row survives; its stated mechanism needs replacing. See CNF-003.