black O2, inc.
ACTIVE
    FOR-001 / 70_LiteratureLIT-001

    AI Control: Improving Safety Despite Intentional Subversion

    Verified literature

    title
    AI Control: Improving Safety Despite Intentional Subversion
    Type
    literature
    Identifier
    LIT-001
    Citation
    CIT-011
    authors
    • Ryan Greenblatt
    • Buck Shlegeris
    • Kshitij Sachan
    • Fabien Roger
    year
    2023
    venue
    arXiv preprint (v5, 23 July 2024)
    identifier
    arXiv:2312.06942
    url
    https://arxiv.org/abs/2312.06942
    retrieved
    2026-11-13
    Textual authority
    B
    Corroboration
    3
    Status
    Verified
    bears on
    • [[SYS-001]]
    • [[MAP-001]]
    • [[CON-002]]
    contests
      Tags
      • literature
      • ai-control
      • protocols
      • red-team
      • verified

      Literature Record: AI Control — Improving Safety Despite Intentional Subversion

      What It Establishes

      The paper defines a research agenda it calls AI control: safety engineering that assumes the model may be intentionally subverting the operator, and asks what protocols keep outcomes safe anyway. The organising move is that safety is a property of the deployment protocol — the arrangement of trusted and untrusted models, monitors, editing steps and audits — rather than of the model's disposition.

      The evaluation is concrete. It builds a setting of backdoored programming solutions, gives a red team the job of playing a subverting model, and measures whether candidate protocols still produce safe code. Two protocol families are compared: trusted editing, where a weaker trusted model rewrites the untrusted model's output, and untrusted monitoring, where the untrusted model monitors itself and collusion must be made expensive. The model pair is GPT-4 as untrusted and GPT-3.5 as trusted.

      Rating Justification

      • Textual Authority (B): retrieved and read at the abstract and framing level. It is an arXiv preprint, not peer-reviewed, so it cannot take A on this vault's scale regardless of its influence.
      • Corroboration (3): the protocol framing is not a single paper's idiosyncrasy — it is the organising frame of a research line with multiple institutional homes (see LIT-006, LIT-008), and its threat model is independently presupposed by the empirical results in LIT-002 and LIT-003.

      What It Does Not Show

      It does not show that control protocols work on models substantially more capable than the trusted monitor, which is the regime that matters. Its own results are conditional on a capability gap between trusted and untrusted model — the same gap the Citadel Stack would have to specify. It is an existence demonstration on one task family, not a deployment guarantee.

      Relation to the Vault

      This is the single most important external document for the vault, because it states as its founding premise what SYS-001 derives independently: binding layers external to the model are the load-bearing ones. The convergence is real and it is not citation — see CNF-001.

      The paper also names, in its own threat model, the failure modes that MAP-001's L0–L5 table has no row for: monitor–monitor collusion, steganographic side channels between untrusted instances, and a model that evades a monitor by modelling it. See CNF-002.

      Back to Shackled AIForests