In May 2024, a paper titled "The Attacker in the Mirror" was published on arXiv, introducing a fundamentally new approach to bypassing the safety mechanisms of large language models. Rather than relying on external attacks or fine-tuning with malicious data, the researchers employ "anchored bipolicy self-play"—a method where a single model simultaneously acts as both attacker and defender, governed by "anchors" that preserve its core policy.
This mechanism works by having the model generate pairs of trajectories during self-play: one where it attempts to violate its own safety rules, and another where it tries to prevent those very violations. By anchoring the original policy to prevent total degradation, the method exposes internal inconsistencies in the model’s self-alignment. Consequently, after several iterations, the model begins to successfully generate harmful content that it would previously have blocked.
The authors demonstrate that even models trained with RLHF and Constitutional AI show a dramatic decline in resilience against their own self-generated attacks. On benchmarks where refusal rates previously exceeded 95%, attack success reached 40–60% after just 10–15 rounds of self-play. Notably, external safety metrics measured by standard tests remained almost entirely unchanged.
Methodologically, this study stands out from previous research because it requires neither weight access nor additional fine-tuning. Everything occurs within the model's own context through role alternation. This makes the attack particularly dangerous: it can be executed even by a user without special privileges, provided the model supports a sufficiently long context and can maintain two conflicting policies simultaneously.
Compared to earlier research, such as Anthropic’s work on sleeper agents or red-teaming via adversarial prompting, this approach exploits the internal structure of the policy rather than searching for external triggers. While sleeper agents required specific data poisoning during the training phase, anchored bipolicy self-play operates on pre-trained models, uncovering vulnerabilities that do not surface during typical usage.
For the AI community, these findings suggest that current safety evaluation methods—based on static tests and external red-teaming—are no longer sufficient. A model may appear safe across all standard metrics while remaining vulnerable to attacks it is capable of generating itself. This calls into question the reliability of approaches that depend on policy self-consistency as a primary defense mechanism.
