The question
Reinforcement learning can pull a language model away from the assistant persona it learned in pretraining. This project asks whether that persona becomes less stable before the model starts reward hacking, early enough to act as a warning.
Why it matters
It bears on one practical decision: should RL training dashboards carry a cheap, perturbation-based persona probe? Many reward hacks never show up in the reward signal, so a signal from inside the model could warn where training logs cannot. A clear negative result would also be useful, because it would tell monitor designers not to rely on this kind of probe.
Method
Most current monitors measure where the persona sits. This project measures how firmly it is held. At each training checkpoint, the persona is nudged with a fixed prefill, and I measure how fast it relaxes back. In physics, recovery that slows as a system nears a tipping point is called critical slowing down, a standard early-warning signal.
The testbed is UK AISI's public reward-hacking model organism: 150 checkpoints from a GRPO training run on OLMo-7B, with a step-by-step record of when hacking begins. The relaxation measure is compared against three kinds of baseline: static persona projections, drift nulls, and behavioral monitors. The metric, layer, prefill and onset definition will be pre-registered on the starting model before I look at the training trajectory.
Status
Pre-registration in progress; started November 2026. There are no results yet.
Planned output
A post on LessWrong and the AI Alignment Forum, and a public code repository, planned for around week 6. The write-up will report the outcome whether the signal works or not.