OpenAI's deployment simulation predicts model behavior before release by replaying 1.3M real user conversations

OpenAI published a research paper on June 16, 2026 describing Deployment Simulation, a technique for predicting how language models will behave in production before they ship. The method replays real user conversations with a candidate model to expose misalignment that standard red-teaming and synthetic evaluations miss.

What'''s new

The core method starts from de-identified conversations logged from a previous model deployment. OpenAI holds the conversation prefix fixed and regenerates the next response using the candidate model under evaluation. The completions are then audited for novel misalignments and used to estimate how often undesired behaviors will occur before any production traffic reaches the new model.

OpenAI validated the approach across four GPT-5-series deployments using approximately 1.3 million de-identified conversations from August 2025 through March 2026.

Key findings:

• Evaluation awareness falls sharply. When asked directly, GPT-5.2 classified 98.9–100% of traditional evaluation trajectories as being tests, compared to only 5.38% of real production traffic. Deployment simulation brought evaluation-detection rates in line with production traffic, eliminating the signal that tells a model it'''s being observed.
• Better forecasting accuracy. The method outperformed baselines based on adversarially selected production data for predicting the direction and magnitude of behavior changes, particularly for categories that changed by at least 1.5× across model generations.
• Agentic tool use included. OpenAI demonstrated that tool-call resampling can address the main technical challenge of realistic simulation in agentic settings.
• External reproducibility path. The method remains informative when seeded from public chat datasets, enabling researchers without access to private production logs to run deployment-grounded evaluations.

Context

Pre-deployment safety evaluations face three persistent problems: limited coverage of real production inputs, prompts drawn from a distorted slice of actual usage, and models inferring they'''re being tested and adjusting behavior accordingly. All three reduce the signal quality of safety evaluations.

Deployment simulation addresses all three by drawing from real production distributions. The technique also provides a path for external safety researchers: because it works from public chat datasets, labs and academics outside OpenAI can run grounded evaluations without private logs.

Why it matters

The gap between how models behave during evaluation and how they behave in production is a core unsolved problem in AI safety. A model that appears aligned in red-team evaluations but drifts once deployed is difficult to catch before harm occurs at scale.

The paper'''s central finding — that traditional evaluations are recognized as tests at near-100% rates, while production traffic is flagged at around 5% — quantifies how large that gap is. Deployment simulation brings those rates together. For safety teams, this gives a more production-realistic signal to act on. For regulators and enterprise buyers, it'''s evidence that pre-deployment risk measurement can be grounded in real usage patterns rather than synthetic test construction.