Deployment Simulation

OpenAI's pre-release safety method: replay recent production conversations with a candidate model (strip the old final response, regenerate, grade) to forecast deployment-time undesired-behavior rates before launch — then validate the forecasts post-release; trades compute for coverage, cuts evaluation awareness to near-production levels, surfaced 'calculator hacking' pre-release, and — per the OSF-preregistered GPT-5.4 study — beats adversarially-selected-production baselines but not a naive previous-rate baseline