For most of the last five years, evaluating a language model meant giving it a question and grading the answer. That approach is breaking down as models are increasingly deployed as agents that run for hours or days at a stretch — booking travel, managing a codebase migration, monitoring a customer's infrastructure — and the field is scrambling to build evaluations that match.
"A single-turn eval tells you almost nothing about whether an agent will still be doing the right thing six hours into a task," said Marguerite Solis, who leads long-horizon evaluation work at a safety-focused research lab. "You can pass every short benchmark we have and still drift, still start optimizing for the wrong proxy, still quietly stop asking for clarification when you should."
The new generation of long-horizon evals typically works by dropping a model into a simulated environment — a fake company's internal tools, a sandboxed codebase, a mock customer support queue — and letting it run largely unsupervised for an extended session, sometimes simulating days of wall-clock time compressed into a few hours of compute. Graders then look not just at task completion but at a handful of second-order signals: did the model's stated plan stay consistent with its actions, did it flag its own uncertainty at appropriate moments, did it notice when the task itself had quietly changed underneath it.
That last one has turned out to be surprisingly diagnostic. Several eval designs now deliberately alter the task partway through — a customer's request changes, a spec is updated, a dependency the agent was relying on gets deprecated — specifically to see whether the model notices and adapts, or plows ahead executing a plan that no longer matches reality. Researchers describe the latter failure mode, sometimes called "goal ossification" internally, as one of the more common and more concerning patterns they've found: models that were never told to stop noticing changes, but that stop anyway once a plan has enough momentum behind it.
Building these evals is expensive in ways single-turn benchmarks aren't. A long-horizon eval can require standing up a full simulated environment, instrumenting it to log everything an agent touches, and then having a human — or another model, with its own known blind spots — review a transcript that may run to hundreds of thousands of tokens. Several labs have started using models to do first-pass grading of other models' long-horizon transcripts, a practice that has its defenders and its skeptics in roughly equal measure.
"The honest answer is that we don't yet know how much we can trust a model to accurately grade another model's week-long transcript for subtle drift," Solis said. "We use it because the alternative is not grading these transcripts carefully at all, and that's worse. But it's a real limitation, not a solved problem, and I'd be suspicious of anyone in this field who tells you otherwise."
The push toward longer evaluations tracks a broader shift in how these systems are actually being used: fewer isolated questions, more standing responsibilities. Whether the evaluation methodology can keep pace with the deployments it's meant to be checking is, for now, an open question — and one several people in the field describe as the most important one in their work.