The task horizon: how fast the models are moving, why, and what a token costs now

There is one number that tells you more about where these tools are than any benchmark score, and it is not a score. It is the length of task a model can finish on its own. This briefing sets out how that is measured, what the measurement hides, why the curve bent when it did, what the same capability costs now against two years ago, and where the whole thing stops working. It is written on 9 February with the sources available then.

The measurement

METR, a non-profit that evaluates frontier models, defines a model's time horizon as the human task duration at which its fitted success curve crosses 50%. Task duration is itself the geometric mean of the times taken by human baseliners who completed it, described in the original paper as skilled professionals in software engineering, machine learning and cybersecurity, mostly from top-100 universities with around five years' experience. The original March 2025 suite was 170 tasks in three families: HCAST, 97 general engineering tasks from a minute to about thirty hours; RE-Bench, seven eight-hour machine-learning research tasks; and SWAA, 66 very short atomic actions.^[1]

Time Horizon 1.1, published on 29 January, grew the suite to 228 tasks, moved the evaluation harness onto the UK AI Security Institute's open-source Inspect framework, and revised the trend. The doubling time since 2023 is now 130.8 days with a confidence interval of 107 to 161, against 196.5 days measured over the full history and 88.6 days if you count only from 2024. Claude Opus 4.5 comes out at 320 minutes, with an interval of 170 to 729.^[2] That interval is the honest part. The update also shows why: of the 31 tasks in the suite lasting eight hours or more, only five have an actual measured human baseline. The rest are estimated. The long end of this curve is the part we know least about.

And 50% reliability is not a standard anyone would run a business on. The original paper gives both numbers for the same model: Claude 3.7 Sonnet had a 50% horizon of about 59 minutes and an 80% horizon of about 15.^[1] A fourfold gap between "works half the time" and "works four times in five" is the single most useful correction to apply to every headline in this field, including the ones in this briefing.

The study that says the opposite

METR also ran the experiment nobody else was running. Sixteen experienced open-source developers took 246 tasks on their own repositories, mature projects averaging over 22,000 GitHub stars and a million lines of code, individually randomised to allow or forbid AI. The work ran from February to June 2025, mostly with Cursor Pro and Claude 3.5 or 3.7 Sonnet. Allowing AI increased completion time by 19%, with a confidence interval from 2% to 39%. Before the study, the developers had forecast a 24% speed-up; afterwards, having been measured as slower, they still estimated AI had made them 20% faster.^[3]

The authors tested twenty possible explanations. The ones that survived: developers accepted fewer than 44% of AI generations, spent about 9% of their time reviewing and cleaning AI output, and gained least where they were already expert and familiar with the code. Their own summary of the mechanism is that the tooling performs worse in more complex environments.^[3] They are also careful about what they do not claim: not that AI fails to speed up developers generally, not that their sample represents most software work, and not that no better way of using the tools exists. Hold both results at once. A model's unaided horizon is growing fast. A person's measured throughput on their own familiar, complex codebase did not grow at all.

Why the curve bent

The mechanism has a name and a date. OpenAI's o1 post of September 2024 stated the principle: performance improves both with more reinforcement learning at training time and with more time spent thinking at inference. The reported numbers were 74% on the 2024 American Invitational Mathematics Examination from a single sample, rising to 83% with a 64-sample majority vote, and 77.3% on GPQA Diamond.^[4] The training signal behind it is reinforcement learning with verifiable rewards, formalised in AI2's Tülu 3 paper: select tasks with verifiable outcomes, and provide reward only when a generation is checked correct. A deterministic checker, an exact match or a unit test, replaces a learned reward model, which is why mathematics and code are the favoured domains and why the gains have been lumpy across others.^[5]

The algorithm most of the open models use came from DeepSeek's mathematics paper. Group Relative Policy Optimisation "foregoes the need for additional value function approximation as in PPO, and instead uses the average reward of multiple sampled outputs, produced in response to the same question, as the baseline".^[6] In practice: sample a group of completions for one prompt, score each with the verifiable reward, and compute each one's advantage against the group's own mean. No critic network is trained at all, and in the standard method the critic is as large as the policy. That is the memory saving that let a smaller lab train a reasoning model.

Because thinking costs money, the labs made it a dial. OpenAI shipped a reasoning-effort setting with o3-mini in January 2025; Anthropic shipped a token budget with Claude 3.7 Sonnet in February, letting a caller cap thinking at a chosen number of tokens up to the output limit; Google shipped a numeric thinking budget with Gemini 2.5 in March.^[7] For anyone costing a workload, that dial is the main lever, and it is the one most teams never touch.

Where it stops working

Apple's "Illusion of Thinking" paper, published in June 2025, tested reasoning models on puzzles with controllable complexity: Tower of Hanoi, checker jumping, river crossing, blocks world. It found three regimes. At low complexity, ordinary models do better. At medium complexity, reasoning models win. At high complexity, both suffer what the authors call complete accuracy collapse, with Hanoi failing at seven or eight disks. Worse, the models counterintuitively reduce their reasoning effort as problems get harder, and performance did not improve even when the solving algorithm was supplied in the prompt.^[8] The rebuttal, posted days later, is worth the same weight: it argues the Hanoi tests systematically exceed the models' output token limits, that the automated grader cannot distinguish a reasoning failure from a practical constraint, and that some river-crossing instances are mathematically impossible for more than five actors given the boat capacity, yet are scored as failures.^[9] The rebuttal's first author was an AI, removed from later versions in line with arXiv policy. Both papers are right about something.

The other limit is context. Every lab sells a window measured in the hundreds of thousands or the million, and the windows do not hold up. NoLiMa tested thirteen models claiming at least 128,000 tokens: at 32,000, eleven of the thirteen had fallen below half their short-context score, and GPT-4o specifically fell from 99.3% to 69.7%.^[10] Nvidia's RULER found the same across seventeen models, with only about half of those claiming 32,000-token support actually holding up at it.^[11] Chroma's context-rot study ran roughly 194,000 calls across eighteen models and found that even a single distractor reduces accuracy, and, strikingly, that shuffled and less coherent haystacks outperformed logically structured ones.^[12] Anthropic's own engineering guidance draws the operational conclusion: treat context as a precious, finite resource, retrieve just in time rather than front-loading, and compact long histories rather than accumulating them.^[13]

What it costs

List prices understate the fall, because the capability at each price rose. The measurement that captures it is Epoch's: the price of matching GPT-4's performance on GPQA Diamond, a set of PhD-level science questions, fell about 40 times a year, within an overall range across benchmarks and thresholds of 9 to 900 times a year.^[17] On top of list price sit two standing discounts that most teams leave on the table. Both OpenAI and Anthropic price batch processing at half the synchronous rate. Prompt caching on Anthropic's tiers costs about 1.25 times the base input rate to write and about a tenth of it to read, which for a workload that re-sends the same long document is the difference between viable and not.^[18] One caution in the other direction: long context carries a premium. Gemini 2.5 Pro doubles from $1.25 to $2.50 per million input tokens above 200,000, and Claude's one-million-token beta did the same.^[19][20]

Underneath the prices, the hardware kept moving. Nvidia announced Rubin at CES on 5 January: 336 billion transistors per GPU, 50 petaflops of four-bit inference, and a rack pairing 72 of them with 36 custom Arm CPUs. Nvidia's own wording was that Rubin systems will start shipping in the second half of 2026, so at the time of writing it is announced, not deployed.^[21] The four-bit format itself is the interesting part: Nvidia's published measurement on DeepSeek-R1 puts the accuracy loss from FP8 at 1% or less on key language tasks for a 3.5-fold memory reduction against FP16.^[22] Google's Ironwood TPU reached general availability in November 2025 at 4,614 teraflops and 192GB of memory per chip; AWS announced Trainium3 in December at 144GB and 4.9 terabytes a second per chip.^[23][24] Three credible silicon suppliers is a better position for a buyer than one.

And reliability

Speed is not the constraint most businesses hit. A paper from OpenAI researchers in September 2025 argued that hallucination is a predictable consequence of pretraining, which resembles binary classification and cannot perfectly separate true from plausible-false statements, compounded by benchmark scoring that rewards a confident guess over an admission of ignorance. Their proposed remedy is not a new hallucination benchmark but re-scoring the existing ones to penalise confident errors more than abstentions, against what they call an epidemic of penalising uncertain responses.^[25] The cleanest head-to-head figure is in the GPT-5 system card: on SimpleQA without web access, gpt-5-main hallucinated on 47% of answers against GPT-4o's 52%.^[26] That is progress. It is not a number you delegate an unchecked decision to.