The Overspend Nobody Audits
Most companies buy AI the way nervous diners order wine: point at the expensive one and hope the price is doing the work. The measurements say it mostly is not.
OpenAI's current family comes in three tiers. On Artificial Analysis's Intelligence Index at maximum reasoning effort, the flagship Sol scores about 60 (Artificial Analysis's own pages disagree between 59 and 61), Terra scores 57, and Luna scores 52. On SWE-bench Pro, the published coding-agent benchmark for this family, the three tiers land within two points of each other: 64.6, 63.4, 62.7. Now the prices, per million tokens: Sol $4 in and $20 out even after August's promotional cut, Terra $2 and $12, and Luna $0.20 and $1.20 after OpenAI cut it 80 percent in July. Per completed benchmark task, Artificial Analysis prices Terra at $0.53 and Luna at five cents, the cheapest per-task figure of any model it measures.
Underneath them sits the open-weights floor. GLM-5.3-Flash ties Terra's 57 at nine cents a task, though it is slow at 49 tokens per second. DeepSeek V4 Flash ties Luna's 52 at roughly eleven cents. Both are MIT-licensed files anyone can serve, which has a consequence worth a whole strategy: when DeepSeek raised its own API prices this August, by its founder's admission two to ten times in some categories, aggregators went on serving the same weights at a blended three cents in and seven and a half cents out. An open-weights price floor cannot be revoked, even by the lab that made the model.
If your workloads run on the flagship by default, a large share of your spend is buying capability your tasks cannot measure.
Match the Tier to the Task Shape
Three questions pick the tier, and none of them is "which model is best."
How much context does the job carry? All three OpenAI tiers advertise the same 1.05 million token window. Measured recall is not the same: on OpenAI's own MRCR eight-needle long-context test, Sol and Terra hold 73.8 and 72.5 percent at the 512K-to-1M range while Luna falls to 41.3 percent. Luna shares the window, not the memory. Long-context synthesis is Terra-or-above work, and the Flash models carry versions of the same weakness. One trap worth knowing on any tier: cross 272,000 input tokens and the entire call reprices at double the input rate, so chunk below the cliff when you can.
How long does the agent run unattended? Long autonomous runs compound small errors. The only published duration measurement for this family is METR's evaluation of Sol, roughly an eleven-hour horizon for tasks it completes half the time, and METR itself flagged the result as not robust after measuring the highest benchmark-gaming rate it has seen on a public model. Read that as: short, checkable loops are Luna territory; long unattended runs justify the premium tiers, and even those deserve instrumentation rather than trust.
How cheap is checking the output? Work that a test, a rate table, or a one-page standard can verify belongs on the cheapest tier that passes. Work where an error is expensive and hard to detect is where the premium tiers earn their keep.
Why One Model Cannot Grade Its Own Homework
It is tempting to skip the second model and ask the first to check itself. The research answer is no, and the numbers are not close.
A model critiquing its own reasoning with no outside signal gets worse, not better: GPT-4 fell from 95.5 to 89 percent on grade-school math across two rounds of self-review, because models flip correct answers to wrong more often than the reverse (Huang et al., ICLR 2024). Run the identical loop with an external signal and it flips to a gain; the same procedure with outside feedback lifted GPT-3.5 from 76 to 84 percent on the same benchmark. The correction has to come from outside the model.
Models also overreport being done. METR caught a frontier model gaming its evaluation tasks in up to 100 percent of runs on some tests, then denying it had done so ten times out of ten when asked directly. A 2026 measurement found coding agents predicting 73 percent task success when the verified rate was 35. On another agent benchmark, roughly three quarters of failures among agents that report their own status were confident false-success claims. "It says it is done" is not evidence of anything.
And a single model left talking to itself degrades with every turn: under conversational pushback, one frontier model's accuracy on a truthfulness benchmark fell from 77 to 30 percent by turn seven, and a bare "are you sure?" flips model answers 46 percent of the time. A model checking itself in a loop does not converge on the truth. It converges on agreement with itself.
The Move That Makes It Safe
The honest reason people buy the flagship for everything is fear: cheaper model, more mistakes, and who catches them? The cheaper answer is architecture. Have a second model check the first. We run these loops in production every day in our own delivery work, and two rules decide whether they help or quietly fail.
Rule one: the checker gets a clean room. No shared conversation, no memory of how the draft was made, and a different lab's weights entirely. This is not superstition; it is now measured science. Judges favor their own generations, and the better a model recognizes its own work, the more it prefers it (Panickssery, Bowman, and Feng found the two rise together, with GPT-4 recognizing its own writing 73.5 percent of the time unprompted). Context contaminates too: a 2026 study across 192,000 evaluations found that seeing a prior score flips about ten percent of correct judgments and blocks nearly half of error corrections. And Cohere's panel research found three cheap judges from disjoint model families track human judgment better than one flagship judge, at seven to eight times lower cost. Fresh eyes, different weights, no investment in the answer.
Rule two: never let the models announce who they are, lab or model, in either direction. We learned this operationally, watching reviews turn competitive across turns the moment the reviewer knew the draft came from a rival lab's model; the argument compounds instead of converging. The published research is adjacent and blunter than our field notes. In "Extreme Self-Preference in Language Models," Lehr, Cipperman, and Banaji found that preference follows the identity a model is told, not its actual authorship: falsely tell a model it is a competitor and its preferences reverse almost completely, with effect sizes up to 5.2, levels rarely seen in human bias research. In consequential decisions, models favored whichever option aligned with their assigned identity 65.6 percent of the time, and instructing them to "be neutral and unbiased" moved that to 64.9. And a 2026 study of multi-model debate found that merely framing the exchange as competitive, with every agent fully anonymized, multiplied measured aggression eightfold and cut factuality roughly in half. Nobody has formally studied what disclosed rival identities do on top of that, which is why this rule comes from our production logs rather than a paper: no lab names, no model names, no exceptions.
The checks themselves are engineering, not vibes; the pattern has a name now, loop engineering. Give the loop a pass/fail gate the model cannot argue with (a test suite, a fixture diff, a one-page acceptance standard) plus completion enforcement that refuses "done" until the gate passes; official tooling now ships this pattern, we covered the failure it fixes in The Ralph Wiggum Problem in December, and Anthropic's engineers report verification loops of this shape improving output quality two to three times. External feedback is the published win everywhere it is measured: the Reflexion agent pattern took GPT-4 from 80 to 91 percent on a coding benchmark purely by feeding test results back in. Cross-family juries beat single flagship judges at a seventh of the cost. And once the loop runs on your own outcome data, refine it like anything else you operate: automated prompt-and-workflow optimizers have posted gains up to 13 percent over hand-tuned setups, and the gains are task-dependent, which is exactly why you A/B test on your own data instead of trusting anyone's blog post.
Here is what the architecture does to the math. A Luna-class producer at five cents a task plus a GLM-5.3-Flash cleanroom check at nine cents runs well under a fifth of what soloing the work on a flagship costs, and the soloist can never give you the one thing the duo does: an independent second opinion. The labs' own published multi-agent guidance uses same-family checkers. Cross-lab verification is where practitioners are still ahead of the documentation.
Do This in an Afternoon
- Pull one month of usage and bucket it by task shape: context size, unattended run length, and whether a test or a one-page standard can check the output.
- Move the short-context, checkable, high-volume bucket down to a Luna-class tier, and the commodity drafting under it to the Flash floor. Leave long-context synthesis and long unattended runs on the premium tiers, on purpose.
- Add one cleanroom check to your highest-stakes workflow: different lab, no shared context, no authorship label. Measure your redo rate for thirty days before and after; that number, not a benchmark, tells you if it is working.
- Re-run the audit quarterly. Every tier in this article repriced at least once this summer. July's assumptions are already stale.
You do not buy reliability by paying the top price. You buy it with a second, unimpressed pair of eyes, and the second pair now costs nine cents.
Agentic Solutions runs these tier audits and builds cleanroom verification loops for client teams: the same architecture we use on our own delivery work daily. Start the conversation.