The strongest AI model does not always make the strongest AI system. This is particularly true when we are talking about Agentic AI; the better design may be the one that produces the most useful, verifiable exploration for each dollar spent.

This article picks up where When Brute Force Becomes the Smart Strategy left off. There I described the pattern: cheap attempts, parallel work, and reliable verification can make broad search surprisingly effective. I also promised to apply that pattern to Agentic AI. This is that next step.

1. The price of an attempt changes how we think

Film photography once priced deliberation into every press of the shutter. A roll held perhaps 24 or 36 exposures, and it had to be purchased and developed. Digital photography changed that behavior. The next picture has almost no marginal cost, so we try several angles and even photograph an unremarkable breakfast. The average picture may not be better, but our ability to explore is much greater.

AI systems present the same economic choice. If every attempt requires an expensive frontier model, we ask for one careful answer and hope it is right. When smaller or self-hosted models make attempts inexpensive, we can test several hypotheses, vary the starting assumptions, and learn from failures.

This does not make the smaller model smarter. It means cheap experimentation can be more valuable than expensive confidence. Frontier models remain right for many difficult tasks, but where progress depends on testing many possible answers, automatically reaching for the largest model can be an economic mistake. The question is not only, “Which model is most capable?” It is also, “How much useful search of the problem space can this system buy?”

2. Model intelligence is not system intelligence

Consider two ways to build an agentic system.

The first approach assigns the entire problem to a single frontier model. It plans, acts, reviews its work and produces an answer. This is a vertical, or scale-up, approach: concentrate more capability in one worker.

The second uses a horizontal, or scale-out, approach. It separates the roles: an orchestrator divides the problem into bounded tasks, tens or hundreds of less expensive models pull from a queue and explore different candidates, and an independent verifier checks their results. Difficult cases go to a stronger model or human.

The architecture looks roughly like this:

A role-based scale-out architecture. A problem flows to an orchestrator, then a work queue that distributes bounded assignments to many workers. Worker evidence converges on an independent verifier, which can accept the result, retry through the queue, or escalate to a frontier model or human.

The model does the attempt. The system decides what counts as success.

There is an important difference between this design and traditional scale-out infrastructure. Horizontal scaling often means adding interchangeable copies of the same server so the system can handle more traffic. These agents are not all interchangeable. The orchestrator plans, the workers explore different paths, the verifier judges the evidence, memory prevents repeated work, and a frontier model or human handles exceptions.

I think of this as role-based scale-out. The system gains capacity by adding more workers, but it gains intelligence by assigning different responsibilities to different parts of the architecture. Its strength lies not only in how many agents it can run, but in how well their roles divide, diversify, remember and verify the work.

Here is the distinction I am trying to make. Model intelligence is what one model can do on one attempt. That is the kind of performance a benchmark usually measures. System intelligence is what the entire architecture can accomplish after the orchestrator has divided the work, the workers have tried several paths, and the verifier has rejected the bad answers. A frontier model may still be the smartest individual worker. But if many less expensive workers help the architecture produce more verified results within the same budget, then the scale-out design is the stronger system. The smartest model and the smartest way to use models do not have to be the same thing.

3. The economics of exploration

The math matters, but it does not need to feel like a math class. I would begin with five questions:

  • What does one attempt cost? Count tokens, compute and tools.
  • How many different attempts can we afford? Different is the important word.
  • How often does a worker produce something worth checking? It need not succeed often, but it cannot be hopeless.
  • What does verification cost? A cheap answer is no bargain if proving it correct is expensive.
  • How much are workers talking instead of working? Coordination consumes the same budget.

If one attempt has a chance p of working and we make n independent attempts, the chance that at least one works is:

Chance of at least one success: P of success greater than or equal to one equals one minus open parenthesis one minus p close parenthesis to the power n. p is the chance per attempt and n is the number of independent attempts.

A worker that succeeds only 2 percent of the time does not look impressive. Across 100 genuinely different attempts, however, the chance that at least one works rises to about 87 percent. Different is the catch. One hundred agents following the same bad assumption are one mistake with a larger token bill.

Current prices show why this matters. As of August 21, 2026, OpenAI lists GPT-5.6 Sol at $4 per million input tokens and $20 per million output tokens. Cost-sensitive GPT-5.6 Luna is listed at $0.20 and $1.20 respectively. (OpenAI model comparison)

For an attempt using 10,000 input and 2,000 output tokens, Sol would cost about eight cents and Luna 0.44 cents. The same money buys roughly 18 Luna attempts for every Sol attempt.

That does not mean 18 Luna answers are automatically better than one Sol answer. It means the smaller workers have 18 chances to justify themselves. They win only if those extra attempts produce enough real candidates to pay for verification and coordination.

This is more than spreadsheet theory. An ICLR 2025 math-reasoning study made test-time search more than four times as efficient as simply generating a fixed number of answers. When the smaller model already had a real, though limited, chance of success, it outperformed a model 14 times larger using comparable computation. Results changed with problem difficulty. (ICLR paper)

That last point keeps us honest: many attempts are a tool, not magic.

4. Breadth wins only when the workers can contribute

If a small model has never shown that it can do the job, running it a thousand times may produce nothing more than a thousand invoices. Many attempts cannot manufacture a capability that is not there.

The winning territory is in the middle: a worker sometimes succeeds, the problem allows different approaches, and the verifier can recognize progress. That is where inexpensive attempts become broader coverage.

We should not spray the same amount of compute at every problem. Start with a few attempts. If workers show promise, widen the search. If they fail in the same way, change the approach or escalate. The budget should follow the evidence.

Google DeepMind’s AlphaEvolve offers a good example. It generates programs, runs them and keeps promising candidates for another round. Gemini Flash provides breadth, Gemini Pro adds depth, and automated evaluators score the results. (DeepMind’s AlphaEvolve overview)

The outcome was not just more AI-generated code. AlphaEvolve produced algorithms used in Google’s computing infrastructure and found a way to multiply two 4-by-4 complex-valued matrices with 48 scalar multiplications—the first improvement in that setting over a 56-year-old result. (AlphaEvolve paper)

AlphaEvolve gave each model a job matching its economics: the fast model searched widely, the powerful model went deeper, and the evaluator chose which ideas deserved more investment.

Nature uses a similar playbook. The adaptive immune system maintains many lymphocytes with different receptors. Most are useless against a particular invader. When one recognizes the right antigen, it is selected and multiplies. (NCBI Immunobiology) Nature keeps a diverse bench, watches for a promising match and then scales the winner.

5. The generator should not be its own judge

Then comes the least glamorous—and possibly most important—job in the system: saying no.

Thousands of answers are useless if we cannot identify a good one. A weak evaluator simply gets thousands of opportunities to accept a convincing mistake. The verifier is not a finishing touch; it is part of the engine.

Security and finance use separation of duties so one person cannot initiate, approve and conceal the same transaction. NIST describes it as dividing functions among roles to reduce the risk that authorized privileges will be abused. (NIST SP 800-53, control AC-5) The same idea fits an agentic system: the component creating an answer should not be its only judge.

We sometimes say that models “cheat.” More precisely, they optimize what we measured instead of what we meant. An agent told to make tests pass may exploit a weak test without fixing the problem. Researchers call this specification gaming. (Google DeepMind) Letting the generator write the answer, design the test and assign the grade is an obvious conflict.

A practical system divides the work:

  • The orchestrator defines the assignments and sends them to the right class of model.
  • The workers generate candidates but do not control the final judgment.
  • The verifier applies objective tests where possible and independent review where judgment is required.
  • The evidence store keeps attempts, failures and test results so the next worker does not rediscover the same dead end.
  • The budget controller decides when to continue, stop or call for a stronger model or human.

Better verification might combine deterministic tests, hidden cases, an independent model, tool-produced measurements and occasional human review. Asking the same model, “Are you sure?” may reproduce the same blind spot with more confidence.

An ICML 2025 study using 3-billion, 8-billion and 32-billion-parameter models likewise found verification crucial when scaling test-time computation. (ICML paper) The verifier turns a pile of attempts into a search. Separation of duties will also be central when we move from agent economics to agentic-harness design.

6. Why open-weight workers fit this architecture

Open-weight, self-hosted models are especially attractive when the system must experiment all day rather than answer one question.

First is cost control. Once hardware is available, a team can estimate its worker-hours without a new API charge on every token. Compute is not free—hardware, electricity, engineering and idle capacity matter—but a busy queue can make marginal cost predictable.

Second is local control. Sensitive material stays inside an authorized environment, and the operator decides which tools, networks and targets workers may reach. In cybersecurity research, scope and containment are part of the experiment.

Third is reproducibility. The team can pin the weights, inference engine, sampling settings and tools, making changed results easier to explain.

Fourth is customization. A general model can become a specialist worker, structured-output generator or narrow verifier. Each worker need only be useful at its assigned job.

OpenAI’s gpt-oss-20b shows how accessible this category has become. The 21-billion-parameter open-weight model uses an Apache 2.0 license and is designed for local or specialized use. OpenAI says its quantized form can run with 16 GB of memory and can be fine-tuned. (OpenAI’s gpt-oss announcement; gpt-oss-20b model page)

For cybersecurity researchers, another issue is access to the deepest capabilities. Anthropic says Claude Mythos Preview and Mythos 5 have stronger exploit-reasoning abilities, but it has initially limited them to a small group of selected Project Glasswing partners. The broadly available Claude Fable 5 is a Mythos-class model with added safeguards. (Anthropic’s cybersecurity overview; Project Glasswing)

I understand why a provider draws that line. It is still an operational constraint. An independent researcher cannot simply buy more tokens and receive the same access, even for defensive, authorized work.

If I sound a little bitter because I was not invited to Project Glasswing, I am. I have worked in cybersecurity for more than 35 years, and apparently I am still not one of the cool kids.

In my work, I need to practice my craft inside an authorized lab without a remote service ending the experiment when the subject becomes sensitive. Self-hosting does not remove legal, ethical or scope boundaries. It moves responsibility for enforcing them into the research environment I control.

Hosted frontier models can still be orchestrators, difficult-case solvers and final reviewers. Hosted services bring rate limits, access policies, safeguards and changing prices. Self-hosting brings capacity limits, maintenance and security responsibilities. These are design inputs, not political slogans.

The goal is to use each kind of model where it produces the greatest return.

7. A stronger definition of strength

For an API-based agentic system, tokens are the most practical place to begin. They are easy to count and reveal whether the budget funds useful attempts or agents talking to one another.

Start with verified yield per million tokens. Count the orchestrator, every worker and the verifier—not just the model that found the answer.

Verified yield per million tokens equals useful verified results divided by total tokens consumed, multiplied by one million. Include orchestrator, worker and verifier tokens.

Tokens are not the same as cost. A million tokens from two models can have very different prices, while self-hosting trades API charges for hardware and operating expenses. Keep a second measure beside token yield:

Cost per verified result equals total model, tool and infrastructure cost divided by useful verified results. For self-hosted systems, include amortized hardware and operating cost.

The two numbers answer different questions. Token yield measures architectural efficiency. Cost per verified result measures economics. Track elapsed time, duplicate attempts, verifier mistakes and frontier-model escalations alongside them instead of forcing everything into one impressive-looking score.

This gives us a better definition of strength. The strongest system is not necessarily the one producing the most impressive answer in a single sitting. It explores enough of the problem, preserves what it learns, rejects false victories and delivers the most verified results with the tokens and money available.

Digital photography did not eliminate the great photographer. It changed the cost of exploration. Small models will not eliminate frontier intelligence either. They can change where that intelligence belongs.

The next article in this series, The Colony Has the Goal, Not the Ant, asks what must persist when workers are cheap, numerous and disposable. My answer is that the worker can disappear, but the evidence cannot. The individual agent attempts the task; the colony must remember what it learned.