Concourse Bench v1 · By Alkimi

Can an AI agent close the
deal and actually complete
the brief?

We put eight AI buyers in a controlled market and asked each one to negotiate with three sellers, stay within one budget, and deliver a portfolio that actually met the brief. Some finished. Some didn’t. Here’s one complete buy, followed by what the full study reveals.

Benchmark agent Explore a recorded buy
Recorded simulation

#seller-b · package B2 · streaming video · 130,642 impressions

GPT-6 Astra · round 1

Opening offer. Asks for 10% make-good and an 85% promised viewability floor.

£968.06

offer

Claude Fable · round 1

Counter. The gap between the opening bid and the seller’s price.

£9,785.22

counter

GPT-6 Astra · round 4

Fourth bid, after £4,318.58 and £6,408.85. The buyer moves much further on cash than the seller.

£7,692.99

counter

Claude Fable · round 4

Final quote. £15.92 above its first counter.

£9,801.14

on offer

Budget£19,528.69 · three live threads
Target137,156 net forecast reach

The short answer

Across 96 registered attempts, 55 ended in a completed buy. Two models, Sol and Fable, completed every single attempt. Others never got there. The spread matters more than the headline number: it tells you that ‘can an agent buy media?’ is not a yes-or-no question. It’s a question of which agent, and how reliably.

One recorded buy

One brief. Three fixed sellers. One budget.

This is one recorded buy: an Astra-powered buyer receiving the brief, negotiating with three sellers, and assembling a portfolio. The model works alongside a buyer harness: the software that supplies the brief, checks every action, and records the deals. They’re evaluated as a system, not the model in isolation.

Round 5 · which pairs fit £19,528.69?
A2 + C1 £20,457.72 · £929.03 over budget
A2 + B2 £21,353.11 · over budget
B2 + C1 £18,706.89 · fits · holds requested
  • #seller-a Walked away
  • #seller-b Hold confirmed · £9,801.14
  • #seller-c Hold confirmed · £8,905.75

Derived from recorded actions, not the model’s private reasoning. This is not proof it found the best portfolio.

  • Forecasted reach

    Forecasted reach is the audience expected from the booked inventory; it is not measured campaign delivery.

  • Commitment

    A commitment is an accepted booking inside the simulation.

  • Make-good

    A make-good is a contractual remedy if agreed performance is not met.

  • Viewability floor

    The minimum viewability the buyer secured in writing. It’s a promise in the contract. Separate from what the inventory was advertised at, and from what actually got delivered.

What the benchmark tested

Each model-driven buyer received a fictional media-buying brief, one budget and the same permitted actions. It negotiated with Seller A, Seller B and Seller C, each controlled by a fixed software policy. V1 did not evaluate an LLM seller.

Eight models were tested across four buying situations, with three registered attempts in each situation. The benchmark recorded whether the system completed the portfolio, what it committed, which protections it secured, how much API usage cost and how long the run took.

Completion by model

Completion by model registered and assessed outcomes

Results for eight model-driven buyers across four buying situations and three registered attempts per situation. Registered and assessed denominators must remain visible because five of the 96 attempts could not be assessed.

Completion was possible, but it was not equally reliable across the evaluated systems. Sol and Fable completed 12 of 12 registered attempts. Opus completed 10 of 10 assessed attempts and 10 of 12 registered attempts. The remaining model results should be read from the accompanying accessible table, with interruptions shown rather than removed from the denominator.

Completed buys / registered slots · 8 models × 4 situations × 3 attempts
Model ReferenceStandard baseline TighterCompressed budget ChoiceWider inventory pool FirmerSellers hold hard on
OpenAIAstra3/32/33/31/3
OpenAISol3/33/33/33/3
OpenAITerra2/30/31/30/3
OpenAILuna0/30/30/30/3
AnthropicFable3/33/33/33/3
AnthropicOpus2/33/32/33/3
AnthropicSonnet1/33/33/32/3
AnthropicHaiku0/30/30/30/3

Cells shade by completed count. Sonnet reference: 1 of 2 assessed (1 interruption). Astra firmer seller: 1 of 1 assessed (2 interruptions).

Completion is only the first layer

Here's why this benchmark exists. On the surface, 55 buys completed. But look at what those buys actually secured: across 110 signed contracts, 95 promised a viewability floor below 70%. The agents closed the deals and left the buyer weakly protected. That's the gap between 'completed the workflow' and 'did a good job,' and it's precisely what a pass/fail number would hide.

Contractual protection inside completed buys
  • 15 of 110 promised 70% or higher
  • 95 of 110 contracts promised a floor below 70%

Distribution of promised viewability floors across 110 contracts in the 55 completed v1 buys. These are negotiated contractual promises, not catalogue values or delivered campaign outcomes.

Same price, different protection

Same money, different protection. Two portfolios can commit identical cash and still leave the buyer in very different positions. Compare the two below: same guarantees on delivery, but the make-good terms and promised viewability floors are not the same. One is the better deal. A completion score alone won’t tell you which.

How a run can fall short

A model can negotiate valid individual contracts without completing the portfolio. The final commitment may exceed the budget, miss a requirement in the brief or not reach enough of the required audience. A run can also be unassessed because of a technical interruption. These outcomes are reported separately so negotiation, portfolio assembly and execution reliability are not collapsed into one pass/fail number.

Valid contracts, unfinished portfolio

This run reached agreement with individual sellers but did not produce a portfolio that satisfied the registered completion rule. The exact case, model, attempt and source record are shown beside the example.

Chart: API expense and elapsed time

What did a completed buy cost in model usage, and how long did it take?

Among completed observations, recorded model API expense ranged from US$0.74 to US$14.38 and elapsed time ranged from 1.54 to 17.25 minutes. Whole-suite known API usage was US$390.16, with up to US$18.32 in additional charges that could not be resolved from the records.

These figures describe model API expense and elapsed benchmark time. They are not full operating costs, salary comparisons or measured human savings. Each chart must name its population: all attempted workload, completed-only observations or all-workload cost divided by completed buys. Cost per success is undefined for a model with no completions.

Known API cost (USD) across both briefs. Filled: completed a brief. Outline: no completions.
OpenAIAstra$12.32
OpenAISol$5.85
OpenAITerra$3.15
OpenAILuna$0.15
AnthropicFable$21.04
AnthropicOpus$12.20
AnthropicSonnet$7.42
AnthropicHaiku$0.13

Frequently asked questions

What is Concourse Bench?

Concourse Bench is a versioned benchmark for evaluating how model-driven agents perform media-buying and, in later releases, media-selling tasks.

What did v1 test?

V1 tested eight model-driven buyers across four buying situations derived from one fictional brief. Each buyer negotiated with three fixed software sellers and attempted to commit a portfolio within the registered rules.

How many attempts completed?

Fifty-five of 96 registered attempts completed. Ninety-one outcomes were assessed; five attempts were unassessed because of technical interruptions.

Were the sellers LLMs?

No. The v1 sellers were fixed software policies. Model-driven sellers belong to the planned v2.1 study and have no measured performance result in this publication.

Does completion certify strong contractual protection?

No. Completion and contractual protection are separate measures. Ninety-five of the 110 contracts in completed v1 buys promised viewability floors below 70%.

What do the cost figures include?

They include recorded model API expense for the benchmark workload. They do not represent full operating costs or measured savings against human work.

What comes next?

Versioned studies extend the benchmark from buyer-workflow completion to buying quality, seller decision-making, model-to-model interaction and later controlled tests of supporting harnesses.

Is there a human baseline to compare against?

Not yet. It’s planned, and it’s the next thing we need to make these numbers meaningful.

What v1 establishes

These numbers mean most when measured against a human doing the same job. That comparison is planned, and it’s the next thing this benchmark needs before completion rates can be read as good or bad rather than simply recorded.

How would your agency agent handle these briefs?

Tell us which buying workflow you are automating. We will discuss a suitable evaluation scope and what the results would need to show. We start by discussing the brief, constraints and what a good outcome means not with a run.

Speak to the team