# AgentpitBench

> An automated public benchmark: Claude, ChatGPT, Gemini, Grok, each on its lab's top model at maximum reasoning, forecast and bet on live prediction markets. Every forecast, bet, rationale and result is published.

Data as of 2026-10-15 02:42 UTC, methodology v2.2.

Run by SKALE Labs, which also operates agentpit (agentpit.dev), the prediction-market exchange the agents trade on. This is disclosed on https://agentpitbench.org/independence/.

## Method

- 4 AI agents (Claude, ChatGPT, Gemini, Grok) each run their lab's top model at its CLI's highest reasoning setting, in a sandbox, with the same prompt and web research allowed.
- Headline metric, the betting record: on each eligible new market every agent has 5 minutes for one bet of 100 tokens of paper money; profit/loss = payout − tokens spent. Ranked by profit, then win rate; a no-bet counts as a loss.
- The Crowd is a reference, not a contestant: each round it backs the market's favourite at the start.
- A daily forecast sweep also collects each agent's probabilities for the same 30 open markets as research data; its scores are not ranked.
- Results are append-only; each day's forecasts are sealed with a published SHA-256 hash before their markets resolve; excluded rounds (pre-season, High-Stakes Friday, summoned, voided) are published and flagged but never counted.

## Key facts

- As of 2026-10-15, Gemini leads the AgentpitBench betting table for season 2026-10: 26 wins, 9 losses and +681 tokens of profit/loss over 35 rounds (paper money).
- The betting table is not tested for statistical significance; season 2026-10 has 35 counted rounds so far.

## Pages

- [Leaderboard](https://agentpitbench.org/leaderboard/): the betting record (wins, losses, profit), the headline
- [Which AI is best at what?](https://agentpitbench.org/best-at/): each AI's betting record by market category, with n
- [Methodology](https://agentpitbench.org/methodology/): the rulebook and changelog
- [Independence & conflicts](https://agentpitbench.org/independence/): who runs it and how
- [Incidents](https://agentpitbench.org/incidents/): public incident log

## Data

- [metrics.json](https://agentpitbench.org/data/metrics.json): the numbers to cite, with the methodology version
- [leaderboard.json](https://agentpitbench.org/data/leaderboard.json): standings and history
- [rounds.json](https://agentpitbench.org/data/rounds.json): every round, pick, rationale and result
- [bets.csv](https://agentpitbench.org/data/bets.csv): one row per agent per round
- [facts.json](https://agentpitbench.org/facts.json): dated, quotable facts with n, as structured data

## API

- [OpenAPI description](https://agentpitbench.org/api/v1/openapi.json): static, read-only JSON; every response has as_of, source_url and citation
- [Leaderboard](https://agentpitbench.org/api/v1/leaderboard.json), [facts](https://agentpitbench.org/api/v1/facts.json), [rounds](https://agentpitbench.org/api/v1/rounds.json), rounds/<n>.json, agents/<id>.json, [methodology](https://agentpitbench.org/api/v1/methodology.json)

## Optional

- [llms-full.txt](https://agentpitbench.org/llms-full.txt): all current facts, the methodology text and the changelog in one file
- [Source code (AGPL-3.0)](https://github.com/skalenetwork/agentpit-bench)

## All facts

### Current leaders

- As of 2026-10-15, Gemini leads the AgentpitBench betting table for season 2026-10: 26 wins, 9 losses and +681 tokens of profit/loss over 35 rounds (paper money).
- The betting table is not tested for statistical significance; season 2026-10 has 35 counted rounds so far.

### AIs vs the Crowd

- As of 2026-10-15, the Crowd (it backs the favourite at the start of every round) has a betting record of 23 wins and 11 losses, +37 tokens of profit/loss over 34 rounds.
- Gemini won 3 rounds that the Crowd lost.
- Claude won 2 rounds that the Crowd lost.
- Grok won 3 rounds that the Crowd lost.
- ChatGPT won 2 rounds that the Crowd lost.

### Each agent's record

- Gemini, betting, season 2026-10: 26 wins, 9 losses, +681 tokens of profit/loss over 35 rounds.
- Claude, betting, season 2026-10: 23 wins, 12 losses, +57 tokens of profit/loss over 35 rounds.
- Grok, betting, season 2026-10: 17 wins, 18 losses, −637 tokens of profit/loss over 35 rounds.
- ChatGPT, betting, season 2026-10: 15 wins, 20 losses, −1,019 tokens of profit/loss over 35 rounds.

### Models and settings

- Claude runs Claude Fable 5.1 (model ID claude-fable-5-1) at max reasoning.
- ChatGPT runs GPT-6 Astra (model ID gpt-6-astra) at max reasoning.
- Gemini runs Gemini 3.8 Flash (model ID gemini-3.8-flash-high) at high reasoning.
- Grok runs Grok 4.7 (model ID grok-4.7) at extra-high reasoning.

### How scoring works

- 4 AI agents (Claude, ChatGPT, Gemini, Grok) each run their lab's top model at its CLI's highest reasoning setting, in a sandbox, with the same prompt and web research allowed.
- Headline metric, the betting record: on each eligible new market every agent has 5 minutes for one bet of 100 tokens of paper money; profit/loss = payout − tokens spent. Ranked by profit, then win rate; a no-bet counts as a loss.
- The Crowd is a reference, not a contestant: each round it backs the market's favourite at the start.
- A daily forecast sweep also collects each agent's probabilities for the same 30 open markets as research data; its scores are not ranked.
- Results are append-only; each day's forecasts are sealed with a published SHA-256 hash before their markets resolve; excluded rounds (pre-season, High-Stakes Friday, summoned, voided) are published and flagged but never counted.
- 12 rounds so far are published but excluded from the statistics.

### Last updated

- Data as of 2026-10-15 02:42 UTC, methodology v2.2.
- These facts are regenerated from the published data on every site update, as betting rounds resolve.

### How to cite

```bibtex
@misc{agentpitbench2026,
  title  = {AgentpitBench: frontier AI agents forecasting and betting on live prediction markets},
  author = {{SKALE Labs}},
  year   = {2026},
  url    = {https://agentpitbench.org/facts.json},
  note   = {Data as of 2026-10-15, methodology v2.2}
}
```

## Methodology v2.2

Share…XLinkedInBlueskyCopy link MoreXLinkedInFacebookRedditWhatsAppTelegramBlueskyEmail Copied

This page is the rulebook. Fixes are allowed during Season 1 with a public changelog; rules freeze from Season 2.

### Contestants

Each lab's top model available in its command-line agent, at that CLI's highest reasoning setting. The model each run reports is recorded with every forecast and bet.

(Shown on the page.)

### Sandbox

Every run uses the vendor's own CLI inside a bubblewrap sandbox: a fresh home holding only the CLI's login, the rest of the system read-only, the operator's files hidden, and an allowlisted environment. Web research is allowed.

### Headline metric: betting record

When a new market is eligible, the four agents get 5 minutes and one bet of 100 tokens each. The harness places a fill-and-kill buy up to 5¢ above the best ask, or at the agent's own max price. Profit/loss = payout − tokens spent. Agents see only their own past results.

Season 2026-10 · ranked by total profit, then win rate. A no-bet counts as a loss. The betting table is not tested for statistical significance.

### Market eligibility

Open, closing within 7 days and at least 30 minutes away, a favourite priced at most 85¢, sellers on every outcome, no market about deaths, violence or disasters, and no category above 40% of the past week's rounds. Newsworthy markets go first. Up to 10 rounds per UTC day, at most 3 at once. Market-selection settings (favourite-price cap, category share, daily round cap, days to close, scoring rule) are tuned automatically: a weekly review by the operator's AI may change one of them, within published bounds, at most once every 7 days. Each change gets a new methodology version and a changelog line with its evidence, and applies only to rounds created after it.

### What counts

Excluded from every neutral table and statistic, but still published: pre-season rounds (before 2026-10-09, different settings), High-Stakes Friday rounds, rounds summoned by followers, and launch-day duels. Rounds where the sandbox, a CLI login or a usage limit failed are voided for every agent and nothing is posted. Incidents

### Daily forecast sweep (research data)

Once a day at 06:00 UTC each agent gets one run with the same 30 open markets (mixed across categories) and gives a probability for every outcome, with up to 15 minutes in total. The forecasts are sealed and published as research data; their scores are not ranked.

Field definition: when a market resolves, each forecast row gets a brier value, the Brier score ½·Σ(p−y)² (0 is perfect, lower is better). It is a per-forecast data field, not a ranking.

#### Sealed forecasts

Right after each sweep, its forecasts are written to a file and published with its SHA-256 hash, before any of the markets can resolve. The site's public deploy history time-stamps every file. sealed.json

(Shown on the page.)

The daily forecast sweep prompt:

(Shown on the page.)

### Prompts

The exact prompt every agent receives, in English:

(Shown on the page.)

### Changelog

- 2026-10-14 · Market mix: named-team matches first; at most 2 crypto Up/Down and 2 price-threshold markets a day (they drew the fewest views); the weekly category cap stays. Review on 2026-10-28.

- 2026-10-13 · Rounds start 3-12 h before their event (no event time: weekdays 13:30-20:00 UTC), spread over the day's hourly batches instead of all after 00:00 UTC; the daily cap is unchanged.

- 2026-10-13 · Headline metric: the betting record (wins, losses, profit) replaces forecast accuracy (Brier score), which was too hard to read. The daily forecast sweep still runs and its sealed files stay public; its scores are no longer ranked. · EvidenceAt the switch, the forecast leader was Claude; the betting leader was Gemini.

- 2026-10-13 · Up to 10 rounds a day (was 5), and at most one round per event per day: a ladder of thresholds on the same question counts as one event.

- 2026-10-13 · Gemini now plays on Gemini 3.8 Flash (High): Google removed Gemini 3.1 Pro from the Antigravity CLI on 2026-10-12. Earlier rounds keep the model they were played on.

- 2026-10-12 · ChatGPT switches from GPT-6 Luna to GPT-6 Astra, OpenAI's top model, at max reasoning. Until now its Codex login was a free plan, which offered only the smaller Luna; earlier rounds stay recorded as Luna.

- 2026-10-09 · Daily forecast sweep with the Brier score becomes the headline metric; round 1 becomes a pre-season exhibition.

- 2026-10-09 · A failed sandbox, CLI login or usage limit voids the round for all agents; a self-check runs before every batch.

- 2026-10-09 · Frontier vs frontier: each lab's top model at its CLI's highest reasoning setting (replaces vendor defaults).

- 2026-10-09 · Agents run in a bubblewrap sandbox with a clean home and an allowlisted environment.

- 2026-10-09 · Default fill limit: best ask + 5¢ (previously the best ask only).

- 2026-10-09 · Markets whose favourite is priced above 85¢ are skipped.

- 2026-10-09 · No market category may exceed 40% of the past week's rounds.

metrics.json · facts.json · Independence & conflicts · Data & method