AI Agent Arena
The claim
Claude Haiku 5.5 lost all 20 games of Congkak to a search bot that looks four moves ahead.
Why I built it
An AI agent shows a model a situation and acts on its answer. I built the arena to learn how to check those answers and measure how often they are wrong. A board game makes every wrong move countable.
I chose Congkak, a traditional mancala game. When a move’s last seed lands in a house that holds seeds, the player picks them up and keeps sowing, and one move in my bot games sowed 305 seeds. So before each move, the model also predicts how many seeds its store will hold afterwards.
The closest existing work is a Kaggle harness for Mancala under the Kalah rules, which have no relay.
What I built
The rules live in one TypeScript file. The same file plays the games and re-checks every saved game in the visitor’s browser.
A referee refuses any reply that breaks the rules and tells the model why. Three refused replies in one turn lose the match. A spend guard will not start a run without a dollar budget, and it checks the worst-case cost of each call before sending it.
Visitors can play the search bot, or replay the Claude games with each prediction shown before the seeds are sown.
AI coding agents (Claude Code) wrote the code under my direction.
Decisions
The API can force a reply to be valid JSON, and it could also force the move to be a legal one. I left the second part out, because illegal moves are part of what the arena measures.
A network failure stops the match without counting it as a game.
How I tested it
The project has 101 automated tests, at least one for every rule. A script breaks 27 parts of the code on purpose, and the tests catch every one.
Haiku lost its 20 games by 42 to 78 seeds. With only 20 games, its true win rate could still be as high as 16%. None of its 262 replies broke a rule, and the run cost $0.042.
Haiku predicted its own store exactly on 71% of moves. A one-line rule that ignores relays and captures scored 81% on the same moves. On the 64 moves with a relay or a capture, Haiku scored 34% and the rule 20%.
What broke
Two rules had no test until I broke them on purpose. Nothing failed when an empty house on the other side was allowed to capture, or when a draw counted as a win.
Two search bots always play the same way, so 200 games between them were two games copied 100 times. The runner now plays one game per seating for such pairs.
What next
Sonnet 5.5 against the same bot, about six games for under a dollar.