How Well AI Writes Kavach

75 small programming tasks, in the style of HumanEval and MBPP, are each given to a Claude model under four conditions. The model's answer passes when its hidden tests print exactly what they should. How everything is done is in method.md.

How many passed

Measured on 2026-11-14 at commit 22a20eb, all 75 tasks, one run of each cell (1,200 answers, $24.66 in all). Every answer is in results/2026-11-14-kavach.json; summarize.py makes these tables from it.

What the run shows

• Without help, models write Rust. Zero-shot, 4–9% pass. Of the 280 failures, 168 stop at Rust's let mut (a Kavach let can already change), and 40 more at a semicolon. At 22a20eb the message for let mut was a plain "expected = and a value"; it now says to write let. The models know the language's shape, not its words.
• The guide does most of the work. With llms.txt as the system prompt (about 1,500 tokens), 53–87% pass, and syntax errors nearly vanish. What remains is mostly type errors: writing a union's variant without its type (decision 0070), joining lists with +, leaving out an argument, a method that isn't there.
• The guide itself was out of date. At 22a20eb, llms.txt still showed variants without their type (status: Status = Pending, which decision 0070 refuses) and Int.parse(text), which is gone. Those two account for 23 of the 81 docs failures. Both are fixed, and its test now compiles every example and refuses such habits.
• The compiler's messages finish the job. Sent back the errors, every model reaches 97–100%, the strongest in 1.2 tries on average. With Kavach's own tools (kavach mcp), all four pass every task, almost always on the first graded try.
• Tools cost more than they help here. repair reaches nearly the same score as tools for less money and time: Sonnet $0.59 against $1.32, 8 minutes against 35.

The run was stopped once: a model's code looped forever inside kavach mcp, which has no time limit of its own. The runner then got its 60-second limit and --resume, and finished the rest.

Since the run, let mut, where most zero-shot failures stopped, gets a message that says what to write instead:

fn main() {
    let mut count = 0
    count += 1
    print(count)
}

error: Kavach has no `let mut`: a `let` can already change, so write `let count`

Tries, cost and time

The average number of graded tries a passing task took.

Dollars and minutes for all 75 tasks.

What went wrong

What the last failed try of each failed task went wrong on: syntax, type (and names), capability, panic, timeout, or logic (it ran, but printed something else).

Each cell was run once, so a small difference between two models may not hold in another run. The tasks, prompts and grading are in method.md, linked below.

Run it yourself

Anyone can run this benchmark, with their own API key. It needs Python 3.10 or later and the kavach command.

--estimate prints the most a run could cost, at the rates in prices.json (read from Anthropic's pricing page; update them if they changed). The whole table above costs at most about $90.

Then, with your key in ANTHROPIC_API_KEY:

• --models: any of haiku, sonnet, opus, fable.
• --conditions: any of zero, docs, repair, tools.
• --tasks: task ids from tasks/tasks.json; all of them when left out.
• --budget: the run stops before spending more than this many dollars (default 5).
• --resume: keeps the answers already in the results file (--out) and runs only the others, after a run was stopped.
• KAVACH: the path of the kavach command, when it isn't on PATH.

Each run writes every answer, its code, tokens, cost, time and failures to results/<date>-<language>.json.

The guide the docs condition gives the model is the site's own llms.txt, and the tools are kavach mcp's.