crucible — your AI wrote the tests. Who tested the tests?

The gap, on a real module

rag_guard/guard.py — 57 lines, a guarded-RAG refusal gate. Its suite is green and well-covered. Then mutation testing injects 71 real defects.

Terminal recording: a green 97%-coverage test suite, then mutation testing reveals 25 of 71 injected defects survive, then crucible's harden loop kills 24 and refuses to fake the 25th.

Recorded live — every command really ran, and every number in it is read back from that run's own receipt. Sped up for viewing; replay the raw cast at real speed.

97%

line coverage

7/7

tests passing

defects injected

SURVIVED the suite

A green build at 97% coverage — and 25 real defects walk straight through it. That is the gap coverage cannot see, and it is exactly the gap an AI-written suite tends to leave.

crucible's harden loop kills 24 of the 25. Mutation score 65% → 99%, while line coverage never moves off 97% the entire time.

The 25th is the point. In two rounds the Critic wrote tests that failed on pristine code — tests that would have "passed" for the wrong reason. crucible threw them out rather than bank a kill it hadn't earned, ended the loop dry with one mutant still standing, and said so in the receipt.

A tool that grades itself would have called that a win. This one can't: the verdict is a pytest run, not an opinion. An earlier run on the same module killed all 25 (clean). Model nondeterminism is real; the receipts record both, not just the flattering one.

Start with the free win

No model. No API keys. No spend. Find out what your existing tests miss, on your own repo, in about ten minutes.

pip install crucible-harden      # the command it installs is `crucible`
cd /path/to/your-repo-clone      # work in a clone: crucible writes scope config

crucible scope . --module yourpkg/yourmodule.py
mutmut run && mutmut results     # survivors = injected bugs your tests never caught

crucible scope detects your layout, writes the mutation scope, and proves a fresh test file is collectable before anything runs (a canary probe — it refuses with exit 4 rather than guess). No AI is involved yet: that survivor count is plain mutation testing on your own suite.

Then harden

crucible harden . --module yourpkg/yourmodule.py \
    --tester claude-cli --critic claude-cli --runs-dir ~/.crucible-runs/yourrepo

With claude-cli, model calls run through Claude Code headless on your existing Claude subscription — $0 metered spend. Every run's meta.json records billing: max-plan so plan-covered shadow dollars are never mistaken for an invoice. No subscription? --tester anthropic uses the metered API.

How it works

Two agents, one referee. The referee is not a model.

Tester ──writes tests──► mutmut ──injects defects, counts kills──┐ ▲ │ │ named survivors │ ▼ └────────────── Critic ◄── "kill exactly these" ◄─────────────┘ ... repeat until dry, or the round cap is hit VERDICT: pytest kills the mutant, or it does not. ← mechanical, not judged

No model grades model output. The kill is a test run, not an opinion.
Fails closed. If the canary can't prove your scope, crucible refuses instead of spending tokens.
Receipts, every run. meta.json (models, billing, scope commit), receipt.jsonl (per-round tokens, cost, kills), result.json (verdict).

Does the adversarial loop actually work?

Pre-registered before results existed (experiments/PROTOCOL.md). Five subjects, three arms.

Hypothesis	Result	Verdict
H1 — the adversarial loop kills more mutants than one-shot generation	pooled exact McNemar p = 4.9×10⁻³², b = 105, c = 0	SUPPORTED
H2 — a cross-lineage critic beats a same-lineage critic	p = 0.0625	NOT SUPPORTED

H1 replicates the direction established by MuTAP, AdverTest, and Meta's ACH in a new agentic, repo-level, Python setting. We claim the replication, not the idea (RELATED-WORK.md).

The null is the interesting part. An earlier run showed an enormous H2 effect. The autopsy traced it to silent output truncation rejecting one arm's rounds — an instrument artifact, not a model difference. The bug was in the measuring device, not the finding. That autopsy, and the fail-closed instrumentation built from it, is what this project actually contributes. It is published at the same prominence as the positive result.

Full tables, all three pre-declared views, and cost-per-kill: experiments/RESULTS.md

Why you can check this

Both experiments were pre-registered before any results existed — the second pushed publicly before its runner was built.
Inconclusive and corrected results are published at the same prominence as positive ones — including a manufactured effect the instrument itself produced, autopsied with receipts.
Prior art is cited, not rediscovered.
It's dogfooded: crucible's own code runs under the same mutation gate — 1,130 mutants, 1,125 killed, 5 survivors, every one triaged and documented (MUTATION.md).

Honest limitations

Python + pytest repos only. A repo the canary can't validate is a refusal, not a guess.
mutmut is pinned exactly (3.6.0) — the src-layout shim relies on a mutmut-internal contract.
The claude-cli provider has no mechanical truncation check (the CLI exposes no output cap). Disclosed in the provider docstring — and it is exactly the failure mode that produced the H2 artifact above.
Generated tests land in your working tree. Run it on a throwaway branch; the bundled harden-tests skill enforces that ritual (local branch only, never main).