Benchmarks & evidence#

This page shows every measurement behind the numbers on this site: what was run, on what data, against which LLM, and what each column means. It also lists where shad0w loses. Every figure comes from numbers.json, which the benchmark kit generates from the result files named under each table.

Which version was measured. The result files were produced with shad0w 0.3.0 on 8 October 2026. Before the 0.3.2 release the latency runs, the certificate simulation and the BANKING77 and CLINC150 comparisons were repeated with 0.3.2 on the same machine; they matched within noise, and JavaScript came out a little faster than the figure shown here.

Hardware: Apple M5 Pro (15 cores, 24 GB), macOS, unless a section says otherwise.

How to read these numbers#

A few terms used in every table:

• The table: shad0w's small learned model, trained on the LLM's logged answers.
• α (alpha): the most disagreement with the LLM you allow among the answers the table gives. The default, 5%, is 1 in 20.
• δ (delta): the chance that a certificate is wrong because of an unlucky calibration sample. The default δ = 0.1 means 90% confidence.
• Answered by the table (or "served"): the share of fresh traffic, never seen in training or calibration, that the table answered without the LLM.
• Realised disagreement: on that fresh traffic, how often the table's answer differed from the LLM's. It may exceed α in up to δ of certifications (why).
• Accuracy vs gold: agreement with the dataset's human labels. It is reported separately because agreement with the LLM is not accuracy: the table copies the LLM, mistakes included.
• Cascade: the table plus the LLM for everything the table defers. Cascade accuracy is what your users get.

Headline: the same certifier for every method#

Setup. BANKING77 (77 intents). The LLM, gpt-4.1-mini (OpenAI), labelled 10,000 messages; those answers are the log. Every method was trained on the same log, certified with the same procedure at α = 5% and δ = 0.1, and tested on the same 3,080 fresh messages, over 3 seeds.

Source: results/b6_compare_openai_banking77.json (learned methods) and results/b7_systemone.json (local decision models). Hosted decision-model rows use vendor-stated latency and prices.

What each column means:

• answered without the LLM @ α = 5%: share of fresh traffic the method answered itself, with its certified threshold. Decision models answer everything, because they replace your LLM.
• disagreement with the LLM on those: realised disagreement on the answers it gave. For decision models: how often they differ from gpt-4.1-mini (OpenAI) over all messages.
• accuracy vs gold (cascade): method plus LLM fallback, against human labels. The LLM alone scores the "keep calling the LLM" row.
• p50 per decision: median time of the method itself, not counting LLM calls for deferred messages.
• cost per 1k decisions: LLM spend per 1,000 messages for the share still sent to the LLM ($0.520 per 1,000 when every message goes to it). Local models cost only your hardware.
• needs and guarantee: what you must run, and what bound you get.

On 3 intent and topic tasks (BANKING77, CLINC150, AG News) the table answered 61–86% of fresh traffic at α = 5%, with realised disagreement of 3.0–6.0%.

The α sweep#

Setup. The same BANKING77 run, at four values of α. "n/a" means nothing was served, so there is nothing to measure.

BANKING77. Lower α = safer, fewer calls saved. Source: results/evidence.json cmp.banking77.openai.shad0w.

A lower α buys safety with calls: at the strictest setting the table answers nothing.

It grows with traffic#

Setup. The same BANKING77 run, trained on the first N logged answers, at α = 5%.

BANKING77. Source: results/b6_compare_openai_banking77.json.

With 77 options, the table certified nothing until it had seen a few thousand answers. Simpler questions start earlier: on AG News (4 options) it began serving after about 1,000.

Latency per runtime#

Setup. The 77-intent BANKING77 table, single thread, median (p50) of many calls. Decision-model and LLM rows are measured on the same machine for comparison.

Source: results/b3_latency.json, results/b7_systemone.json, results/b4_e2e_openai_gpt-4.1-mini_8.json.

• C: the bundled C core, called directly.
• Python: Model.decide through the C core, without per-option probabilities. With them it is 5.2 µs; without the C core (numpy only) 44 µs.
• JavaScript: Bundle.decide in Node, one question.
• via the proxy: a full HTTP round trip from the openai SDK to shad0w proxy and back on localhost, for a message the table answers.

End to end through the proxy#

Setup. A real app path: the openai Python SDK → shad0w proxy → the LLM. The proxy logs the LLM's answers, trains, then answers fresh messages. Agreement is checked by re-asking the LLM directly.

The openai SDK → shad0w proxy → the LLM, on this machine; agreement re-asks the LLM directly. Source: results/b4_*.json.

• LLM p50: median LLM latency seen by the client.
• answered by the table (fresh traffic): share of new messages answered by the proxy without calling the LLM.
• agreement with the LLM on those: how often those answers match what the LLM says when asked directly.
• answers logged / fresh messages: training log size and test size.

In the latency benchmark, a table answer through the proxy reached the client in 0.53 ms. The LLM's median in the 77-intent run was 1397 ms.

Decision models head to head#

Setup. Decision models answer typed questions directly, instead of your LLM. Here each one answered the same BANKING77 prompts. The last column asks: if we put the same certifier on each, how much could it serve while staying within 5% of the reference LLM?

BANKING77, 77 options, the same 2,000 test prompts for every engine, teacher gpt-4.1-mini (77% accurate). Decision models answer everything themselves; the last column is what the same certifier would let them serve while staying within 5% of gpt-4.1-mini. Source: results/b7_systemone.json.

The local decision models are more different from the reference LLM than α allows, so the certifier lets none of them serve on its own. They are still useful as the LLM shad0w learns from: with Kev-0.8B as the teacher, the table answered 70% of BANKING77 traffic.

At game speed#

A game frame is a hard deadline. In the snake arena, shad0w (trained on a hosted LLM's answers), three System One decision models running on the Apple M5 Pro's 16-core GPU (Laya, Kev-0.8B, Kev-4B) and the hosted LLM played the same 8 seeded games at every speed from one move a second to 120. A move lands on the first tick after its answer arrives; until then the snake keeps going straight. Every decision was a real call, timed as it happened.

• shad0w scored 28.0 food per game at every speed, deciding in about 1.6 µs per move from a 2 KB table.
• The hosted LLM scored 25.9 with no clock, and 0.1 at 10 moves a second (about 912 ms per answer).
• Laya (about 16 ms) and Kev-0.8B (about 23 ms) make most frames up to 30 moves a second; their low scores come from how they play without examples, not from time (compare the no-clock column).

The arena page replays every game and lists memory, model size and cost per lane. Pong, learned the same way from 3,000 answers, decides in about 1.5 µs per frame (Node on the Apple M5 Pro) and is playable in the browser. Sources: arena/results/arena_summary.json, arena/results/pong_eval.json; how to rerun: arena/README.md.

Certificate validity by simulation#

Setup. 1,000 calibration sets per row, drawn from known populations, for each procedure, α and calibration size. "flat" means the LLM makes random mistakes regardless of the message; "rising" means disagreement rises as the table's confidence falls. The column that matters is share of certificates above α: it must stay at or below δ = 10%. "upper CI" is the upper end of a 95 percent confidence interval on that share, given 1,000 draws.

δ = 0.1 allows up to 10%. Source: results/b8_certificate_sim.json.

Result: 0.0–8.1% of certificates exceeded α in simulation (δ allows 10%). The default is auto.

Earlier runs (0.1, embedding-model teacher)#

Setup. Earlier release, earlier certificate procedure, with an embedding model or Kev-0.8B as the teacher instead of an LLM. Kept for the wider range of tasks.

Earlier runs (pre-0.1.1 certificate procedure), 3 seeds, 7k fit / 3k calibration rows. Two tasks landed above α on the mean; the bound allows that in up to δ = 10% of certifications. Source: legacy/benchmarks/results/phase31_fidelity.json.

"cascade acc − teacher acc" is the change in accuracy against gold, in points, from putting the table in front of the teacher.

Where shad0w loses#

• Other learned methods served more on BANKING77. The kNN router answered 69% and the fine-tuned head 75%, against the table's 61%. Both run an embedding model on every message (7.2 ms vs 4.0 µs) and ship a much larger model. If milliseconds are fine and you already host embeddings, they are a reasonable choice.
• CLINC150: realised disagreement above α, because the traffic mix moved. CLINC150's test split has far more out-of-scope messages than its training split, and the training split is what the log looks like here. The certificate was computed on mostly in-scope traffic, then served a test set full of off-topic messages. Realised disagreement went above α on every seed (the top of the 3.0–6.0% range), and 5% of out-of-scope messages were answered with a real intent. This is the "traffic like the calibration traffic" condition failing, not a bug: calibrated on a test-like mix, the bound held again at a lower served share. In a live deployment the drift guard flags this shift after a few hundred decisions and defers those messages; this benchmark compares methods without the guard, so it shows the unguarded number. What to do: limits.
• AG News: two of three seeds landed slightly above α at α = 5% (5.1% and 5.6%; the third 3.5%), though the mean stayed below it. This is the δ allowance at work, and why spot checks exist.
• Sentiment (SST-2): 0% served. The table tops out near 79% accuracy on sentiment, so it never certifies and defers everything.
• Legal clauses (LEDGAR): 13% served in an earlier run. Long texts where one clause decides are hard for a word-pattern table.
• A noisy LLM is hard to copy. With a small local model (qwen3.5 4B through Ollama) as the LLM on 77 intents, the table certified little or nothing: when the LLM is inconsistent, no threshold keeps disagreement under α.
• Accuracy is capped by your LLM. Kev-0.8B alone scored 80.0% against gold, more than the table-plus-gpt-4.1-mini (OpenAI) cascade (76.5%). The table can only be as accurate as the model it copies.
• Cold start. Nothing is served until enough answers are logged and train() has run (see the growth table above).

Reproduce#

bash bench/setup.sh && bash bench/run_all.sh      # writes results/*.json, results/SUMMARY.md, results/evidence.json
cd shad0w-studio && python make_numbers.py --check && python build_site.py

The OpenAI steps skip themselves when OPENAI_API_KEY is unset. BENCHMARK.md in the kit lists each benchmark, its options and its runtime.

Provenance of every number#