Same rows. Same certifier.
Microseconds against seconds.
Decision models answer "pick one of these options" for your app. We put shad0w next to five open ones (Julia-1, Laya, Kev-0.8B, Decider-0.8B and GLiNER2.5-Decide) on six public tasks, on a laptop CPU with no GPU. Every system answers the same test rows, one at a time, and every system's confidence goes through the same 5% certificate. Every answer is saved, so anyone can re-check the numbers without running a model.
The summary
What it shows, and what it doesn't
shad0w learns each task from its training split; the decision models answer zero-shot. A decision model fine-tuned on the task would score higher than here.
Here the teacher is the dataset's labels. In your app shad0w learns from your LLM's answers, and its certificate is against that LLM, not the truth.
Decision-model times include localhost HTTP and are on a CPU. They would be lower in-process, and much lower on a GPU (llama.cpp publishes 3–12 ms there).
- Rows. 500 test rows per task, a fixed random sample of the test split; 500 calibration rows from the training split, held out from training. A model that takes over 1 s per decision answers the first 150 rows, marked "150 rows".
- Certified uses Learn-then-Test with a Clopper–Pearson bound (α = 5%, δ = 10%) for every system: a confidence threshold picked on the calibration rows. Coverage is the share of test rows at or above it; served error is the error on those rows. A system whose confidence cannot separate right from wrong certifies 0%.
- Time is the wall time of one call with one input, after 20 warm-up calls. shad0w runs in-process on one thread through the C core. Decision models run with 5 threads; their single-thread time is shown where it was measured.
- Left out: Laya-multilingual, Laya-typed-decisions and d1-omni-600M did not load in this llama.cpp build. The 4B–9B models take seconds per decision on a CPU and are optional in the harness.
Check every number, with or without the models
Every answer of every system is saved, one JSON line per row ({"split": "test", "i": 17, "y": 3, "pred": 3, "conf": 0.9981, "us": 2.41}). Re-scoring needs no model, no network and no GPU.
git clone https://github.com/2796gaurav/shad0w && cd shad0w
pip install -e ".[compile]"
python bench/score.py # every number on this page, from the saved predictions
python bench/run.py # or re-run everything (llama.cpp's llama-server on PATH; about two hours on a laptop CPU)Protocol and caveats: bench/README.md · raw results: results.json · machine: