The proxy: zero code changes#

shad0w proxy sits between your app and any OpenAI-compatible API. Your code keeps using its OpenAI SDK; only the base_url changes. This page covers starting it, marking which calls are decisions, keys, what comes back, and every flag.

TL;DR#

pip install "shad0wllm[compile]"
shad0w proxy --upstream https://api.openai.com/v1 --auto-train 1000

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8010/v1")             # 1. the only change to your client
r = client.chat.completions.create(
    model="gpt-6-luna", messages=[{"role": "user", "content": "my card was stolen"}],
    extra_headers={"X-Shad0w-Question": "intent", "X-Shad0w-Options": "refund,lost_card,balance,other"})  # 2. mark it
print(r.choices[0].message.content)                                   # lost_card

At first every marked call goes to the upstream LLM and its answer is logged. With --auto-train 1000, the proxy trains a table (shad0w's small learned model) after 1,000 answers. From then on it answers the calls it is certified on by itself, in microseconds. ("Certified" = tested on answers it never trained on, and allowed to answer only where it disagrees with your LLM on at most α of them; α = 5% by default.) Everything that is not a marked decision is forwarded untouched.

1. Start it#

shad0w proxy --upstream https://api.openai.com/v1

shad0w proxy on http://127.0.0.1:8010/v1  ->  https://api.openai.com/v1
  dashboard http://127.0.0.1:8010/   metrics http://127.0.0.1:8010/metrics   data /home/you/app/shad0w
  decisions API: POST /v1/decisions and /v1/systemone need no marking
  chat: mark a decision with header 'X-Shad0w-Question: <name>', model 'shad0w/<name>', or a forced tool
  upstream key: each client sends its own

The upstream can be OpenAI, Groq, Together, OpenRouter, Mistral, a LiteLLM proxy, Ollama (http://localhost:11434/v1), vLLM, or anything else that speaks the chat completions API.

2. Point your client at it#

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8010/v1")

import OpenAI from "openai";
const client = new OpenAI({ baseURL: "http://localhost:8010/v1" });

Keys#

There is no flag that takes the key itself: it would show up in ps and shell history. The two flags cannot be combined. The startup banner says where the upstream key comes from, masked: upstream key: set via OPENAI_API_KEY (sk-…3f9a).

Access token#

On its own machine (--host 127.0.0.1, the default) the proxy needs nothing more. As soon as it listens on a network (a container, --host 0.0.0.0, a LAN address), give it an access token, so only your apps can use it:

export SHAD0W_PROXY_TOKEN="$(openssl rand -hex 24)"    # or --token-env NAME, or --token-file /run/secrets/shad0w-token
shad0w proxy --host 0.0.0.0 --api-key-file /run/secrets/openai

Every route except GET /v1/health then needs the token, sent one of three ways:

The header that carried the token is never sent upstream. A --token-file is re-read on every request, so a rotated secret needs no restart. Requests without the right token get a 401 and never reach your LLM.

Refused on purpose: listening beyond this machine with the proxy's own key (--api-key-env / --api-key-file) and no token. Anyone who could reach the port could spend your key and read recent messages on the dashboard, so the proxy will not start. If the network itself is private and you accept that, pass --insecure-open. A non-local proxy without a token that forwards each client's own key starts with a warning.

3. Mark which calls are decisions#

Using the OpenAI Decisions API or a System One model (Jev, Kev, Laya)? Nothing to mark. Requests to /v1/decisions and /v1/systemone already name each question and list its answers. See Decisions API & System One.

For chat completions, pick whichever is easiest:

msgs = [{"role": "system", "content": "Classify the support message."},
        {"role": "user", "content": "my card was stolen"}]

# a header
client.chat.completions.create(model="gpt-6-luna", messages=msgs, extra_headers={"X-Shad0w-Question": "intent"})

# a model name: "shad0w/<question>@<upstream model>" (or "shad0w/<question>" with --model on the proxy)
client.chat.completions.create(model="shad0w/intent@gpt-6-luna", messages=msgs)

# a forced tool whose parameters hold exactly one enum: the tool's name is the question
client.chat.completions.create(model="gpt-6-luna", messages=msgs,
    tools=[{"type": "function", "function": {"name": "intent", "parameters": {"type": "object",
        "properties": {"intent": {"type": "string", "enum": ["refund", "lost_card", "balance", "other"]}}}}}],
    tool_choice={"type": "function", "function": {"name": "intent"}})

Which shapes count as decisions is set by --capture (or capture in Configuration):

json_schema is off by default because many apps use named schemas for things that are not decisions. Turning it on treats every such request as one. --capture takes a comma-separated list and can be repeated; it replaces the default list.

Where the options come from, first match wins: the request's own enum (JSON schema or tool parameter), the X-Shad0w-Options: refund,lost_card,balance,other header, --schema, then a trained table. Without any of these the proxy learns the option set from the short answers your LLM gives.

The text shad0w learns from is the last user message. Put the thing to classify there and keep instructions in the system message.

model="shad0w/<name>" with no @<model> and no --model flag is refused with a 400, because the proxy would not know which upstream model to call.

What happens to a decision#

A table answer is shaped like the request asked for:

• plain text: the option name;
• JSON object mode: {"intent": "lost_card"};
• your JSON schema: {"<your field>": "lost_card"};
• a forced tool: a tool_calls message with finish_reason: "tool_calls";
• stream: true: server-sent events, for text and tool calls;
• /v1/decisions and /v1/systemone: the same JSON shape the upstream would return (details).

Its usage is all zeros, and it carries an extra shad0w object: question, source, answer, confidence, certified, flag, latency_us.

Response headers on a marked chat completion:

On /v1/decisions and /v1/systemone, x-shad0w-source is table, mixed or teacher. Request headers starting with X-Shad0w- are never forwarded upstream.

4. Train#

shad0w stats --log shad0w/intent/log.jsonl        # how many answers, which options, ready yet?
shad0w train --dir shad0w --question intent       # the running proxy picks up the new table within a second

Or let the proxy do it: --auto-train 1000 retrains a question every 1,000 new answers. By default (retrain = "gated") the new table replaces the old one only if it certifies at least four fifths (0.8) of the old one's share. Training needs pip install "shad0wllm[compile]". See Training.

Flags#

Every flag can also come from shad0w.toml or a SHAD0W_* variable, and a [questions.<name>] section sets values for one question; see Configuration. A flag beats both.

Routes#

The dashboard's Playground uses that last route: type a text, pick a question, and see the table's answer, its confidence against the threshold, and why; with --model set, also ask the upstream LLM and compare the two side by side.