Connect your LLM#

This page shows how to point shad0w at the LLM you already use, how to give it the API key, and what shad0w sends to that LLM.

The LLM that answers today is shad0w's teacher: the table (shad0w's small learned model) learns its answers and is certified against them. Name it with a "provider/model" string, or pass any function.

TL;DR#

import os, shad0w

intent = shad0w.decision("intent", options=["refund", "lost_card", "other"],
                         llm="anthropic/claude-haiku-4-5",
                         api_key=os.environ["ANTHROPIC_API_KEY"])

Providers#

Model names are examples; use any model your provider serves. If the key is missing, decision() raises a ValueError right away that says which variable it looked for. shad0w doctor --llm openai/gpt-6-luna checks the key without calling the model.

API keys#

Pass the key itself, the name of the variable that holds it, or a function that fetches it:

import os, shad0w

opts = ["refund", "lost_card", "other"]

# 1. the key itself
intent = shad0w.decision("intent", options=opts, llm="openai/gpt-6-luna", api_key=os.environ["OPENAI_API_KEY"])

# 2. the NAME of an environment variable (read on every request, so a rotated value is picked up)
intent = shad0w.decision("intent", options=opts, llm="openai/gpt-6-luna", api_key_env="SUPPORT_BOT_OPENAI_KEY")

# 3. a zero-argument function, called on every request (vaults, rotating keys)
intent = shad0w.decision("intent", options=opts, llm="openai/gpt-6-luna", api_key=lambda: my_vault.get("openai"))

# 4. once for the whole process
shad0w.configure(llm="openai/gpt-6-luna", api_key_env="SUPPORT_BOT_OPENAI_KEY")
intent = shad0w.decision("intent", options=opts)         # uses the defaults above

When several are set, the first match wins:

The key never leaks. It does not appear in repr(), logs, traces, pickles, shad0w doctor or shad0w config output, or error messages. At most you see sk-…3f9a, enough to tell two keys apart. To check where a key came from:

print(intent.teacher.key_source)     # set via SUPPORT_BOT_OPENAI_KEY (sk-…3f9a)
print(intent)                        # Shadow('intent', options=3, ..., llm=openai/gpt-6-luna, key: set via ... (sk-…3f9a))

Rules that keep keys out of the wrong places:

• shad0w.toml never holds a key. Put the variable's name there (api_key_env = "OPENAI_API_KEY"); an api_key = ... line is an error.
• api_key, api_key_env and base_url only apply to a "provider/model" string. With a function as llm, the function holds its own key, so passing them raises TypeError.
• A mistyped keyword raises TypeError with a suggestion: apikey → did you mean 'api_key'?.
• shad0w.configure() with no arguments clears every process-wide default.

In JavaScript it works the same way: apiKey (a string or () => string | Promise<string>), apiKeyEnv, and configure({ llm, apiKey, apiKeyEnv, baseURL }). See JavaScript. For the proxy, see --api-key-env and --api-key-file.

Any other OpenAI-compatible server#

Give the model name and the server's base URL:

intent = shad0w.decision("intent", options=["refund", "lost_card", "other"],
                         llm="my-model", base_url="https://llm.internal/v1", api_key_env="INTERNAL_LLM_KEY")

base_url also overrides a known provider's URL (for example, a regional endpoint). To set it for every teacher at once, use shad0w.configure(base_url=...), SHAD0W_BASE_URL or base_url in shad0w.toml. The first one set wins, in that order, before the provider's default.

Decision models as the teacher#

Decision models answer with a typed choice instead of text, so there is no reply to parse.

• openai-decisions/<model> calls POST /v1/decisions (the OpenAI Decisions API).
• systemone/<model> calls POST /v1/systemone on the server in base_url (/v1 is added when missing).
• Yes/no questions are sent as predicate and noul questions respectively.

import shad0w

intent = shad0w.decision("intent",
                         options={"lost_card": "card lost or stolen", "refund": "wants money back", "other": "anything else"},
                         llm="systemone/kev", base_url="http://127.0.0.1:8081")   # e.g. llama-server -hf ggml-org/Kev-0.8B-GGUF --port 8081

To use a decision model with no code change at all, put shad0w proxy in front of it instead: see Decisions API & System One.

Reuse an SDK client#

Pass an openai SDK client (its proxies and custom auth then apply). shad0w calls client.chat.completions.create(...):

from openai import OpenAI
import shad0w

intent = shad0w.decision("intent", options=["refund", "lost_card", "other"], llm="gpt-6-luna", client=OpenAI())

Anything else: give it a function#

A function that returns the reply text. shad0w builds the prompt and maps the reply to an option, exactly as for a built-in provider:

import shad0w

def complete(messages):                      # messages = [{"role": "system", ...}, {"role": "user", ...}]
    return my_llm(messages)                  # return the reply text, e.g. "lost_card" or '{"answer": "lost_card"}'

teacher = shad0w.llm_teacher(["refund", "lost_card", "other"], question="intent", complete=complete)
intent = shad0w.decision("intent", llm=teacher)

A function that returns the answer. No prompt, no parsing: whatever it returns is the answer (it should be one of your options):

intent = shad0w.decision("intent", options=["refund", "lost_card", "other"], llm=lambda text: my_classifier(text))

This is also how you use LangChain, LiteLLM or any SDK: see Frameworks.

LLM settings#

These keywords go to decision(), Shadow(llm=...) or shad0w.llm_teacher(...). They only apply to a "provider/model" string (or to llm_teacher).

What shad0w sends#

One chat request per decision: a system message listing your options (with their descriptions), then the user's text, at temperature=0. It asks for structured output: a JSON object whose answer must be one of your options. If the server does not support that, shad0w notices once and switches to plain text.

Every reply is mapped back to exactly one option. These all work:

• an exact match, or one that differs only in case or spacing (Lost Card → lost_card);
• an option named inside a short sentence (The answer is lost_card.), when exactly one option fits;
• {"answer": "refund"}, also inside a fenced code block.

If no option matches, the call raises shad0w.TeacherError and nothing is logged, so bad replies never become training data. The error says what the model replied and why it was asked.

Other built-in behaviour:

• Retries. Rate limits (429) and server errors (5xx) are retried with backoff, honouring Retry-After.
• Rejected parameters. If a model rejects temperature or max_tokens (as reasoning models do), that parameter is dropped and the request retried.

Yes/no questions#

import shad0w

spam = shad0w.decision("spam", options={"type": "yesno", "instructions": "Is this message spam?"},
                       llm="openai/gpt-6-luna")
spam("WIN A FREE CRUISE, click now").answer      # True

options=bool does the same without instructions. The answer is a Python bool.