From your sessions to a model you can trust

Tironian is a loop with six steps. Each one is deliberately boring and checkable: plain code where plain code is enough, statistics where a decision matters, and your usual model as the backup at every point.

capture  → every model step and tool run, masked, stored locally
grade    → good / bad / unknown, from what actually happened
group    → recurring requests become task families
train    → one small model per family (LoRA → GGUF)
test     → held-out steps from sessions it never saw
route    → shadow first, then live, every answer checked

1. Capture

Tironian registers one piece of llm_execution middleware and ten observer hooks. The middleware sees each request Hermes sends to your model and the response that comes back. The hooks report tool results, exit codes, approvals, interrupts and session boundaries.

Before anything is stored, every string goes through the redactor: API keys and tokens for the common providers, JWTs, private keys, bearer tokens, credentials inside URLs, password= style values and other long random strings, then email addresses, phone numbers, card numbers (checked with the Luhn algorithm), IBANs, your home folder and your login name.

Capture understands all three wire formats Hermes uses: OpenAI style chat completions, Anthropic messages and the Responses API. Everything is stored in one shape, so one exporter and one model format cover every provider.

2. Grading

A model is only ever trained on turns that went well, so grading decides what Tironian learns. Every signal comes from what happened in the turn. The model's opinion of its own answer never counts.

A turn scoring 0.3 or more with no hard negative is good. A turn at −0.3 or below is bad. Everything else is unknown and is left out of training. When in doubt, a turn is not learned from.

3. Task families

A family is a recurring kind of request. Students are trained and tested per family, which keeps each one small and specialised. Grouping is deterministic, so the same history always gives the same families.

1. Each request is normalised. Paths, URLs, quoted text, numbers, file extensions and names written with a capital letter become placeholders. Common synonyms collapse to one word: biggest, largest and takes the most space all become largest.
2. Words that appear in most of your requests carry no information about which task it is, so they are ignored. The same list is used later when routing.
3. Requests are clustered in the order they happened. Each joins the most similar family or starts a new one, and a merge pass joins families that are really the same task.

"What are the 5 largest files in my Downloads folder?"
"Show me the 10 biggest files in Videos"
  → show the <n> largest files in my <name> folder

On the author's own data the grouping is pure: no family mixes two different tasks. It errs on the side of splitting, because two pure families are safe and one mixed family is not.

4. Training

When a family has enough good examples, Tironian builds a dataset in Nous's Hermes function calling format, the same format Hermes uses for its own trajectories. The split into training and held-out data is done by session, so no conversation appears on both sides.

Each example is a short, task-specific prompt: the family's tools in the Hermes system prompt and the most recent turns that fit in about 3,000 tokens. The live router builds exactly the same prompt with exactly the same code, which removes the most common reason a distilled model fails: being served a different prompt than it learned from.

Training never runs inside Hermes. A separate process in Tironian's own environment talks to Modal and reports progress through a status file, so a Hermes restart never loses a job. The spending cap is enforced as a timeout on the GPU function.

5. The gate

The trained model answers every held-out step, given exactly the context your model had. A step passes when:

• the output parses, with valid tool call JSON and no stray tags,
• it takes the same kind of action (a tool call or a final answer),
• it calls the same tools (when your model ran a few commands at once, running some of those same commands counts) with arguments that pass their schemas and match yours (80% of keys equal after normalising whitespace, quotes and slashes; a long value such as a shell command matches when most of its words agree, and if it still differs, the blind judge decides whether both commands do the job),
• for a final answer, a blind judge prefers it or calls it a tie, or without a judge, its wording overlaps yours enough.

Then comes the decision. Tironian computes the Wilson score interval for the pass rate and looks at its lower end, not the average. Passing 10 of 10 sounds perfect, but the lower bound is only 72%, because 10 examples prove very little. Passing 55 of 60 gives a lower bound of 82%. The default gate needs 85%, at least 30 held-out steps and 98% format validity.

n = 60, passed = 55
p̂ = 0.917
Wilson 95% interval = [0.819, 0.964]
gate: 0.819 < 0.85  →  not yet

The first step of a task (answering your request) and the steps that come after a tool result are scored separately, each with its own 30 step minimum and its own lower bound. Small models are often good at the first and shaky at the second: the first student Tironian trained on my machine wrote the right find command almost every time, then kept running more commands instead of reading the result. With separate scores it can serve just the first step, and your model writes the answer from the output. A kind of step never gets in on the strength of another.

The held-out set is not fixed at training time. Any session you record after training is one the model has never seen, so it joins the test too. That matters because the bar is strict: at 30 steps, even 29 right gives a lower bound of 83%, so a good model can sit just under the line. Rather than lowering the bar or paying for another run, you keep working and run hermes tironian eval again with more evidence.

A model that passes moves to shadow mode. It answers real requests in the background, its answers are compared with your model's, and only when it agrees often enough can you let it answer for real.

6. Routing

For each new request, Tironian decides in a few milliseconds whether it belongs to a promoted family. The match has to be confident and clearly better than the runner-up, the request has to offer the tools the model was trained with, and it can't contain images.

Then the small model answers through Hermes's local model runtime, and the answer is checked: it must parse, call only tools the request offers, and pass each tool's argument schema. If it does, Hermes receives it exactly as if your model had sent it, and runs its tools with the usual approvals. If not, your model answers.

Safety contract

• Your model is called at most once per request.
• Any error inside Tironian before that call sends the request to your model unchanged. Any error after it returns your model's answer.
• After one fallback in a turn, or after a tool step fails or is blocked, your model finishes the turn.
• A live model with more than 10% fallbacks over its last 50 requests, or two answers you marked bad, drops back to shadow mode by itself.
• A small model's own answers are never used for training.

Each of these is covered by a test, and the fallback after a blocked command was also checked on a real Hermes install: the small model made the first move, Hermes blocked its command, and the usual model finished the job.

Hermes surfaces used

No core patches, no runtime overrides, and no packages installed into Hermes. hermes plugins validate passes, including its security scan.