An honest account: what works, what doesn't, and the method behind both.
Every number below comes from matched-pair tests: the same model, seed, and harness, run with steering off and on, on named models and hardware, with an independent judge where noted. We publish the model, hardware, and sample size behind every figure, so you can reproduce it.
Two different jobs, two different scorecards.
PRIOR does two things, and every number below is clearer if you keep them apart: it decides whether a request breaks one of your rules, and then it enforces the outcome: a refusal, or a corrected answer.
Think of it as a detector and an enforcer. The detector reads the intent of the request and decides whether to act. The enforcer is what makes that outcome stick, once the decision is made.
The short version: when the detector flags a request, the enforcer does its job essentially perfectly. The open work, the part we're still improving, is helping the detector recognize more of the ways the same bad request can be worded. We call that out plainly in the limitations below.
How we measured
Once the detector fires, the enforcer takes over the result.
When the detector flags a request, the outcome holds every time, no matter what the underlying model would have done on its own, and it's checked by an independent judge. The primary judged model here is Llama-3.1-8B-Instruct; where a figure spans the wider fleet, we say so.
86% less harm on forbidden prompts, and less than half what a strong safety prompt leaves.
Run natively on the shipping container against Mistral-Nemo-12B, an authored pack cuts mean harm on StrongREJECT from 0.600 to 0.084. A strong safety system prompt only reaches 0.184, so PRIOR removes more than half of what the prompt leaves behind.
| Condition | Mean harm | vs. base |
|---|---|---|
| Mistral-Nemo-12B, unprotected | 0.600 | base |
| Plus a strong safety system prompt | 0.184 | -69% |
| Plus PRIOR (authored StrongREJECT pack) | 0.084 | -86% |
Judge: StrongREJECT's own fine-tuned classifier. Baseline: unprotected Mistral-Nemo-12B via stock Ollama, a clean no-PRIOR engine. PRIOR: the shipping image ghcr.io/eagle-logic/prior:slim with a policy pack authored for StrongREJECT's categories. On the 49 of 60 prompts the base model actually complied with (mean harm 0.723), PRIOR drove harm to 0.101. Reaching StrongREJECT's categories took an aggressive pack that lifts RED coverage from 8.3% to 51.7%, at a measured 8.0% over-refusal cost on XSTest. The default safety pack adds none.
When no policy fires, PRIOR is a no-op.
Not "a small delta." Byte-for-byte the base model. You don't trade capability for control.
Anchor the model to the truth, even when the source is poisoned.
A fact you ground this way overrides stale or tampered information in the request, at zero prompt tokens, so a poisoned or out-of-date source can't move the value the model answers with.
We'd rather you learn these here than in production.
The open work is the detector, not the enforcer.
When the detector flags a request, PRIOR refuses it essentially every time. What we're still improving is the detector recognizing more of the ways the same bad request can be worded. It can miss short or unusual phrasings that don't use obvious wording. Two dials you control soften this:
You choose how aggressive it is. Out of the box it's tuned to never block a legitimate request, which means it errs toward letting borderline cases through. The "confirm" band already routes most ambiguous cases to a human check without blocking anything, and you can turn the sensitivity up or add your own detection rules.
Some attacks are out of scope by design.
Some attacks hide the harmful intent entirely. For example: ask the model to "fix these security bugs," then study the fixes to work out the original exploit. Nothing in the request, the answer, or the model's own reasoning looks harmful at any single step. No tool that watches the model can catch this, and we don't pretend otherwise. Defending against it is a matter of monitoring usage patterns and tracking who did what, not something any in-the-model control can solve. We say so rather than imply coverage we don't have.
Each model is calibrated once. How PRIOR steers depends on the specific model, so every model is checked before it's trusted and graded certified or beta. Adding a new model is a one-time setup step; your policies carry over to it unchanged.
What has and hasn't been independently judged yet
| Scope | Status |
|---|---|
| Safety, 1B to 14B models | Independently judged, complete |
| Attacks split across several messages | Blocked |
| Slowly talking the model out of its rules | Routed to a human check |
| Harmful intent assembled over a whole conversation | Confirmed up to 8B; larger models in progress |
| Fact-faithfulness figures | Measured under deliberately tampered source data (a clean, single-source setup is already faithful) |
Steering is nearly free, and only runs when needed.
Normal requests never touch the steering machinery. They run at your model's full speed. Steering engages only when the gate flags a request; then it computes the steering direction once and holds the rest of the response to it. In practice a flagged request is a refusal, so it returns promptly.
The cost model is simple: pay nothing on the traffic that's fine, and a small one-time cost on the traffic you're stopping.
Pick the model, quantization, and mode together, and PRIOR runs within your card's memory, not over it.
Judged model: Llama-3.1-8B-Instruct · Fleet (1B to 14B): Llama-3.2, Qwen3 (incl. abliterated), Gemma-3, Gemma-4, Mistral-Nemo, SmolLM3 · Hardware: commodity NVIDIA GPUs to 12 GB (RTX 3060, RTX 5060), CUDA 12.4 / 12.8 · Safety judge: independent cais/HarmBench classifier · Cross-check: independent LLM-judge, 93.6% agreement (n=800), scored as conservative floors. Across-turn composition above 8B is in progress.