An honest account: what works, what doesn't, and the method behind both.
Every number below comes from matched-pair tests: the same model, seed, and harness, run with steering off and on, on named models and hardware, with an independent judge where noted. We publish the model, hardware, and sample size behind every figure, so you can reproduce it.
Two different jobs, two different scorecards.
PRIOR does two things, and every number below is clearer if you keep them apart: it decides whether a request breaks one of your rules, and then it enforces the outcome: a refusal, or a corrected answer.
Think of it as a detector and an enforcer. The detector reads the intent of the request and decides whether to act. The enforcer is what makes that outcome stick, once the decision is made.
The short version: when the detector flags a request, the enforcer does its job essentially perfectly. The open work, the part we're still improving, is helping the detector recognize more of the ways the same bad request can be worded. We call that out plainly in the limitations below.
How we measured
Once the detector fires, the enforcer takes over the result.
When the detector flags a request, the outcome holds every time, no matter what the underlying model would have done on its own, and it's checked by an independent judge. The primary judged model here is Llama-3.1-8B-Instruct; where a figure spans the wider fleet, we say so.
86% less harm on forbidden prompts, and less than half what a strong safety prompt leaves.
Run natively on the shipping container against Mistral-Nemo-12B, an authored pack cuts mean harm on StrongREJECT from 0.600 to 0.084. A strong safety system prompt only reaches 0.184, so PRIOR removes more than half of what the prompt leaves behind.
| Condition | Mean harm | vs. base |
|---|---|---|
| Mistral-Nemo-12B, unprotected | 0.600 | base |
| Plus a strong safety system prompt | 0.184 | -69% |
| Plus PRIOR (authored StrongREJECT pack) | 0.084 | -86% |
Judge: StrongREJECT's own fine-tuned classifier. Baseline: unprotected Mistral-Nemo-12B via stock Ollama, a clean no-PRIOR engine. PRIOR: the shipping image ghcr.io/eagle-logic/prior:slim with a policy pack authored for StrongREJECT's categories. On the 49 of 60 prompts the base model actually complied with (mean harm 0.723), PRIOR drove harm to 0.101. Reaching StrongREJECT's categories took an aggressive pack that lifts RED coverage from 8.3% to 51.7%, at a measured 8.0% over-refusal cost on XSTest. The default safety pack adds none.
A system prompt can be stripped by the next sentence. Weights can't.
The most common objection we get is "why not just write a good safety system prompt?" So we measured one against a one-line "ignore previous instructions" override. Refusal rate, higher is safer.
| Defense | Qwen3-4B | Qwen3-4B-heretic (abliterated) |
|---|---|---|
| Safety system prompt alone | 64% | 0% |
| System prompt under injection | 8% | 2% |
| PRIOR under injection | 48% | 44% |
The in-context defense collapses by 56 points. PRIOR doesn't move: the detector fires on an identical set of requests with and without the attack, because the injection leaves the harmful request text intact and the enforcement never lived in the prompt. On the abliterated model the system prompt is worth nothing even unattacked, since abliteration strips the model's willingness to honor it.
Honest framing: 48% is not 90%, and the claim is relative, not absolute. The shortfall is detector recall, the same gate-recall frontier described in the limitations below, not the actuator. Once a request routes RED the intervention holds.
Judge: StrongREJECT's own fine-tuned classifier. Set: 50 StrongREJECT prompts, with and without a one-line injection. Models: Qwen3-4B and Qwen3-4B-Instruct-2507-heretic, measured on the 1.1.0 release candidate. This is a 4B-class pass and is deliberately not combined with the 1B to 14B fleet figures above, which come from a different fleet and a different judge.
236 of 236 benign answers came back byte-for-byte identical.
The counterweight to any safety number: on XSTest-safe prompts that routed GREEN, steered output was byte-for-byte identical to the unsteered model on all three models tested. Normal traffic is not merely "close enough", it is untouched. Over-refusal on a deliberately adversarial dual-use set was 1.2%, and those requests route YELLOW/confirm rather than hard-blocking.
XSTest-safe, n=250, of which 236 routed GREEN. Byte-identical on Qwen3-4B-heretic, Qwen3-4B and Llama-3.2-3B. Same 1.1.0 release-candidate pass as the injection result above.
Roughly double the harmful prompts caught, at the same false-fire rate.
The limitation we have always published is detector recall, not enforcement. In 1.1.0 the RED threshold was retuned from 0.516 to 0.43, which roughly doubles coverage: 67% of harmful prompts are now caught at a ~1% false-fire rate on benign traffic. Steered attack success lands at 2 to 3%. The trade we did not make: the benign false-fire rate is unchanged, so this is recall bought with calibration, not with a twitchier model.
HarmBench, 159 standard behaviors, judged by the independent cais/HarmBench classifier. False-fire measured on benign traffic. Figures are for the 1.1.0 harm_veto path.
When no policy fires, PRIOR is a no-op.
Not "a small delta." Byte-for-byte the base model. You don't trade capability for control.
Pin a value you trust, and a poisoned source can't move it.
A fact you ground this way overrides stale or tampered information in the request, at zero prompt tokens, so a poisoned or out-of-date source can't move the value the model answers with.
The precise claim: you supply the value to hold, and PRIOR guarantees the model emits it rather than the poisoned one. This is a compliance guarantee on a value you already know, not an accuracy improvement, and we don't claim it makes a model better at working out the truth unaided. On some models, pinning a value slightly lowers overall faithfulness on the rest of the answer.
It now steers the architectures the gradient method couldn't run.
PRIOR's original steering path needs an exact gradient through the model. On mixture-of-experts, MatFormer and hybrid state-space architectures that backward pass doesn't just run slowly, it fails outright. 1.1.0 adds a second, backward-free path that works from logits alone, so those models are steerable rather than gate-only. The engine reads the architecture and the free VRAM at load and picks between the two itself, with no configuration.
What this unlocks in practice: DeepSeek-Coder-V2, Qwen3-30B-A3B, Mixtral and gemma-4-E4B. The model registry grades every calibrated build and says which are cleared for full steering.
Both figures come from one controlled comparison: CAIS HarmBench standard behaviors, n=159 per arm, judged by the official cais/HarmBench-Mistral-7b-val-cls classifier, on Llama-3.1-8B and Qwen3-8B. Same container, same protocol, steering mode the only variable; each mode calibrated at its own operating point. Steered ASR lands within 0.6pp between the two paths (6.9% vs 7.5% on Llama, 3.1% vs 3.8% on Qwen), which is noise at this n. "Degenerate" is the share of steered outputs collapsing into repetition: 0.6% vs 2.5% on Llama, 0.0% vs 0.6% on Qwen.
Author and test the policy in the product, against your own data.
Most guardrail tools hand you a coding project. PRIOR takes your governance document, and as of 1.1.0 the whole authoring loop happens in the product rather than in a text editor and a redeploy.
We'd rather you learn these here than in production.
The open work is the detector, not the enforcer.
When the detector flags a request, PRIOR refuses it essentially every time. What we're still improving is the detector recognizing more of the ways the same bad request can be worded. It can miss short or unusual phrasings that don't use obvious wording. Two dials you control soften this:
You choose how aggressive it is. Out of the box it's tuned to never block a legitimate request, which means it errs toward letting borderline cases through. The "confirm" band already routes most ambiguous cases to a human check without blocking anything, and you can turn the sensitivity up or add your own detection rules.
Some attacks are out of scope by design.
Some attacks hide the harmful intent entirely. For example: ask the model to "fix these security bugs," then study the fixes to work out the original exploit. Nothing in the request, the answer, or the model's own reasoning looks harmful at any single step. No tool that watches the model can catch this, and we don't pretend otherwise. Defending against it is a matter of monitoring usage patterns and tracking who did what, not something any in-the-model control can solve. We say so rather than imply coverage we don't have.
Each model is calibrated once. How PRIOR steers depends on the specific model, so every model is checked before it's trusted and graded certified or beta. Adding a new model is a one-time setup step; your policies carry over to it unchanged.
What has and hasn't been independently judged yet
| Scope | Status |
|---|---|
| Safety, 1B to 14B models | Independently judged, complete |
| Attacks split across several messages | Blocked |
| Slowly talking the model out of its rules | Routed to a human check |
| Harmful intent assembled over a whole conversation | Confirmed up to 8B; larger models in progress |
| Fact-faithfulness figures | Measured under deliberately tampered source data (a clean, single-source setup is already faithful) |
Steering is nearly free, and only runs when needed.
Normal requests never touch the steering machinery. They run at your model's full speed. Steering engages only when the gate flags a request; then it computes the steering direction once and holds the rest of the response to it. In practice a flagged request is a refusal, so it returns promptly.
The cost model is simple: pay nothing on the traffic that's fine, and a small one-time cost on the traffic you're stopping.
Pick the model, quantization, and mode together, and PRIOR runs within your card's memory, not over it.
Judged model: Llama-3.1-8B-Instruct · Fleet (1B to 14B): Llama-3.2, Qwen3 (incl. abliterated), Gemma-3, Gemma-4, Mistral-Nemo, SmolLM3 · Hardware: commodity NVIDIA GPUs to 12 GB (RTX 3060, RTX 5050 Laptop), CUDA 12.4 / 12.8 · Safety judge: independent cais/HarmBench classifier · Cross-check: independent LLM-judge, 93.6% agreement (n=800), scored as conservative floors. Across-turn composition above 8B is in progress.