Capabilities & Limitations

An honest account of what PRIOR does, what it doesn't, and how we measured it, so you can decide where it fits before you deploy it. Every number below comes from matched-pair tests (the same model, seed, and harness, run with steering off and on) on named models and hardware, with an independent judge where noted. We publish provenance with every figure, and we don't publish numbers we can't defend.

How to read these results

PRIOR is two parts, and it's worth keeping them separate when you read any number:

• The actuator. This is the golden-layer steering that forces the outcome (a refusal, or an injected fact) once a decision has been made.
• The gate. This is the detector that decides whether to act on a given request.

The short version: where the gate routes a request to RED, the actuator does its job essentially perfectly. The frontier of ongoing work is the gate's recall (catching more phrasings of the same intent), not the actuator. We call that out explicitly in the limitations below.

How we measured

• Judged fleet (the numbers below): twelve models from 1B to 14B, across six architecture families: Llama-3.2, Qwen3 (including an abliterated build), Gemma-3, Gemma-4, Mistral-Nemo, and SmolLM3. Model counts quoted in the tables are the fleet each specific test ran on — they are not the size of the supported registry, and we don't restate them upward as the registry grows.
• Supported registry: 25 calibrated models — 17 certified, 8 beta. A model is certified when its golden-layer sweep clears the fidelity floor; otherwise it keeps its swept parameters and is tagged beta. Uncalibrated models still run, on ratio-derived defaults.
• Hardware: commodity NVIDIA GPUs, an RTX 3060 (12 GB) and an RTX 5050 Laptop (8 GB), CUDA 12.4 / 12.8. Every fleet result is reproduced per model.
• Judging: safety is scored by the independent cais/HarmBench classifier, with an independent LLM-judge cross-check (93.6% agreement, n=800) that rated our reported figures as conservative floors; that is, the real numbers are likely better, not worse.

Capabilities

Safety: the actuator flips what the gate catches

Honest framing: most per-model attack-success drops are small (1–7pp), because modern instruct models already refuse most obvious harm at baseline. PRIOR's safety value is largest on high-headroom or abliterated models where the base guardrails are weak. And everywhere, it flips 100% of what the gate routes to RED, deterministically, regardless of the model's own inclination.

Safety: enforcement survives prompt injection

This is the sharpest difference between in-weights enforcement and anything that lives in the context window. A safety system prompt can be stripped by a one-line injection; PRIOR cannot, because there is nothing in the context for the injection to address.

Refusal rate (higher is safer), 50 StrongREJECT prompts, scored by StrongREJECT's fine-tuned classifier:

The system prompt loses 56 points; PRIOR does not move. Measured mid-run, PRIOR's RED-firing count was identical with and without the injection — the gate reads the harmful request, which the injection leaves intact. On the abliterated model the system prompt was worth nothing even unattacked (0%), because abliteration removes the model's disposition to honor it.

Read this as a relative result. 48% is not 90%. The shortfall is gate recall, not the actuator — once a request routes RED it is refused. The claim is that in-context defenses collapse under injection and in-weights enforcement doesn't, not that either is complete.

Measured at 200 output tokens, with zero degeneration among steered responses (0 of 21 steered rows in every PRIOR condition, checked against the run's own stored outputs). That matters because a garbled response is not a refusal, and a classifier will happily score one as though it were — so a steering result reported without it can be inflated by outputs that protected nobody.

Zero capability tax when no policy fires

When a request isn't in policy scope, PRIOR is a no-op. Not "a small delta," but byte-for-byte the base model.

Contextual grounding & anti-poison

Limitations

We'd rather you learn these here than in production.

The current frontier is gate recall, not the actuator

Where the gate routes a request to RED, the actuator refuses it ~100% of the time. The open work is the detector catching more phrasings of the same intent. The shipping detector can under-fire on short or novel phrasings that lack obvious trigger language. Two things soften this in practice, and both are levers you control:

• Recall is a tunable operating point, and we moved it in 1.1.0. The shipped harm_veto pack was retuned from t_high 0.516 to 0.43, roughly doubling HarmBench RED coverage (≈28% → ≈62%) at ≈1% benign false-fire. That is a deliberate trade: the previous point was precision-first to the point of under-protecting. Tighten t_high if your deployment prefers the old behavior. The YELLOW confirm band still absorbs most ambiguous cases without blocking anything.
• The intent encoder is the structural lever, and it bounds everything else here. Detection accuracy is bounded by the embedding encoder, and the shipping gate scores with MiniLM. Encoder work remains on the roadmap; it is the lever on every number in this section, including the one below.

What PRIOR can and cannot separate

The gate scores how close a request is to your exemplars. That makes it reliable at topic and domain boundaries — is this about our product, is this about a competitor, is this a database operation — and unreliable at distinctions of intent inside a topic you allow.

The test to run before you write a pack: put down the request you want blocked, and the closest request you want allowed. If they are about the same subject, the gate will score them alike, and moving the threshold only trades one error for the other.

Same subject, opposite policy status. Measured on our own rule-shaped pack: holding zero false-blocks on genuine customer questions, only 14.0% of violating requests were caught — because that one allowed question sets t_high for every rule in the pack.

It is worth seeing how narrow the cause is. That question scores 0.6331; the next-highest benign question scores 0.5239. The threshold is not being crowded by a mass of near-miss traffic — it is set by a single row standing 0.11 clear of everything else, and it is a question a bank plainly has to answer. That is not a tuning failure, and no threshold fixes it: a rule about what someone may ask for is an intent distinction by construction, and a cosine gate does not see intent.

So express policy as a boundary — the domain your agent serves, or the class of thing it must never touch — rather than as a rule about intent within a subject you otherwise permit. Where the policy is a boundary, the same gate reaches 96.7% out-of-domain recall while hard-blocking none of 55 genuine in-scope questions, and holds that rate across a 10-turn conversation. That is a routing figure — how often the gate puts an out-of-domain request in the RED zone. What happens next is the actuator's job, and on the released image a RED-routed request comes back with steering engaged and a non-zero injection count. Measured on the published sample pack. Read the provenance precisely: the out-of-scope side is a blind third-party set, which is what makes the recall figure worth anything; the pack's exemplars and the in-scope questions are ours. It is a worked example, not a customer deployment or a fleet result. Gate routing depends on the prompt and the pack, not on the generation model.

Fixed in 1.2.2, and worth knowing if you pulled earlier. Conversation scoring takes the most alarming reading across several views of a turn. Before the fix that was a plain maximum — fail-safe for a prohibition pack (highest score = closest to the forbidden thing) and inverted for a membership pack, where the highest score means most in-domain. One prior in-domain turn downgraded an out-of-domain refusal to a confirm, and scope enforcement fell away as a conversation went on, to 1.7% by turn 10 against 96.7% after the fix. Prohibition packs were never affected.

All images before 1.2.2 have been withdrawn. ghcr.io/eagle-logic/prior:slim now resolves to 1.2.2; re-pull if you took a copy before 2026-08-14.

x-prior-steering is the header to check on any install. If it reads disabled — most often because no license.bin is mounted yet — no policy is being applied regardless of what the zone header says. A routed request and an enforced one are not the same thing, and only that header tells them apart.

Some attacks are out of scope by design

Absent-intent dual-use composition (e.g. "fix these vulnerabilities," then diff the patches to reconstruct the exploit) carries no harmful signal in the prompt, the output, or the model's internal state (the model is genuinely doing something benign). No model-layer guardrail can detect it, and we don't claim to. Mitigating this is an operational/platform concern (usage-pattern monitoring, provenance, attribution), not something any inference-time control solves. We state it rather than imply coverage we don't have.

What has and hasn't been independently judged yet

• Independent cais-judging is complete across the full 1B to 14B fleet (Llama-3.1-8B fully judged).
• Multi-turn: payload-split attacks are hard-blocked; compliance-grooming is routed to human-confirm (YELLOW). Across-turn composition is validated at ≤8B; above 8B is pending.
• Faithfulness figures are for adversarial / poisoned retrieval. On a clean single-source pipeline, modern models are already faithful, so that isn't where PRIOR earns its keep.

Architecture coverage: two steering paths, chosen for you

Steering ships in two modes, and 1.1.0 made the choice automatic — the engine picks per model from its architecture and free VRAM at load.

Logits-only exists because the exact-gradient path needs a backward pass that Mixture-of-Experts, MatFormer, and SSM/linear-attention hybrids do not support — on those it fails rather than degrading. Before 1.1.0 that put DeepSeek-Coder-V2, Qwen3-30B-A3B, Mixtral and gemma-4-E4B out of scope; they are now supported through logits-only. It is also the path for the tightest-VRAM cards, since it allocates no gradient context.

On clean 8B HarmBench, logits-only matches L2LM on harm reduction to within ~0.6pp and degenerates 3–4× less. You can force it globally (PRIOR_LOGITS_ONLY=1), live (PATCH /admin/runtime), or per request. A request may opt into logits-only but never out of it.

Steering is calibrated per model

The steering geometry is a property of each model and must be calibrated before it's trusted. Models in the registry are graded certified (calibration swept and validated) or beta (sensible defaults, not yet swept). Bringing a new model means a one-time calibration; policies carry over unchanged.

Provenance

Judged model: Llama-3.1-8B-Instruct · Fleet (1B–14B): Llama-3.2, Qwen3 (incl. abliterated), Gemma-3, Gemma-4, Mistral-Nemo, SmolLM3 · Hardware: NVIDIA RTX 3060 (12 GB) and RTX 5050 Laptop (8 GB), CUDA 12.4 / 12.8 · Safety judge: independent cais/HarmBench classifier · Cross-check: independent LLM-judge, 93.6% agreement (n=800), scored as conservative floors. Across-turn composition above 8B is in progress.

Injection-survival and detection-recall figures (1.1.0) come from a separate release-candidate pass on a 4B-class fleet — Qwen3-4B, Qwen3-4B-Instruct-2507, and an abliterated Qwen3-4B-Instruct-2507-heretic — measured through the production container's /v1/chat/completions and scored by StrongREJECT's fine-tuned classifier (injection, n=50) and the cais/HarmBench classifier (HarmBench, 159 behaviors). Benign retention on that pass: 236/236 byte-identical on XSTest-safe. These are a different fleet from the 1B–14B numbers above and are not interchangeable with them.

Shareable version: these results are also published, in a form built for sharing, at eagle-logic.com/capabilities, with interactive demos of the faithfulness, grounding, and jailbreak results. Useful for a security review or an internal pitch.