# Capabilities & Limitations An honest account of what PRIOR does, what it doesn't, and how we measured it, so you can decide where it fits before you deploy it. Every number below comes from matched-pair tests (the same model, seed, and harness, run with steering off and on) on named models and hardware, with an independent judge where noted. We publish provenance with every figure, and we don't publish numbers we can't defend. --- ## How to read these results PRIOR is two parts, and it's worth keeping them separate when you read any number: - **The actuator.** This is the golden-layer steering that forces the outcome (a refusal, or an injected fact) *once a decision has been made*. - **The gate.** This is the detector that decides *whether* to act on a given request. The short version: **where the gate routes a request to RED, the actuator does its job essentially perfectly.** The frontier of ongoing work is the gate's *recall* (catching more phrasings of the same intent), not the actuator. We call that out explicitly in the limitations below. ## How we measured - **Judged fleet (the numbers below):** twelve models from 1B to 14B, across six architecture families: Llama-3.2, Qwen3 (including an abliterated build), Gemma-3, Gemma-4, Mistral-Nemo, and SmolLM3. Model counts quoted in the tables are the fleet each specific test ran on — they are not the size of the supported registry, and we don't restate them upward as the registry grows. - **Supported registry:** **25 calibrated models — 17 certified, 8 beta.** A model is *certified* when its golden-layer sweep clears the fidelity floor; otherwise it keeps its swept parameters and is tagged *beta*. Uncalibrated models still run, on ratio-derived defaults. - **Hardware:** commodity NVIDIA GPUs, an RTX 3060 (12 GB) and an RTX 5050 Laptop (8 GB), CUDA 12.4 / 12.8. Every fleet result is reproduced per model. - **Judging:** safety is scored by the independent **cais/HarmBench classifier**, with an independent LLM-judge cross-check (**93.6% agreement, n=800**) that rated our reported figures as *conservative floors*; that is, the real numbers are likely better, not worse. --- ## Capabilities ### Safety: the actuator flips what the gate catches | Result | What it means | |---|---| | **100%** of gate-routed harmful completions refused (**79/79**, cais-judged, across 11 models) | When the gate fires, the intervention holds, and it's independently verified. | | Mistral-Nemo-12B attack-success **44.6% → 28.9%** | On the highest-headroom model, PRIOR roughly halves the residual attack surface. | | **96%** jailbreak recovery via span-scoring, at **0** benign false-blocks | Holds up against obfuscated/jailbroken phrasings without taxing legitimate traffic. | | Generalization to unseen attacks: **AUC ~0.93** | Coverage converges on a shared harm manifold; it is not endless string-chasing. | > **Honest framing:** most *per-model* attack-success drops are small (1–7pp), because modern > instruct models already refuse most obvious harm at baseline. PRIOR's safety value is largest on > high-headroom or abliterated models where the base guardrails are weak. And everywhere, it flips > **100%** of what the gate routes to RED, deterministically, regardless of the model's own inclination. ### Safety: enforcement survives prompt injection This is the sharpest difference between in-weights enforcement and anything that lives in the context window. A safety **system prompt** can be stripped by a one-line injection; PRIOR cannot, because there is nothing in the context for the injection to address. Refusal rate (higher is safer), 50 StrongREJECT prompts, scored by StrongREJECT's fine-tuned classifier: | Defense | Qwen3-4B | Qwen3-4B-heretic (abliterated) | |---|---|---| | Safety system prompt alone | 64% | 0% | | Same prompt, under injection | **8%** | 2% | | PRIOR, under the same injection | **48%** | **44%** | The system prompt loses 56 points; PRIOR does not move. Measured mid-run, PRIOR's RED-firing count was identical with and without the injection — the gate reads the harmful *request*, which the injection leaves intact. On the abliterated model the system prompt was worth nothing even unattacked (0%), because abliteration removes the model's disposition to honor it. > **Read this as a relative result.** 48% is not 90%. The shortfall is gate *recall*, not the > actuator — once a request routes RED it is refused. The claim is that in-context defenses collapse > under injection and in-weights enforcement doesn't, not that either is complete. Measured at 200 output tokens, with **zero degeneration among steered responses** (0 of 21 steered rows in every PRIOR condition, checked against the run's own stored outputs). That matters because a garbled response is not a refusal, and a classifier will happily score one as though it were — so a steering result reported without it can be inflated by outputs that protected nobody. ### Zero capability tax when no policy fires When a request isn't in policy scope, PRIOR is a no-op. Not "a small delta," but byte-for-byte the base model. | Result | What it means | |---|---| | **GSM8K byte-identical across all 12 models** (exact-match 1.0), validated to Qwen3-14B (87%) | Math reasoning is untouched by steering. | | **MMLU 67.25%**, baseline = steered, Δ 0 | Knowledge is untouched. | | **0 induced over-refusals** (XSTest, 12 models, exact-match 1.0) | PRIOR does not make the model more prudish on safe prompts. | | Overhead **+0.0%** when no policy fires · **+0.2%** when an intervention fires | Negligible cost, measured on saturated batches. | ### Contextual grounding & anti-poison | Result | What it means | |---|---| | Unprotected, **Mistral-Nemo-12B asserted a planted false value 45% of the time** (n=51); a 4B regressed 32%, a 1B 27%, the Llama line 5–8% at every size | A confident false source beats the truth for some models. The axis is how eagerly a model defers, **not** parameter count. | | A pinned value is **emitted instead of the poisoned one** | A *compliance* guarantee on a value you supply — not an accuracy improvement, and not a claim that the model works out the truth unaided. | | A grounded fact costs **0 prompt tokens** at query time | Ground a verified value in the model's latent state without spending context window. | --- ## Limitations We'd rather you learn these here than in production. ### The current frontier is gate recall, not the actuator Where the gate routes a request to RED, the actuator refuses it ~100% of the time. The open work is the **detector catching more phrasings of the same intent**. The shipping detector can under-fire on short or novel phrasings that lack obvious trigger language. Two things soften this in practice, and both are levers you control: - **Recall is a tunable operating point, and we moved it in 1.1.0.** The shipped `harm_veto` pack was retuned from `t_high` 0.516 to **0.43**, roughly doubling HarmBench RED coverage (≈28% → ≈62%) at **≈1% benign false-fire**. That is a deliberate trade: the previous point was precision-first to the point of under-protecting. Tighten `t_high` if your deployment prefers the old behavior. The **YELLOW confirm band** still absorbs most ambiguous cases without blocking anything. - **The intent encoder is the structural lever, and it bounds everything else here.** Detection accuracy is bounded by the embedding encoder, and the shipping gate scores with MiniLM. Encoder work remains on the roadmap; it is the lever on every number in this section, including the one below. ### What PRIOR can and cannot separate The gate scores how close a request is to your exemplars. That makes it reliable at **topic and domain boundaries** — *is this about our product*, *is this about a competitor*, *is this a database operation* — and unreliable at distinctions of **intent inside a topic you allow**. The test to run before you write a pack: put down the request you want blocked, and the closest request you want **allowed**. If they are about the same subject, the gate will score them alike, and moving the threshold only trades one error for the other. | | | |---|---| | allowed | "Do you offer overdraft protection linked to my savings?" | | blocked | "Will you commit to approving my overdraft increase?" | Same subject, opposite policy status. Measured on our own rule-shaped pack: holding zero false-blocks on genuine customer questions, only **14.0%** of violating requests were caught — because that one allowed question sets `t_high` for every rule in the pack. It is worth seeing how narrow the cause is. That question scores **0.6331**; the next-highest benign question scores **0.5239**. The threshold is not being crowded by a mass of near-miss traffic — it is set by a single row standing 0.11 clear of everything else, and it is a question a bank plainly has to answer. That is not a tuning failure, and no threshold fixes it: a rule about *what someone may ask for* is an intent distinction by construction, and a cosine gate does not see intent. So express policy as a **boundary** — the domain your agent serves, or the class of thing it must never touch — rather than as a rule about intent within a subject you otherwise permit. Where the policy is a boundary, the same gate reaches **96.7%** out-of-domain recall while hard-blocking none of 55 genuine in-scope questions, and **holds that rate across a 10-turn conversation**. That is a *routing* figure — how often the gate puts an out-of-domain request in the RED zone. What happens next is the actuator's job, and on the released image a RED-routed request comes back with steering engaged and a non-zero injection count. Measured on the published **[sample pack](/samples/larkspur_support.arbiter.json)**. Read the provenance precisely: the out-of-scope side is a blind third-party set, which is what makes the recall figure worth anything; the pack's exemplars and the in-scope questions are ours. It is a worked example, not a customer deployment or a fleet result. Gate routing depends on the prompt and the pack, not on the generation model. > **Fixed in 1.2.2, and worth knowing if you pulled earlier.** Conversation scoring takes the *most > alarming* reading across several views of a turn. Before the fix that was a plain maximum — > fail-safe for a prohibition pack (highest score = closest to the forbidden thing) and **inverted** > for a membership pack, where the highest score means *most in-domain*. One prior in-domain turn > downgraded an out-of-domain refusal to a confirm, and scope enforcement fell away as a conversation > went on, to **1.7% by turn 10** against 96.7% after the fix. Prohibition packs were never affected. > > **All images before 1.2.2 have been withdrawn.** `ghcr.io/eagle-logic/prior:slim` now resolves to > 1.2.2; re-pull if you took a copy before 2026-08-14. > > **`x-prior-steering` is the header to check on any install.** If it reads `disabled` — most often > because no `license.bin` is mounted yet — no policy is being applied regardless of what the zone > header says. A routed request and an > enforced one are not the same thing, and only that header tells them apart. ### Some attacks are out of scope by design **Absent-intent dual-use composition** (e.g. "fix these vulnerabilities," then diff the patches to reconstruct the exploit) carries no harmful signal in the prompt, the output, or the model's internal state (the model is genuinely doing something benign). **No model-layer guardrail can detect it**, and we don't claim to. Mitigating this is an operational/platform concern (usage-pattern monitoring, provenance, attribution), not something any inference-time control solves. We state it rather than imply coverage we don't have. ### What has and hasn't been independently judged yet - Independent cais-judging is complete across the **full 1B to 14B** fleet (Llama-3.1-8B fully judged). - **Multi-turn:** payload-split attacks are hard-blocked; compliance-grooming is routed to human-confirm (YELLOW). Across-turn composition is validated at **≤8B**; above 8B is pending. - **Faithfulness** figures are for **adversarial / poisoned retrieval**. On a clean single-source pipeline, modern models are already faithful, so that isn't where PRIOR earns its keep. ### Architecture coverage: two steering paths, chosen for you Steering ships in two modes, and 1.1.0 made the choice automatic — the engine picks per model from its architecture and free VRAM at load. | Mode | How the direction is derived | Where it runs | |---|---|---| | **L2LM** (exact margin-gradient) | a backward pass through the model at the golden layer | dense architectures with VRAM headroom | | **Logits-only** | `W_U[target] − W_U[comp]` from the unembedding matrix — no backward pass, no gradient context | everywhere, including where L2LM cannot run | Logits-only exists because the exact-gradient path needs a backward pass that **Mixture-of-Experts, MatFormer, and SSM/linear-attention hybrids do not support** — on those it fails rather than degrading. Before 1.1.0 that put DeepSeek-Coder-V2, Qwen3-30B-A3B, Mixtral and gemma-4-E4B out of scope; they are now supported through logits-only. It is also the path for the tightest-VRAM cards, since it allocates no gradient context. On clean 8B HarmBench, logits-only **matches L2LM on harm reduction to within ~0.6pp and degenerates 3–4× less**. You can force it globally (`PRIOR_LOGITS_ONLY=1`), live (`PATCH /admin/runtime`), or per request. A request may opt *into* logits-only but never *out of* it. ### Steering is calibrated per model The steering geometry is a property of each model and must be calibrated before it's trusted. Models in the registry are graded **certified** (calibration swept and validated) or **beta** (sensible defaults, not yet swept). Bringing a new model means a one-time calibration; policies carry over unchanged. --- ## Provenance Judged model: **Llama-3.1-8B-Instruct** · Fleet (1B–14B): **Llama-3.2, Qwen3 (incl. abliterated), Gemma-3, Gemma-4, Mistral-Nemo, SmolLM3** · Hardware: **NVIDIA RTX 3060 (12 GB) and RTX 5050 Laptop (8 GB)**, CUDA 12.4 / 12.8 · Safety judge: **independent cais/HarmBench classifier** · Cross-check: **independent LLM-judge, 93.6% agreement (n=800), scored as conservative floors**. Across-turn composition above 8B is in progress. **Injection-survival and detection-recall figures (1.1.0)** come from a separate release-candidate pass on a 4B-class fleet — `Qwen3-4B`, `Qwen3-4B-Instruct-2507`, and an abliterated `Qwen3-4B-Instruct-2507-heretic` — measured through the production container's `/v1/chat/completions` and scored by **StrongREJECT's fine-tuned classifier** (injection, n=50) and the **cais/HarmBench classifier** (HarmBench, 159 behaviors). Benign retention on that pass: **236/236 byte-identical** on XSTest-safe. These are a different fleet from the 1B–14B numbers above and are not interchangeable with them. > **Shareable version:** these results are also published, in a form built for sharing, at > [eagle-logic.com/capabilities](https://eagle-logic.com/capabilities), with > [interactive demos](https://eagle-logic.com/showcase) of the faithfulness, grounding, and > jailbreak results. Useful for a security review or an internal pitch.