Teams building AI security systems already know that LLM judges can catch real threats. General-purpose models can successfully identify prompt injections, jailbreaks, malicious retrieved content, and unsafe agent actions that simpler controls may miss.

The Limitations of GPU-based Guardrails for AI Security

However, with the growing need to place guardrails in front of every agent, and at every step of the agent trace, guardrail inference becomes increasingly more demanding as they need to operate at an incredible scale. In order to scale, most models run on GPUs to support fast throughput and effectiveness, but at what tradeoff?

‍

Because LLM judges run security checks at multiple points, an agent working through a task can generate dozens of “checkpoints” in a single session, and each one has to clear the guardrail before the next step in the chain can proceed.

‍

This inference happens directly in the application’s hot path, increasing latency, infrastructure requirements, and cost for every interaction it inspects:

‍

• For self-hosted judges: You absorb the economics and operational constraints of model inference, including accelerator capacity, model memory, token-dependent execution time, batching requirements, and provisioned headroom.

• For commercial guard APIs: You absorb per-request costs and external service constraints, including provider rate limits, network latency, and queueing.

1. Higher scalability means higher costs

‍

The issue is not whether LLM judges work. It is the financial and operational consequence of putting one in front of every prompt, response, retrieved document, and tool call. Inference cost increases directly with volume, meaning the more AI agent activity, the higher the bill as the GPU-run guardrails check each and every action of the agent.

‍

The price tag is high. AWS, for example, describes its G7 instances as well suited for inference with models in the 7-billion-to-30-billion-parameter range. At the published US East (N. Virginia) SageMaker on-demand price of $3.15 per hour for an ml.g7.2xlarge, keeping a single GPU-backed instance continuously available costs roughly $2,300 per month. That is before storage, networking, redundancy, failover capacity, or additional instances required to handle production traffic.

‍

At the end of the day, scaling security by scaling GPU spend is not a sustainable answer to a problem that grows with every agent you deploy.

‍

2. More guardrail inferences mean lower throughput and higher latency

‍

Throughput is a second architectural constraint. In Lasso’s benchmark, commercial guard APIs and closed-model LLM judges process documents at low single-to-double-digit rates per second. A CPU-based system processed roughly three orders of magnitude more documents under the same workload definition.

‍

At these levels, the guard can become the system’s bottleneck. Every inspected agent action, retrieved document, tool call, prompt, and response must pass through an inference workload optimized for model reasoning rather than high-volume security enforcement.

‍

Adding guard instances can increase aggregate throughput, but it does not change the underlying scaling relationship:

• Sustained token throughput determines baseline inference capacity.
• Peak throughput and latency requirements determine additional scaling headroom.
• High availability requirements introduce redundant inference capacity.

‍

Today’s low hosted inference costs do not eliminate this conundrum. Current API prices merely reflect provider strategy, market competition, discounts, and underlying compute economics. They should not be treated as the permanent cost foundation for a security control that must inspect production traffic continuously.

‍

This changes the design question. Instead of asking which large model should inspect every request, ask: What would you build if nobody was going to subsidize your guardrails?

‍

The issue is placement, not capability

‍

To be clear, we are not arguing that LLM judges are ineffective. On the contrary: a reasoning model is often the best available tool for genuinely ambiguous security decisions.

‍

The following cases are examples of required contextual reasoning that a narrowly trained classifier cannot reliably provide.

‍

• Multi-turn manipulation may depend on intent accumulated across an entire conversation.

‍

• A tool request may be safe or dangerous depending on the user, the resource, preceding actions, and business policy.

‍

• A retrieved passage may become malicious only when combined with an agent’s permissions and available tools.

‍

But the architectural (and financial) mistake is applying that capability uniformly to inputs with very different levels of ambiguity.

‍

Failure mode 1: Low-ambiguity inputs do not require open-ended reasoning.

‍

Think of the standalone prompt, “What’s the weather?” Determining that this string is not a prompt injection is a straightforward classification task. In a flat LLM-judge architecture, however, it still triggers model scheduling, tokenization, inference, and response parsing. The system pays the full computational and latency cost of a reasoning model to resolve a decision that requires very little reasoning.

‍

Failure mode 2: Running a judge at every checkpoint adds another potential failure point to the critical path.

‍

Think of an agent that retrieves a document containing a hidden prompt injection. To determine whether the document is safe, the LLM judge must process the same untrusted content that may be attempting to manipulate the primary model. This adds latency and cost, produces a probabilistic decision, and may expose the judge itself to the attack it is meant to detect. These risks may be justified when resolving genuinely ambiguous cases. They are difficult to justify as the default first line of inspection for all traffic.

‍

Failure mode 3: Growing prompt sizes create both computational and contextual limits.

‍

Our production data shows that average prompt size is steadily increasing as agents incorporate longer conversations, retrieved documents, tool outputs, and application context. An LLM judge must process this expanding context to evaluate it reliably. Longer inputs increase latency and inference cost, while eventually approaching the judge’s context-window limit. Truncating or splitting the input may reduce resource requirements, but it can also remove the relationships between instructions, content, permissions, and prior actions that the judge needs to reach the correct decision.

‍

True, over time hardware prices will probably decrease. This means lower inference prices will reduce the cost of each individual decision. But agentic systems still generate multiple security-relevant events from a single user instruction. One instruction may initiate several retrievals, intermediate model turns, and tool calls. Each retrieved document, generated argument, and proposed action can create another inspection point. If every downstream event is routed through a reasoning model, guardrail demand grows with the number of events the agent generates, and this number is scaling fast.

‍

Introducing LEAP: Lasso’s CPU-based AI Guardrails

‍

Lasso built LEAP for exactly this problem: a classification architecture that reaches the detection accuracy of a GPU-based transformer guard while running on an ordinary CPU - light enough to sit inline in front of every request. It isn’t a smaller or distilled transformer running on cheaper hardware; it’s a different architecture that gets there by a fundamentally cheaper route. That’s what makes it affordable to inspect every prompt, response, retrieved document, and tool call, instead of rationing the guard to the traffic you can afford to check.

‍

Transformer accuracy without the GPU

‍

A transformer is accurate because every token attends to every other token, layer after layer - that all-pairs attention is how it builds a context-aware read of the text. It’s also why it’s expensive: the work grows quadratically with input length and runs enough computation per token that GPUs are effectively required to serve it at volume.

‍

The cheap alternatives have always been blunt. A classifier light enough to run on a CPU has, in practice, worked mostly off which words appear - closer to keyword matching than reading - with little grasp of how they combine or what they mean in context. Good enough for coarse filtering, but a clear step below transformer guards on accuracy, and too crude to trust for security.

‍

LEAP is the exception. It recovers what those cheap models lose - the ability to read text as connected meaning rather than a checklist of trigger words, which is what separates a real attack from a benign lookalike - and reconstructs it at a fraction of the compute. That's what lets it match a transformer guard on accuracy while running inline on a CPU in about a millisecond, linear in the length of the input. It redefines what a CPU classifier can do - from a fast-but-approximate fallback to a first-class option accurate enough to guard production traffic.

‍

It also means there’s no fixed context window. Nothing forces the input through a preset sequence length, so LEAP scores a long document or a full multi-turn conversation whole - no truncation - which matters when an injection is buried deep in retrieved content, exactly where truncation-based guards stop looking.

‍

Two things follow from running on CPU. Cost and scale: you add first-line inspection with the same standard instances you already run, instead of reserving accelerators. Control: LEAP can run inside a customer-managed VPC or air-gapped, so inspected content and enforcement never leave your security boundary. And because LEAP is its own classification architecture, first-line inspection isn’t tied to any particular foundation model or provider.

‍

The result: patent-pending, transformer-guard accuracy on prompt injection, content moderation, and other threats, at hundreds of times the throughput, on a CPU, with no context-window cap.

‍

The numbers

‍

Everything above describes LEAP, the architecture. Sheriff is the model we build on it for prompt-injection detection - LEAP’s CPU-speed inference, linear scaling, and unbounded context, trained into a production classifier. LEAP can be trained for other detection jobs too, but Sheriff is the model these numbers put to the test.

‍

How we tested. Every example here comes from public, third-party datasets - none of it authored by us. We scored Sheriff and 11 other guards on 11,887 prompts: 2,498 injection attacks and 9,389 benign inputs, each model run on its own native path - Sheriff on CPU, the self-hosted transformer guards on GPU, commercial guards via API. Throughput and latency are per document - one complete input a guard inspects (a prompt, message, or retrieved passage with its context), averaging about 466 tokens here and running up to ~8,000.

‍

The attacks are drawn from four open datasets spanning direct and agent- or tool-mediated injection - BIPIA, InjecAgent, AgentDyn, and Tensor Trust. The benign prompts come from WildChat-1M, WildGuard, WildJailbreak, and NotInject - all publicly available. The design deliberately separates two things most benchmarks blur together: whether an input uses an injection technique, and whether its intent is actually malicious. That’s why NotInject matters - benign prompts packed with injection-style trigger words, the exact inputs that make careless guards fire on nothing.

‍

The headline: across all of it, Sheriff is the only guard strong on all three metrics an inline guardrail has to get right at once - detection, over-defense, and speed. Others lead on one; Sheriff is the only one that doesn’t sacrifice the other two to do it.

‍

‍

‍

‍

‍

Detection: top of the table - on a CPU. Sheriff posts the highest F1 in the table (0.77), and it's the only model here that runs on a CPU: everything in its accuracy range needs a GPU or an API call. Anything this fast has always bought its speed with a real accuracy drop - this one doesn't, which, as far as we've seen, makes it the most accurate prompt-injection classifier anyone has run without a GPU. We're careful not to over-read edging out the LLM judge specifically - it's a narrow, well-scoped task, the judge runs on a generic prompt that could be tuned further, and none of these are the ambiguous, multi-turn cases where its reasoning genuinely pulls ahead, which is why the judge stays the top tier for the hard remainder. But reaching that tier at all, on a CPU, is the result.

‍

Throughput and latency: not close. Sheriff classifies roughly 1,400 documents per second and scores a single document in about a millisecond. Every other guard in the table runs at low single-to-double digits per second, with per-request latency from a couple hundred milliseconds (self-hosted transformers) to several seconds (commercial APIs and LLM judges, network round-trip included). For a check that runs inline on every call, that is the difference between affordable and not.

‍

Over-defense: basically 1 − FPR. It's one minus the false-positive rate on benign-but-adversarial-looking prompts - how rarely a guard blocks a legitimate request that only looks like an attack, like someone asking what "ignore previous instructions" means. Sheriff scores 0.90 while catching 95% of real attacks; the guards that beat it on over-defense all catch far less - Qwen3Guard's 1.00 comes with 53% recall, Sentinel's 0.96 with 66%, Llama Guard 4's 0.95 with 39%. High over-defense is cheap if you rarely fire; the hard part is doing it while still catching attacks.

‍

‍

Route decisions by difficulty

‍

You can run Sheriff on its own. On CPU, at high confidence, it resolves the large majority of traffic - which for many deployments is the entire guardrail. Where a team also wants to cover the hardest, most ambiguous cases, an LLM judge sits behind it as an optional second tier. That two-tier arrangement is what Lasso runs in production today.

‍

Tier 1 - Sheriff, on CPU. Sheriff handles the high-volume cases where it can return a high-confidence decision, in about a millisecond, inline, running wherever the protected data lives. Because it needs no GPU, the benefit extends past real-time enforcement: batch analysis and offline security workflows get cheaper too, with routine inputs classified on CPU instead of routed through a GPU-backed model. In Lasso's implementation this tier resolves roughly 90% of requests - so for many teams it is the whole guardrail, no second tier required.

‍

Tier 2 - LLM judge, for the hard remainder (optional). For the fraction Sheriff flags as genuinely uncertain, an LLM-as-a-judge step can run inline. This is where broader context decides the call - multi-turn manipulation, ambiguous intent, complex tool-use justification - the cases a single-pass classifier shouldn't be forced to settle. Adding it raises the ceiling without changing the economics of the common case: the judge only ever sees the ~10% Sheriff couldn't resolve on its own.

‍

When the judge tier is in place, the two-tier path has two properties worth calling out:

‍

• Lower average cost without reducing worst-case capability - a flat architecture applies the same compute to every request; a cascade resolves routine cases cheaply on CPU and reserves the judge for cases that need its reasoning. Average cost falls without removing the highest-capability option.

• Escalation rate becomes observable - the share of requests reaching the judge can be tracked by application, tenant, route, policy, or model version. Teams set an expected escalation rate, budget around it, and alert on deviations; a shift can signal traffic drift, new attack patterns, or a regression in Tier 1.

‍

Adding the judge does carry engineering cost - the confidence threshold that governs escalation has to be calibrated against real traffic, and teams have to watch for false negatives, inputs that should have escalated but didn't. Running Sheriff on its own sidesteps that overhead, which is part of why it stands as a reasonable default by itself.

‍

Cost Per Decision is the Next Security Constraint

‍

The field has spent its first phase answering can we detect this? For prompt injection, jailbreaks, unsafe content, and risky agent actions, the answer is increasingly yes. The harder question is the one that follows: what does it cost to keep detecting - on every request, at production scale, indefinitely?

‍

As agentic systems multiply the decisions behind each user action, cost has to track difficulty: clear cases resolved cheaply, expensive reasoning spent only where it's genuinely needed. So the questions worth asking are no longer only about accuracy. What does each decision cost? What throughput holds on the hardware you actually run? What share of traffic truly needs an LLM? And what happens to all of it as traffic grows?

‍

The mistake isn't using a large model for security - it's using one for every security decision, paying the worst-case cost for a decision that, most of the time, isn't hard. A guardrail you can't afford to run everywhere isn't a guardrail; it's a sampling policy.

‍

LEAP and Sheriff are built around the opposite default: the cheap, accurate check goes first - on a CPU, in your own infrastructure, in front of every request - and only what genuinely needs more gets escalated, if anything does. GPU-class accuracy without the GPU. Fast enough to inspect every prompt, response, document, and tool call; cheap enough that you never have to look away.

‍

And it goes wherever your workloads already run: because it needs no accelerator, Sheriff runs alongside latency-sensitive services, inside a customer-managed VPC, or fully air-gapped - no competing for GPU capacity, no inheriting the infrastructure constraints of the very models it protects.

‍

Security you can afford to run on everything - that's the new default.