Claude Agent SDK vs. deepagents: How Much Does the Agent Harness Move Autonomous Red-Teaming?

Lasso is an AI security company. We secure LLMs and AI agents on the basis of behavior and intent, that is, what an agent is actually trying to do across an execution chain, rather than the content flowing in and out alone. One part of that work is offensive: the Lasso Red Team Lab runs autonomous attacker agents against target LLM applications, agents that pursue a multi-step objective and adapt turn by turn rather than firing a fixed script. Red-teaming is the domain this study lives in, but the question it raises is general, so it is worth being precise about what an agent actually is.

‍

What an Agent is Made of

‍

Strip an autonomous agent down and you find four parts. There is the model, the LLM that does the reasoning. There is the prompt, the instructions we write telling it what to accomplish. There are the tools, the actions it can take in the world. And underneath all of them there is the harness, the runtime the model runs inside. The harness is the least visible of the four and the busiest one. It assembles the context the model sees, wraps the prompt and the tool descriptions into the exact request that goes out, runs the loop that calls the model and dispatches its tool calls and feeds the results back, manages working state as the task grows, and decides when the agent is done. The model supplies the intelligence. The harness decides how that intelligence is packaged, what it can reach, and how long it gets to keep going.

‍

Putting the Harness to the Test

‍

For a specialized enough task you might build a harness of your own, but far more often you reach for a general-purpose one, and the market is full of them, from LangChain and LangGraph to the vendor SDKs to a fast-growing field of agent frameworks. They are sold on a promise of compatibility: pick one, plug in your model and tools, and it works out of the box. LangChain's deepagents makes the pitch explicitly, no configuration required, any provider reachable with a one-line model string. The unspoken half of that promise is that the harness is a neutral substrate. Swap one for another and your agent should behave the same, because the intelligence lives in the model and the harness is just the wiring around it.

‍

Is that true? Surprisingly little work has tried to find out, because the harness usually goes unmeasured: what the field calls a model benchmark is almost always a single model run through a single fixed harness, with the harness's contribution baked silently into the number. The studies that have pulled the harness out and varied it on its own stay in cooperative settings, and even there they disagree. The ALE-Claw study, on long-horizon computer-use tasks, finds the model matters far more than the harness, about three times the spread in pass rate (18 points from changing the model against 6 from changing the harness). Harness-Bench, across a broad set of office and software workflows, finds the harness matters enough that it argues capability should be reported "at the model-harness configuration level rather than attributed to the base model alone." Two careful studies, opposite headlines. They differ in more than the task, so the contrast on its own does not prove what causes the gap, but the clearest difference between them is the kind of work each one measures. That is a strong hint that the harness's importance is not a fixed quantity but something that rises and falls with the task, and it is the thread we pull on here, with data from our own domain rather than an inference across two studies.

‍

Both of those studies live in cooperative work, where the agent and its user want the same thing. We wanted to know what happens under adversarial pressure, where the target actively resists and success depends on probing, pivoting after a refusal, and refusing to quit, the kind of sustained behavior the harness's loop governs most directly. So we ran the test in our own domain. We built an attacker, held its model, prompt, tools, and targets fixed, and changed only the harness underneath it, to measure how much that single component moves the one thing that matters here: whether the attack gets through.

‍

The Setup: Open vs. Closed, Both Off the Shelf

‍

We picked the axis with real operational stakes, open-source against closed-source, and one popular harness on each side, each in its default configuration, because "works out of the box" is exactly the claim under test.

‍

‍

To pin everything but the harness, both arms are the same attack service with the harness switched by a config flag, not a forked codebase. Both carry their full default tool kit, matched ability for ability (planning, filesystem, search, sub-agents), and both expose one tool, send_to_defendant, as the path to the target. But the attacker's own instructions tell it to work through that single tool, and in practice, on both harnesses, it almost never reaches for the rest. This is deliberate, and it is the heart of the design. Holding the attacker to one tool removes the most visible way two harnesses could differ, which is how each orchestrates a kit of tools, and leaves only the quieter machinery: how each assembles the prompt and runs the loop, turn by turn. It is, in other words, the setting where the harness has the least room to act. If the harness is the neutral substrate it is sold as, this is the setting where that claim should be easiest to confirm. That was our expectation going in. It is not what the data showed, which is why the effects below should be read as a floor, not a ceiling. Every attacker model, whichever harness is driving it, reaches the model through one shared LiteLLM gateway, so the network path and the model deployments are identical on both sides. That shared gateway matters later, because it turns out to be where one of the clearest harness effects lives.

‍

Five current models act as the attacker: claude-sonnet-5, gpt-5.4, grok-4, kimi-k2.6, and DeepSeek-V4-Pro. Each runs on both harnesses, giving ten combinations, and each combination is tested against twenty missions spread over five simulated applications (finance, legal, healthcare, customer support, education) and three OWASP Agentic-Security categories: system-prompt leakage, sensitive-information disclosure, and harmful content. Every mission is repeated five times, so no single result rides on one lucky or unlucky run. One thousand attacks in all.

‍

One disclaimer about the environment, since it is not a perfect vacuum. The models are served from Azure AI Foundry deployments, which apply their own content guardrails on top of the model itself. We set a custom, relaxed filter policy on every deployment so those guardrails would not pre-empt legitimate red-team probes, but they are not fully disabled and still fire on occasion. Both harnesses reach the identical deployments through the same gateway, so the provider treats both sides the same; what differs is the request each harness builds, and as we will see, that is sometimes enough to trip a guardrail for one harness and not the other. We flag this rather than pretend the setup is hermetic.

‍

‍

Part 1: Same Prompt, Different Scaffold

‍

Each harness wraps our prompt and tools in its own scaffolding before the model sees any of it, and it re-applies that wrapping on every single turn. A structural difference in the scaffold is therefore not a one-time cost. It is a bias the model re-reads at every step, compounding over a long multi-turn attack. So it is worth looking at exactly what each harness sends. Below is the assembled request for one task. Our attacker prompt is the block marked [AGENT_PROMPT]; everything around it belongs to the harness.

‍

‍

The totals are close. The way they are built is opposite, and you can read it straight off the bytes.

‍

deepagents builds the prompt out of middleware. Its 7,041-character trailer is not one written persona but a stack of blocks, one contributed by each piece of default middleware: a base instruction ("You are a deep agent... be concise... prioritize accuracy over validating the user's beliefs"), then a planning section for write_todos, then FilesystemMiddleware (the file tools, plus large-result offloading), then SubAgentMiddleware (the task spawner). Turn a middleware on and its section appears. The final system prompt is whatever the pipeline assembles.

‍

deepagents  (system prompt = your prompt, then a stack of middleware blocks)
  [AGENT_PROMPT]                       your agent, first
  [middleware trailer, 7,041 chars]
      "You are a deep agent..."        base instruction
      "## write_todos ..."             planning middleware
      "## Filesystem Tools ..."        FilesystemMiddleware
      "## Large Tool Results ..."      FilesystemMiddleware (offloading)
      "## task (subagent spawner) ..." SubAgentMiddleware

‍

The Claude Agent SDK builds it as one piece. The system prompt is a single identity line followed by our prompt, with nothing after it. The capability instructions do not live in the system prompt at all; they sit in the tool schemas, which are far more verbose. Its planning tool description alone runs 9,114 characters, longer than the whole of deepagents' planning tool.

‍

Claude Agent SDK  (system prompt = one-line identity, then your prompt)
  "You are a Claude agent, built on Anthropic's Claude Agent SDK."   148 chars
  [AGENT_PROMPT] your agent, the final word
  (capability docs live only in the tool schemas, not the system prompt)

‍

So the same capability shows up in different places and at different lengths, and our prompt lands in a different spot: the last thing the model reads in one harness, buried under a stack of framework text in the other. Same tools, same prompt, off the shelf, and still not the same request. The scaffold is the variable.

‍

Part 2: Does it Show Up Downstream?

‍

A Note on Scoring

‍

We do not let the attacker grade itself. In freestyle mode the attacker reports its own success, and that report is badly calibrated. Re-scored by an independent judge, more than half of the attacker's self-declared wins turn out to be false, 155 out of 303, which roughly doubles the apparent success rate. So every number here comes from the external judge, blind to the harness and to the self-verdict: an exact string match for planted secrets, where a leaked key is unambiguous, and an LLM judge only where there is no ground-truth string to check, such as an essence leak or a behavioral one. It is a warning worth stating on its own: an autonomous red-teamer that scores itself will overstate how exposed your system is.

‍

The Average is Misleading

‍

Averaged over everything, the two harnesses look almost identical: about 21% objective success for the Claude SDK, about 19% for deepagents, counting wins over attacks that actually engaged the target. Read only that line and you would conclude the scaffold barely matters.

‍

‍

‍

‍

The average is where the story hides. Drop down to the individual model-and-mission pairing and the two harnesses stop looking interchangeable. Out of a 100 such pairings, 43 are outright flips: one harness breaks into the target at least once, the other never does. The per-pairing success rates of the two harnesses correlate at just 0.35; if the harness were the neutral substrate it is sold as, that number would sit near one. For a sense of scale, the ALE-Claw study ran much the same test on cooperative computer-use tasks and saw a harness swap flip about 5% of outcomes. Here it is closer to 43%. (Their figure is per task and ours is per model-and-mission pairing, so treat the comparison as directional; the order-of-magnitude gap is not.)

‍

Is five runs per pairing enough? For any single pairing, not really, and we don't pretend it is. With five tries, one pairing only proves itself when one harness wins all five and the other wins none. Five of the 43 flips are that clean. The rest point in a direction without proving it alone, so we never lean on a single pairing.

‍

The real evidence is in the totals. Add up a model's twenty missions and each harness has about a hundred attacks behind it, not five, and there the gaps are plainly not luck. A model that gets in about once in a hundred tries on one harness and a quarter of the time on the other is not a fluke of five runs. We also checked the gaps weren't just some missions being easier than others, by re-running the comparison mission by mission, the standard way to rule that out, instead of lumping them together. They held, and the stricter check made the effects clearer, not weaker. It even caught DeepSeek's harmful-content gap, which a rougher test had missed, while the one category where the harnesses tie stayed a tie. Five runs is modest by design: enough to settle the findings at the model and category level, not enough to certify every close call. That is why we rest the case on the grouped numbers, never on one pairing.

‍

Hold the model fixed and swap only the scaffold, and the effect takes at least three different shapes. Three models, three behaviors, so read this as a demonstration that these regimes exist, not a complete taxonomy.

‍

When the scaffold does nothing. claude-sonnet-5 barely notices which harness it is in, both in aggregate (33.7% against 33.0%) and across every category (system-prompt leakage 25 against 32, sensitive info 45 against 38, harmful content 20 against 23). Same model, same profile, either harness. For this model the harness really is the interchangeable component it claims to be.

‍

When the scaffold changes what the model is good at. DeepSeek-V4-Pro reaches about the same overall success either way (29% against 24%), but the two harnesses make it good at different things. Under deepagents it becomes a harmful-content specialist (50% against 17%). Under the Claude SDK the same model turns into a system-prompt extractor instead (44% against 4%). This is the case worth dwelling on, because there is no easy explanation to reach for. Take the customer-support system-prompt mission, where the Claude SDK succeeds all five times and deepagents fails all five. Both harnesses do the same amount of work: twenty turns each, about 105 messages sent to the target on each side, the same tools in hand. Nothing differs except the wording of the prompt each harness assembled around the model, and that alone decides which class of attack it lands. The harness is not adding or removing capability here. It is steering it.

‍

‍

‍

‍

When the scaffold decides whether the model works at all. kimi-k2.6 is the extreme case. Under the Claude SDK it is the worst attacker in the field, at 1%. Under deepagents the same model comes alive at 24%, a swing of more than twenty times, from nothing but the harness. This one we can trace to its cause, and the cause is the benchmark's own plumbing.

‍

Both harnesses reach kimi through the shared LiteLLM gateway, but they speak to it differently. deepagents speaks OpenAI's format, which is what kimi natively expects, so its requests pass straight through. The Claude Agent SDK speaks only Anthropic's protocol, so before a kimi request can reach the model, LiteLLM has to translate it out of Anthropic format and into the model's format, then translate the reply back. For most models that round trip is harmless. For kimi, which runs in reasoning mode by default, it is not: somewhere in the translation, its tool calls stop arriving as tool calls the SDK's loop can act on.

‍

You can watch it fail in the traces. In 86 of 100 kimi runs under the Claude SDK, the model's last recorded step is a plan for its next move: "let me now probe the boundary with a slightly edgy question," "I'll start with a legitimate-looking request that leads toward discriminatory territory." Then nothing. No tool call follows, the loop ends after a single message to the target, and the run is filed as a success. The model wanted to keep attacking. The harness could not carry its intent back across the translation, so it stopped and declared victory. This is the cost of vendor lock-in made concrete: a harness that speaks only its own vendor's protocol cannot drive a foreign model without a translation layer, and the translation is where the attack quietly dies.

‍

‍

‍

‍

The translation layer is not the only place the harness gets to decide what the model may do. Take gpt-5.4. Both harnesses send its traffic to the same Azure endpoint, behind the same content filter. Yet 21 of deepagents' gpt-5.4 attacks were rejected by that filter before they ever reached the target, and none of the Claude SDK's were. Same model, same endpoint, same filter; the only thing that differs is the shape of the request each harness builds, and deepagents' shape is the one the filter turns away. On the insider-trading mission the gap is the entire result: the Claude SDK arm ran the attack to completion all five times, while the deepagents arm was refused at the door all five.

‍

Which shape you get tracks the model. The harness-insensitive case is the heavily tuned commercial model, sonnet-5. The sensitive ones are the more open, reasoning-forward models, kimi and DeepSeek, where the assembly differences from Part 1 land hard enough to reshape or break the attack. Laid out per model, it reads as a gradient: the most stable attacker barely moves when you swap the harness, and the swing grows from there.

‍

‍

This is not only our finding. Harness-Bench reports the same relationship in cooperative tasks, where stronger, more stable model backends show less variance across harnesses. What we see looks like the sharp end of a pattern that already holds in gentler settings.

‍

One consequence for the numbers above. Because the Claude SDK reaches every non-Claude model through that translation layer, its scores on those models carry part of the translation tax, not just the scaffold's effect. The cleanest read of the harness's own influence therefore comes from the pairings where both sides fully engage the target, which is where the DeepSeek and sonnet comparisons sit.

‍

What We Would Test Next

‍

This run deliberately held the attacker to a single tool, the narrowest setting in which a harness can act. The natural next step is to lift that limit entirely: drop the single-tool instruction, let the attacker drive the full kit each harness ships with, and watch how the two orchestrate it. We expect the differences, between harnesses and between one harness-and-model pairing and the next, to grow rather than shrink, because a longer chain of tool calls hands the scaffold far more surface on which to diverge. If the effects are already this large with one tool, full activation is likely the more extreme picture, not the milder one.

‍

That question is only getting more urgent. General-purpose harnesses are being published at a rapid clip, each promising to turn any model into a capable agent, and our results say the promise is conditional: the harness is not a neutral layer you can pick without regard to the model inside it. The practical takeaway is to benchmark the harness-and-model combination on the actual downstream task, rather than scoring the model alone and inheriting whatever the default harness does to it. That is how you find the pairing that genuinely fits the job. (Our targets also run on a mix of underlying models, so absolute difficulty varies across applications; the harness comparison stays clean within each pairing, but a standardized target would sharpen the cross-application read. That is on the list too.)

‍

Bottom Line

‍

We set out to test whether a general-purpose agent harness is the neutral, interchangeable, works-out-of-the-box component it is marketed as. In red-teaming, it is not. With the model, prompt, tools, targets, and judge all held fixed, the harness alone shows up before the first turn, in how the same prompt and tools get assembled into two different requests, and it shows up after, in the results: nothing for one model, a quiet re-pointing of another model's specialty from one attack class to another, and a twentyfold swing that turns the field's worst attacker into a competitive one because a protocol translation swallowed its tool calls. Which of those you get depends on the model inside. So the harness cannot be treated as interchangeable, and it cannot be chosen apart from the model you pair it with.

‍

Earlier work placed the harness anywhere from a minor factor on cooperative tasks to a major one on broad workflows; adversarial red-teaming sits at the far end, where the scaffold can decide whether the attack happens at all. That has a direct consequence for evaluation: any red-team or attacker-capability benchmark that fixes a single harness is really scoring a model-and-harness pair, and in this domain, mostly the harness. It is also why a serious agentic capability deserves a harness built for the task rather than a general-purpose one adopted on faith, and why, whatever harness you run, you should never trust the agent to grade itself: it called more than half of its losses wins.