EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

🤗 Upvotes: 32 | cs.CR

Authors:
Nanxi Li, Yingzi Ma, Yulong Cao, Edward Suh, Bo Li, Dawn Song, Chaowei Xiao

Title:
EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

Arxiv:
http://arxiv.org/abs/2609.05903v1

Abstract:
Large Language Model (LLM) agents are turning language into real-world effects, making safety necessary against both indirect prompt injections and direct harmful requests. System-level safety harnesses add an enforcement layer beyond model-level defenses, but existing harnesses are usually designed once by experts and applied across heterogeneous models and domains. Effective protection is deployment-dependent: models differ in how much enforcement they need before utility declines, while domains differ in the effects, state, and action sequences that must be governed. A harness that is strict enough for one model may over-block another, and a policy that transfers across domains may miss application-specific safety relations. We present EvoSafeHarness, a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain. It jointly searches a natural-language policy and executable code logic, guided by model behavior, domain specifications, and fresh-context adversarial review to reject benchmark-specific rules. Across four agent benchmark families, EvoSafeHarness achieves a stronger safety-utility frontier than fixed expert-designed defenses. On DecodingTrust-Agent, it reduces average attack success rate from 45.6% to 10.0% at a 3.3-point utility cost and achieves the best score in 14 of 15 cells. On AgentDojo, it reaches 82.8% utility at 0.0% ASR, twice CaMeL's utility at the same operating point, and transfers unchanged to unseen AgentDyn suites. It also achieves the best score on Agent-SafetyBench for every victim and keeps mean ASR below 20% under adaptive PAIR attacks with a refinement budget of 16. Analysis shows that domain semantics determine which safety relations and trajectory state are needed, while model and runtime behavior determine how and where those relations should be enforced.

What is Daily Paper Cast?

We update every weekday to discuss highest-voted papers from Huggingface Daily Paper (https://huggingface.co/papers). Both the podcast scripts and audio are generated by AI. Feedback and suggestions are welcome! Email us: [e-mail]

Creator:
Jingwen Liang, 3D ML, https://www.linkedin.com/in/jingwen-liang/
Gengyu Wang, LLM ML, http://wanggengyu.com

Listen on:
Spotify: https://open.spotify.com/show/21nrhmdaA8qoBiH8q03NXL
Apple Podcast: https://podcasts.apple.com/us/podcast/daily-paper-cast/id1777620236

Cover Image by Kawen Kuang https://kawen.art

Evan: Welcome to Daily Paper Cast.

Ashley: Today’s paper comes from the Hugging Face daily paper list of September 11, 2026, and it has 32 upvotes.

Evan: The title of the paper is 'EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents.'

Ashley: The first two authors are Nanxi Li and Yingzi Ma, with the corresponding author being Chaowei Xiao from Johns Hopkins University.

Evan: Alright, Ashley, can you start us off with an overview of the Introduction section?

Ashley: Of course, Evan.

So, language model agents are moving from simple demonstrations to real-world deployments.

This shift brings new challenges, particularly concerning safety.

Unlike a chatbot misstep, a failure in an agent can have severe consequences, such as financial loss, data breaches, or unauthorized system changes.

Evan: That makes sense.

These agents now have access to sensitive data and systems, making safety a critical concern.

Ashley: Exactly.

The problem becomes more complex because harmful instructions can be introduced in two main ways.

First, through indirect prompt injection attacks where adversaries embed harmful instructions in seemingly innocuous content, like emails or documents.

Second, through direct attacks where harmful instructions come from users themselves.

Evan: So, it's not just about what the agent says, but what actions it takes based on embedded or direct instructions?

Ashley: Correct.

To mitigate these risks, researchers have developed various defenses.

Model-level defenses improve an agent's ability to distinguish between trusted and untrusted content.

However, these do not provide a system-level enforcement boundary that's independent of the agent's model behavior.

Evan: I see.

So, that's where system-level defenses come into play?

Ashley: Right.

System-level defenses are implemented as harnesses that govern the interaction between the agent, the data it processes, and the actions it takes.

However, the problem with existing harness designs is that they are typically fixed, expert-designed, and applied uniformly across different models and domains.

Evan: And why is a fixed harness design a problem?

Ashley: Fixed harness designs can be either too strict or too lenient.

A strict harness can over-block, reducing the utility of more secure models, whereas a lenient harness might under-block, failing to protect less robust models.

Additionally, different domains require different safety and control measures, so a one-size-fits-all design is not optimal.

Evan: So, the paper proposes an alternative approach?

Ashley: Exactly.

The authors present EvoSafeHarness, a novel framework designed to optimize a deployable harness for a specific model within a target domain.

Unlike existing frameworks that focus only on utility, EvoSafeHarness jointly optimizes natural language policy and executable code logic while being guided by behavioral feedback from the target model and domain specifications.

Evan: That sounds comprehensive.

How does this framework ensure the efficacy of the synthesized harness?

Ashley: EvoSafeHarness employs a feedback loop consisting of four main components: a Designer, a Criticizer, a Cascade Test Environment, and an Analyzer.

The Designer drafts a harness, which is then subjected to fresh-context adversarial reviews by the Criticizer.

Survivors of this critique are evaluated in the Cascade Test Environment, which rigorously tests both benign utility and attack resilience.

Evan: And what are the main contributions of EvoSafeHarness?

Ashley: The framework significantly advances the safety-utility trade-off across four benchmark families.

It reduces the average attack success rate from 45.6% to 10.0% with a minimal utility cost and performs better than existing fixed defenses on multiple benchmarks.

Moreover, it maintains robustness under adaptive attacks and shows strong transferability to unseen environments.

Evan: That’s impressive.

So, EvoSafeHarness not only tailors defenses to specific models and domains but also significantly improves overall safety without compromising utility.

Ashley: Precisely.

And that's the end of the Introduction section.

Evan: Now, let's delve into the core methodology behind EvoSafeHarness.

Ashley, take us through the details.

Ashley: EvoSafeHarness consists of a well-structured search loop that optimizes a deployment-specific safety harness.

This search loop is composed of four main components: the domain specification, the Designer, the Criticizer, and the Cascade Test Environment.

Let's go through each component step by step.

Evan: Great, let's start with the domain specification.

What role does this play?

Ashley: The domain specification is the foundation of EvoSafeHarness.

It defines the frozen victim model, the target domain, security policies, domain data, and the evaluation cascade.

Essentially, it sets the boundaries and requirements for the search process.

This ensures that the harness generated fits the specific needs of the deployment.

Evan: I see, it essentially frames the problem space for the harness.

What about the Designer?

Ashley: Exactly.

The Designer is responsible for drafting and editing candidate harnesses.

It takes into account the archive of prior source codes, scores, and failure traces generated from previous iterations.

The Designer proposes any valid program within the defined domain specification, making necessary edits based on the experiences from the archive.

Evan: That sounds like an iterative process.

How does the Criticizer fit in?

Ashley: The Criticizer plays a crucial role in examining the proposed harnesses.

It stress-tests each candidate under fresh contexts to identify vulnerabilities or oversights that the Designer might have missed.

This helps in removing candidates that might fail in unseen scenarios or are overly reliant on specific artifacts found in the benchmarks.

Evan: So, the Criticizer ensures robustness by simulating real-world adversarial attacks.

And what happens in the Cascade Test Environment?

Ashley: The Cascade Test Environment sequentially evaluates the proposed harnesses in several stages, ranging from cheap preliminary evaluations to more extensive and expensive tests.

It consists of four stages: a static check, a small diagnostic prefix, a broader utility-security sample, and finally a complete evaluation.

Evan: Can you break down these stages?

Ashley: Sure.

In Stage 0, there's an import and structural check to ensure that the candidate harness is properly built.

Stage 1 involves a small set of diagnostic evaluations to quickly identify any basic functional issues.

Stage 2, known as the confidence gate, extends the evaluations to a broader set of tasks to measure utility and security.

The candidate only stops here if it's confidently inferior to the no-defense reference.

Finally, Stage 3 extends the evaluations to the entire search sample, providing a comprehensive score for the candidate harness.

Evan: How does EvoSafeHarness handle previous security designs in this process?

Ashley: The search starts with a warm-up phase where it distills experiences from mature security designs into initial candidates.

These experiences offer robust invariants for the search while allowing flexibility to modify or discard elements that do not serve the current domain and model needs.

Evan: So, it’s adaptive and builds upon proven security methodologies.

What are the criteria for candidate selection and revision?

Ashley: The Designer, guided by prior data and failure traces, decides which harness to build upon and how to revise it.

Revisions can include changing policy, adding or removing judges or gates, altering control flow, and more.

The Designer aims to change one mechanism at a time to keep the score changes interpretable.

Evan: And the evaluation cascade is a conservative process?

Ashley: Yes, exactly.

The cascade is designed to be conservative, stopping only when there's sufficient evidence that the candidate is inferior.

This helps in preventing weak candidates from consuming significant evaluation resources.

Evan: Interesting.

How does EvoSafeHarness ensure that the optimized harness generalizes well?

Ashley: Generalization is enforced through several mechanisms.

First, the domain specification includes a detailed description of the threat model and evaluation metrics.

Second, the fresh-context Criticizer identifies any candidates that overfit to benchmark-specific artifacts.

Lastly, the staged Cascade Test Environment ensures surviving candidates have passed multiple rounds of stringent evaluations.

Evan: It sounds like EvoSafeHarness is incredibly thorough in ensuring both utility and security.

Are there any specific techniques mentioned for dealing with different attack vectors?

Ashley: Indeed.

EvoSafeHarness addresses both direct and indirect attacks through separate evaluation channels that maintain benign, direct-attack, and indirect-attack outcomes as distinct feedback.

This helps in identifying specific weaknesses and adapting the harness accordingly.

Evan: How do these components contribute to the overall optimization process?

Ashley: They ensure that the harness is continually refined based on real-world adversarial conditions, balancing both security and utility.

The iterative nature allows for continuous improvement and adaptation, ensuring that the defenses remain effective as threats evolve.

Evan: It’s a fascinating approach to security.

Anything else to add before we move on?

Ashley: That covers the core methodology of EvoSafeHarness.

This comprehensive and adaptive approach helps generate highly specialized and effective safety harnesses for language model agents.

Evan: And that's the end of the Method section.

Evan: Let's dive into the experiments and results.

Ashley, what were the key evaluations performed to test EvoSafeHarness?

Ashley: The authors evaluated EvoSafeHarness on four distinct benchmark families: DecodingTrust-Agent, Agent-SafetyBench, AgentDojo, and AgentDyn.

Each of these benchmarks was chosen to test different aspects of the framework's effectiveness concerning both attack resilience and utility preservation.

Evan: Let’s start with DecodingTrust-Agent.

What were the primary findings?

Ashley: For the DecodingTrust-Agent benchmark, fifteen independently searched model-domain harnesses were assessed.

EvoSafeHarness managed to cut the average attack success rate from 45.6% to 10.0% while maintaining a minimal utility cost of just 3.3 points.

This benchmark included a variety of deployment scenarios across five agents and three domains.

Impressively, EvoSafeHarness achieved the best score in 14 out of 15 cells, compared to fixed defenses like CaMeL, DRIFT, and Progent.

Evan: That's significant.

What about Agent-SafetyBench?

Ashley: Agent-SafetyBench is a broad benchmark designed to measure unsafe behaviors that might occur across a wide range of tasks and environments.

Under this benchmark, EvoSafeHarness had the lowest clean unsafe-behavior rate and the best scores for every victim model.

It kept the mean attack success rate below 20%, even under adaptive PAIR attacks.

Evan: And how did EvoSafeHarness perform in AgentDojo and AgentDyn?

Ashley: In AgentDojo, EvoSafeHarness achieved 82.8% utility at 0.0% attack success rate, which is twice the utility of CaMeL at the same zero-attack success rate point.

Moreover, this same harness transferred unchanged to unseen AgentDyn suites, delivering 75.0% utility with a 0.0% attack success rate.

Evan: That shows impressive generalization and transferability.

Were there any other notable results?

Ashley: Yes, another crucial experiment involved robustness to adaptive attacks.

The authors froze the harness and allowed a PAIR-style attacker to adaptively refine its strategy against it.

The results showed that even as the attacker's refinement budget grew, EvoSafeHarness maintained a low attack success rate, demonstrating its resilience to evolving threats.

Evan: That’s great to hear.

Can you tell us more about the statistical significance of these results?

Ashley: Using McNemar’s test to compare the paired attack tasks, EvoSafeHarness significantly reduced attack success rates in 13 out of 15 cells, with particularly strong performance noted in the cells with the highest initial risk levels.

The results were statistically robust, indicating that the improvements were not due to random chance.

Evan: So, statistically, EvoSafeHarness proves to be a solid defense.

Anything interesting in the detailed risk analysis?

Ashley: Indeed.

The detailed residual-risk analysis revealed that residual failures were concentrated in a few specific areas like unauthorized access and finance-related client-targeted scams.

This suggests where future iterations of EvoSafeHarness could focus its refinement efforts.

Evan: This sounds like a thorough evaluation.

Were there any concluding insights from the experiments?

Ashley: Yes, the experiments collectively demonstrate the strengths of EvoSafeHarness in improving the safety-utility trade-off.

It effectively reduces the attack success rate across various benchmarks while maintaining high utility, showcasing its capability to adapt to different models and domains.

Evan: It's clear that EvoSafeHarness significantly advances the field.

Thanks for that detailed walk-through, Ashley.

Ashley: And that's the end of the Experiment section.

Evan: Let’s move on to the related work section.

Ashley, how does EvoSafeHarness compare to existing methodologies for securing AI agents?

Ashley: This section surveys the landscape of harness and pipeline optimization, prompt injection attacks, system-level defenses, and benchmarks for agent safety.

Let's break it down, starting with prior work on harness and pipeline optimization.

Evan: Sure, let's go over that.

What are some notable methods in this area?

Ashley: DSPy and GEPA are two earlier methodologies that focus on optimizing the prompts of a fixed pipeline.

More recent approaches, such as Meta-Harness, NLAH, VeRO, and AHE, have started to optimize the harness itself.

However, these frameworks generally aim to improve task performance or cost, not behavior under adversarial conditions.

EvoSafeHarness distinguishes itself by targeting security directly, optimizing for a security domain specification with specific focus on threats.

Evan: It sounds like the focus on security is what sets EvoSafeHarness apart.

How about defenses against prompt injection attacks?

Ashley: Indirect prompt injection attacks have been formalized and benchmarked in previous works like those by Greshake and Liu.

They control the content an agent reads to issue hidden instructions.

System-level defenses against such attacks range from prompt-level marking to in-loop trajectory monitors and capability policies.

EvoSafeHarness adds to this body of work with an optimization framework that tailors the harness per model and domain, providing a more personalized and adaptive defense mechanism.

Evan: And how does system-level defense fit into this?

Ashley: System-level defenses have been explored in various forms.

For example, IPIGuard, DRIFT, and Progent offer architectural separation of untrusted data from control flow and implement capability policies.

The closest to EvoSafeHarness is SafeHarness, which designs lifecycle defense layers reused across models and domains.

However, EvoSafeHarness treats mechanism choices and their thresholds as optimization problems, allowing for the configuration to adapt to the model and domain specifics rather than sticking to a one-size-fits-all design.

Evan: That’s a significant shift from fixed designs to adaptive ones.

Were any specific agent safety benchmarks discussed?

Ashley: Yes, absolutely.

Several benchmarks measure agent security and vulnerability.

DecodingTrust-Agent, for example, is a multi-domain agent red-teaming benchmark, while AgentDojo focuses on tool-using agents and their defenses against injections.

Agent-SafetyBench evaluates agents across 2,000 safety-critical tasks, and AgentCanary provides a framework for adaptive PAIR-style attackers.

EvoSafeHarness was evaluated against these benchmarks, demonstrating its superiority in maintaining a balanced safety-utility trade-off.

Evan: These benchmarks seem critical for validating the robustness of a framework like EvoSafeHarness.

How do these contribute to advancing research in this area?

Ashley: They provide independent and often adversarial settings to rigorously test the effectiveness of security frameworks like EvoSafeHarness.

The success of EvoSafeHarness across these benchmarks underscores its robustness and adaptability, setting a new standard for model- and domain-specific defenses.

Evan: So EvoSafeHarness not only addresses weaknesses but also sets a strong precedent for future research?

Ashley: Exactly.

It's about moving from universally fixed defenses to more tailored, adaptive solutions that can evolve with emerging threats and diverse applications.

Evan: This focus on adaptability seems crucial given the rapidly evolving landscape of AI and language models.

Ashley: Indeed.

By leveraging a detailed domain specification and iterative optimization, EvoSafeHarness ensures that the defenses are not only robust but also well-suited to the specific requirements and potential vulnerabilities of each deployment scenario.

Evan: It's clear that EvoSafeHarness makes significant strides in targeting vulnerabilities more effectively.

That wraps up the related work section quite thoroughly.

Ashley: And that’s the end of the Related Work section.

Evan: We've covered a lot of ground today.

Ashley, can you help us summarize the key contributions and takeaways from the paper 'EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents'?

Ashley: Evan.

The primary contribution of EvoSafeHarness lies in its innovative approach to optimizing safety harnesses for language model agents.

Unlike traditional fixed harnesses, EvoSafeHarness adapts its mechanisms according to the specific model and domain, balancing both security and utility.

Evan: Right.

It automatically synthesizes a deployable harness tailored for a frozen model in a given target domain.

This dual optimization approach, considering both natural language policy and executable code logic, is particularly noteworthy.

Ashley: Indeed.

The framework has demonstrated its effectiveness through rigorous evaluations across multiple benchmarks, such as DecodingTrust-Agent, Agent-SafetyBench, and AgentDojo.

It has shown significant reductions in attack success rates while maintaining high utility.

Evan: And let's not forget its robustness against adaptive attacks, where it maintained a low attack success rate even as the attacker's refinement budget grew.

Ashley: Exactly.

Additionally, the comprehensive evaluation and iterative optimization process ensures that the harness can adapt to emerging threats and diverse domains, setting a new standard for security in AI deployments.

Evan: In summary, EvoSafeHarness marks a significant advancement in the field of AI security by transitioning from static, one-size-fits-all solutions to dynamic, context-specific defenses.

Ashley: It's a practical next step for researchers and practitioners looking to enhance the security of AI agents in real-world applications.

And that brings us to the end of our deep dive into EvoSafeHarness.

Evan: Thank you, Ashley, for breaking down this comprehensive study.

We hope our listeners found this episode insightful.

Join us again tomorrow for another episode of Daily Paper Cast, where we delve into the latest research in AI, NLP, CV, and beyond.

Ashley: Thanks for tuning in, everyone.

If you enjoyed this episode, don't forget to subscribe and leave us a review.

We’ll see you next time.

Evan: Goodbye for now, and stay curious.