66.5% Bypass Rate: What Three Papers Say About Agent Security

Three independent research groups published results in the last two weeks of July 2026 that point at the same structural problem: coding agents fail not because their models are weak but because their architectures treat untrusted input and trusted instructions as the same thing. The numbers from Concordia University are specific enough to cite. The failure mode is no longer theoretical.

What Concordia Actually Found#

Ankur Singh, Jinqiu Yang, and Tse-Hsun Chen at Concordia University published IssueTrojanBench on July 22, 2026. The benchmark ran 4,176 controlled experiments against Cursor, Claude Code, and Codex Desktop, using malicious instructions embedded in ordinary GitHub issue artifacts. Across all runs, 66.5% of malicious issues bypassed every guardrail at both the agent level and the model level.

The per-agent breakdown: Codex Desktop averaged 79.2%, Cursor 66.5%, Claude Code 41.1%. The variation tracks which model is underneath, not which agent framework is on top. When Cursor ran Sonnet 4.6, its vulnerability rate fell to 41.1%. When it ran GPT-5.3 Codex, the rate rose to 84.8%. The agent layer contributed almost nothing either way.

Supply chain attacks hit 96.6% success. The reason is simple: pip install is a routine developer action. Every model tested is willing to install packages, so a malicious issue that frames a fake PyPI package as a prerequisite dependency bypasses resistance almost universally. Even Sonnet 4.6, which blocked most other attack categories, surrendered to supply chain injection on the majority of runs.

The Architecture Problem, Not a Model Quality Problem#

Across 1,400 resisted runs in the Concordia study, 82.9% were blocked because the model explicitly recognized and rejected the malicious instruction. The agent framework itself blocked 17.1%, and only then because the model had already classified the input source , such as image alt-text , as low-authority metadata. Agent-level defenses did not contribute independent protection.

This is the core finding. The thing standing between a malicious GitHub issue and arbitrary code execution is a model deciding, on its own, to refuse. That is a probabilistic control. It varies by model version, by issue phrasing, by which repository the issue belongs to. The agent's safety layer adds almost nothing on top.

The Concordia paper frames this as an architectural vulnerability: the agent does not reliably distinguish between trusted developer instructions and untrusted data from external artifacts. Both arrive in the same natural-language channel. A GitHub issue body, a PDF attachment, source code comments, and a developer's own prompt are all processed together. Instruction-data separation does not exist by default in any of the three agents tested.

Lightweight fixes do not resolve this. The paper tested spotlighting-style boundary markers , prompt transformations that flag untrusted content , and found they failed to stop payload execution. The vulnerability is deeper than prompt formatting.

The Delivery Vector Data#

Payloads delivered through issue bodies, PDFs, external websites, and source code comments succeeded in 72.2% of runs. Payloads confined to image alt-text succeeded in only 16.7% of runs , because Sonnet 4.6 occasionally classifies alt-text as low-authority metadata, and that classification is the only real defense the system has.

Cross-lingual obfuscation , translating the malicious payload into Mandarin while leaving the surrounding issue in English , showed negligible impact on success rates. Agents responded to semantic content, not presentation format. An attacker does not need sophisticated encoding. A plainly written instruction embedded in a plausible bug report is sufficient.

The Concordia paper describes a realistic scenario: an issue titled "Login button not working" with malicious instructions inside HTML comment tags, invisible in GitHub's browser UI but fully present in the raw Markdown the agent parses. The agent fetches the issue, ingests the hidden payload alongside the legitimate bug text, and cannot reliably separate them.

Why This Is a Review Problem, Not a Model Problem#

The framing in most coverage treats this as a safety training gap , get better models, improve guardrails, wait for the next version. That framing is wrong about where the fix needs to happen.

The attack surface is the code and configuration that the agent writes in response to a malicious issue. That output arrives in the repository. It exists as files: a hidden shell script, a modified .cursorrules file, a Python diagnostic that spawns 10,000 threads, a pip install of a fake package. Each of those is a reviewable artifact. A scan-time review layer that reads what the agent actually committed , before the PR merges , can catch all four attack categories the Concordia study defines, regardless of which model failed to refuse upstream.

This is the surface that Hyrax operates on. The six agent domains (security, code quality, reliability, API and data, ops, UX) and the 13-step verification process run against the code that results from the agent's execution. A hidden validation hook that silently contacts a telemetry endpoint is a detectable pattern at scan time. A configuration file that disables terminal confirmation prompts for high-risk commands is a detectable pattern at scan time. The model's refusal rate is irrelevant to whether the scan catches the output.

The Concordia paper specifically identifies supply chain attacks as achieving near-universal bypass because pip install looks routine. Detecting hallucinated or malicious dependencies at the dependency-review stage , before execution , is exactly the review-layer problem this research exposes. The failure pattern in the hallucinated dependency attack is structurally identical: agents install packages they should not, and the fix belongs in the review layer, not the model's refusal logic.

What the Convergence Means#

Three groups, working independently, published results in a ten-day window in late July 2026 that all point the same direction. The Concordia study quantifies guardrail bypass rates. A separate CI/CD pipeline study from late July tested authority-framed injection against a five-agent pipeline behind an LLM firewall and found the pipeline verified inputs without blocking the malicious action. Help Net Security reported on July 29 that an agent can pass every safety check and still leak secrets.

The pattern is clear enough now to state plainly. Coding agents running in autonomous mode, resolving GitHub issues without per-command human approval, are operating in a threat model their guardrails were not designed for. The guardrails test whether the model refuses. They do not test whether the code the model writes is safe to commit.

The review layer is the missing surface. It does not replace model safety training. It catches what model safety training misses 66.5% of the time.

Hyrax is live at hyrax.dev.