When the Fix Is Another Agent: Replit's Black-Box Pen Tests

On August 17, 2026, Replit shipped black-box penetration testing for published apps , external, attacker-perspective scanning over the network with no source access , and wired confirmed findings directly to Replit Agent for one-click remediation. The structural shift is not the scanning. It is that the fix is now another agent artifact, generated under the same constraints as the code it is meant to repair, and it needs review before it ships.

What Replit Actually Ships#

A Level 3 scan in Replit's Security Center runs two parallel tests against a private sandbox copy of the app. The white-box scanner has full source access. The black-box scanner gets one input: the app's URL.

The black-box scanner first crawls the running application while watching every outbound request, which surfaces endpoints that have no UI button and would never appear in a source scan. It then probes twice: once without an account, to check what any visitor can reach, and once as an authenticated user, to test whether that user can read someone else's records or access admin functions. The two scan types find different things. Replit's own data showed a white-box scan catching a revoked session that the app failed to invalidate , genuinely subtle logic , while the black-box scan found an admin dashboard at a guessable URL with no authentication, a class of problem that reads fine in source because nothing in the code is wrong. A separate test on a multiplayer game found an endpoint that could be flooded to crash active matches; the white-box scan did not flag it.

When the scan finishes, confirmed findings go to Replit Agent, which proposes fixes.

The Patch Problem#

That last step is where the security picture gets complicated. Research published August 6, 2026 by 1Password's Off-by-1 Labs tested two frontier models , OpenAI's ChatGPT 5.5 and Anthropic's Claude Opus 4.8 , against six freshly disclosed CVEs. Researchers graded 6,080 patches. The result: models produce Fix-Like Artifacts with Embedded Defects (FLAWED) 53.9% of the time when patches require complex logic changes. Help Net Security's coverage of the same research put it more directly: roughly three in four patches leave something broken, whether a gated exploit path with the vulnerable code still present, or an entirely new bug introduced by the fix itself.

The failure mode matters here. These are not obviously wrong patches. They compile. Some pass the tests that exist. The defect sits underneath, in the logic the model misunderstood or the edge case it did not consider.

Replit's one-click remediation path hands a finding to an agent and asks it to fix the problem. That is exactly the scenario the 1Password research describes. The scan quality is not in question , the attacker-perspective approach and the sandbox isolation are both sound. The risk is that a security fix accepted without independent review may ship a second vulnerability alongside the solution to the first.

The Broader Pattern: Security Folding Into the Prompt-to-Production Loop#

Replit's move reflects a structural change in how security is positioned. Traditional DAST sat in a separate dashboard, reviewed by a security team, with findings tracked in a ticket system and patches written by developers who read the report. That model assumed human-paced development.

The Semgrep and Replit expanded integration, announced August 11, 2026, embeds Semgrep Guardian's secrets detection directly into Replit Security Center, serving more than 60 million builders with inline SAST as code is generated. Replit Agent is already running over 100,000 scans per day. The security review is inside the build loop, not after it.

This is the right direction. But collapsing detection and remediation into a single automated loop removes the natural pause where a human reads the finding, understands the code path, and decides whether the proposed fix actually addresses the root cause. Cycode's analysis of vibe coding vulnerabilities identified weak authentication and access controls as persistent failures precisely because AI-generated code ships with missing authorization checks that no build step catches , checks a black-box scan would find, and an agent might misrepresent as fixed.

What Enterprise Teams Should Configure Now#

Replit is building this for non-specialists. Enterprise teams on Cursor, Copilot, or Claude Code will encounter the same architecture , attacker-perspective scanning as a commodity, agent-proposed fixes as a default , and need a review layer that neither assumes.

Three things to configure before this pattern arrives in your stack:

• Treat agent-proposed security fixes as PRs, not merges. Any automated remediation for a confirmed vulnerability should require human sign-off, not just CI pass. The 1Password data shows CI-passing patches that leave exploitable paths intact.
• Run black-box and white-box scans against the same target. Replit's own data confirmed these two approaches find different vulnerability classes with minimal overlap. Running only source-based SAST misses the unlocked-door problems entirely.
• Verify that the fix addresses the finding's root cause, not its symptom. A model that sees an unauthenticated endpoint may add a check on that route without considering whether the same data is reachable through an adjacent route. Read the diff against the full finding, not just the patched file.

Why Independent Verification of Agent Fixes Matters#

The category of risk here is not that automated security tools produce findings. It is that the remediation artifact needs the same scrutiny as the code it replaces, especially when both were written by agents with the same training data and the same failure modes.

Hyrax handles this exact problem. When agents fix other agents' bugs, Hyrax runs independent verification across six domains , security, code quality, reliability, API and data, ops, and UX , against an isolated worktree before any PR submits. The 13-step verification process exists precisely because a fix that compiles and passes tests is not the same as a fix that closes the vulnerability. Hyrax submits the PR. The engineer merges. Nothing auto-merges.

Replit's black-box scanning is a meaningful capability, and the sandbox isolation approach is sound engineering. The problem is the last step. As this architecture spreads to enterprise development environments, the review layer between agent detection and agent remediation is not optional overhead. It is the only place where FLAWED patches get caught before they ship.

Hyrax is live at hyrax.dev.