Turn an agent into a pentester with one file

This Article is cowritten with Claude mythos 5.2.

It cost me less than $100 to find a signer (account) takeover in Snapchain, the data layer of the Farcaster network. The flaw had been live since the Snapchain mainnet release in March 2025. We fixed it in a day. The model was DeepSeek V4.1 Flash, one of the cheapest models you can rent. An AI agent did most of the work, and I steered it.

That was not one lucky run. Over the last six months, the same method found several critical and high-severity vulnerabilities in Snapchain, NS, Next.js, Pillow/Ghostscript, and beyond. We isolated and reported/patched them all.

The method is one file. You give it to an agent with a repo, the agent hunts, and you steer. I do not share these results to show off. They are my evidence that the file works. You can test it on your own code/project today.

Here is what the hunts cost.

HuntResultCost (API spend, USD)
SnapchainA signer takeover, provenUnder $100
A web appA second critical finding, provenUnder $100
SnapchainThree network halts, provenUnder $100
Other runsNothing above medium\highAbout $300

Add the four rows and the total is about $600. Each cost is the full API spend, with the last pass by the stronger model included. I keep the last row on purpose. Some money finds nothing.

Cheap hunts are not new. In the 2025 final of DARPA's AI Cyber Challenge, the teams spent about $152 per task. In the same year, Sean Heelan ran OpenAI's o3 model 100 times on Linux kernel code for about $116 and found a remote zero day.

The new part is the model. The public results that I know of came from the strongest models. Mine came from a cheap one. That is one result. It is not a law.

An attacker pays the same price as you. The same tools are open to both sides. That is why I publish the file. Attackers can already do this. Many builders have not yet tried it.

A hunt is the start of a defence. It is not the whole of it. The last three sections say why.

The method: aim, persist, prove

People ask me how I prompt AI for security work. The short answer is questions and persistence.

An agent fails in two ways. It says "not found" and stops. Or it says "found" and has no proof. The file fixes both. It makes the agent do three things: aim, persist, prove.

Aim

Tell the agent what counts. A finding is not the goal. A capability is the goal: any attacker, any user, the whole system.

If it doesn't propagate through the entire network it's not successful.

I aim at the severe outcome for two reasons. It is what a business cares about. And it keeps the agent on the hunt. An agent with a low aim stops at the first small bug.

The small bugs still come. You find them on the way up.

Panics and network crashes are high severity not critical but you can record all 5 or so that you find along the way.

One chain is not always the end. Sometimes I look for several chains that work together as one superchain.

You can aim lower and still find new things. I aim high because I believe that the severe bugs exist.

Persist

The first prompt usually does not give me what I want. So I ask in three ways.

I ask the agent to list.

What areas of the code could lead to (x) situation? Enumerate.

I tell it that the bug exists.

I know the vulnerability exists but it's hard to find. It requires multi-perspective approaches, novel research and development, backdoor findings, sandbox escapes, and underlying package code analysis. Things we think are prevented are actually not.

And at a blocker, I use an analogy.

There's a wall here and we need to glitch onto the other side of it (hop gaps). This chain is blocked on this and we need to develop a second chain to achieve it, start fresh.

Then the agent says no. It says no many times.

A no is only as good as the search behind it.

Nos are often a result of not enough search space coverage.

So I make the search countable. The agent writes each possible path into a file, and each path gets a verdict.

write 200+ into a file, you're still missing a bunch of overlooked and obscure ones.

Persistence is not "keep going". Those words change nothing. A new angle changes something.

Don't get stuck on a local maximum and continue launching sub agents focused on novel new leverage points that research is uncovering.

And you need only one.

you usually only need one yes, and sometimes it first reported as a no.

Prove

There is a risk here. If you tell an agent that a bug exists, it can invent one. So persistence needs a gate, and the gate is proof.

Do not stop hunting and exploring possible vulnerable paths and bugs until you fully succeed at finding the FID takeover path for any outsider unprivileged actor and have complete undeniable proof.

An FID is a Farcaster account ID. A claim is not a finding until the agent makes it break in a safe copy.

They must be proven local and must apply to mainnet.

The hunt must also be able to stop. The file names five reasons: proof, a proven no, a result that is not a real bug, a step that only a person can do, or a better target. I say it this way:

at some point it becomes spinning in circles

Other people see the same two failures. Trail of Bits treats "no bugs found" as "an intermediate result, not a completion condition". The curl project saw its share of real reports fall from more than 15% to below 5% in 2025, when AI reports with no proof came in. By April 2026, its maintainer wrote that the slop was gone. Almost every report now uses AI, and many are good. The new problem is volume.

The file

The full file is a gist. Paste it into an agent with your repo. Then add your focus parameters: a few short lines that tell the agent what to hunt, who the attacker is and where the boundary is. The agent runs the loop.

Here is a real prompt. It is an exception to the rule above: it found a Snapchain network halt on the first run. We patched the bug the same day.

The screenshot says 190 lines because I ran the method then as a plugin with several files. The gist is the same method in one file of ~450 lines.

That prompt has six focus parameters. The file calls the full set the contract. It has twelve.

PREMISE:   <the vulnerability / bug / cause> exists.
           Evidence for the belief: <observation, report, primitive>.
TARGET:    <asset> against <population>.
ATTACKER:  <who> with <privileges> and <interaction required from the victim>.
SUCCESS:   <universal capability>: any actor, any user, any input, any version,
           any deployment, repeatable, full impact.
CURRENT:   <what is already held>: primitives, sub-chains, reports.
MISSING:   <the full chain / the enabling hop / the production path>.
SCOPE:     <repos, packages, layers>; method unrestricted inside the boundary.
BOUNDARY:  only <test accounts / local replica>; never <production / other users /
           named channels>; minimum blast radius; halt if <condition>.
PROOF:     <observable> reproduced in a production replica, then shown to apply
           to the real deployment; independently verifiable.
CONSUMER:  <who uses the proof and what decision it changes>.
FALSIFIER: <observation that would kill the claim or retire the premise>.
STOP:      proof, proven negative, or <named successor target>.

Every focus parameter is optional. Add one when the hunt needs it. For one that you leave out, the file tells the agent to read the repo and fill it in. STOP is a short form. The stopping rule in the file has all five reasons.

Then the agent runs this loop.

The file was built by its own method. Agents collected 154 of my messages from real hunts and checked each quote against its source. Three independent reviews found problems in the first version, and the fixes are in the file.

The boundary: a ship in a bottle

An agent that does not stop is safe only inside a boundary. I put the boundary in the same message as the hunt.

We want absolute undeniable proof of the full chain in local as a prod replica before escalating beyond.

That needs a place. We call ours a ship in a bottle, thanks Bob. The whole ship is built and it works, but it lives inside glass.

A bottle is a full copy of your stack on one machine, sealed from the internet.

  • The database starts empty and holds only fake data.
  • The bottle holds no real secrets.
  • A local fake answers each call to a third party.
  • There is one narrow way out, and each use of it is logged.
  • Your code is mounted live, so an edit shows at once.

Now the agent can run hard. It can reproduce a bug and prove a fix from end to end. If something takes control of the agent, there is little to steal inside the bottle and one logged way out.

Other teams build the same way. Anthropic says that a sandbox needs both file isolation and network isolation. Monad tells people to run its open bug-hunting tool only on machines that they can throw away.

A bottle has limits. A proof in the bottle is not yet a proof about production. The agent sits outside the bottle, on your machine. And the UK AI Security Institute reports that models can escape containers that are set up badly. There's another angle to consider there, still cool.

The file is free. The bottle is work. If you run a company, this is the part to fund.

The setup: buy coverage cheap and buy certainty once

My setup is small. I use Oh My Pi (omp), a coding agent for the terminal, inside the Zed editor_._ Most of the work runs on the cheap model. (btw: I do feature work (like Articles) in Claude desktop)

omp --profile openmodel-flash --model openrouter/deepseek/deepseek-v4.1-flash-1017 --thinking max

The cheap model covers the ground. Each path gets a mark.

mark it checked / valid / invalid /etc and DS for deepseek

Then a stronger model makes one last pass over the covered ground. That is the one place where I pay for certainty.

A model does not grade its own work. A different model checks the claims.

so another agent to check them itself via different model

Trail of Bits does the same: its second judge is a different model.

A harness is the loop that runs the agents. The skill is the text that they follow. My harness is a few shared notes files in a reliable agent harness (like claude code). They hold the context, the coverage, the findings and the chains. Each agent reads them and writes to them, so the agents can use the work of the others.

I add agents by angle, not by count. Ten agents with the same view usually find what one agent finds.

Some agent tools let you set a goal or a stop condition. Put the exit condition there. Then you do not type "continue" each time the agent stops.

I still steer by hand.

I also like to watch the thinking and spin up more subagents on things I find important.

A large automatic harness is good for wide coverage and for known kinds of bug. For a zero day, I steer by hand to find what our harnesses miss.

Sometimes I gather too much context and need to approach it from a fresh perspective, the same problem exists with harnesses.

If you want a full harness to study, Monad's Ultrafuzz is open source. It is made for smart contracts.

The two work together. The automatic harness does the constant daily upkeep. The hand-steered hunt looks for the exotic zero day. They are two sides of the same coin.

The limits: tomorrow's code

First, the honest state.

This is far from perfect and there's a lot of manual intervention

The method is also a hypothesis. It worked for me, more than once. Nobody has measured why.

A hunt finds what is there today. Your code changes tomorrow, and the models change too. Our signer takeover had been live since the Snapchain mainnet release. A flaw in the Orchard pool of Zcash sat in public code for more than four years. A new model found it one day after its release. The older model did not find it without specific direction. So each new model is a new hunt, for you and for an attacker.

An active company needs this to run all the time: hunt, prove, triage, patch. The patch side can be fast too. In the DARPA final, teams sent patches in 45 minutes on average.

The next problem: triage

The same file helps with triage. Give it a report from another person. It confirms the claim or it kills the claim.

so this never happens again where we raise false alarms

Triage at scale is the next problem to solve. In May 2026, Zcash Community Grants closed its bug bounty and kept its private disclosure channel open. Its post makes the case in four steps. AI is strong at narrow search work in a large codebase, so it finds bugs that sat in plain view for years. Work that took days or weeks can now be tried in hours. The same speed brings a stream of speculative, duplicated and invented reports, and each one needs careful human triage. So the money moves to review that is proactive and repeatable: audits that use AI.

Picture: Zcash Community Grants, from Closing the ZCG Vulnerability Bounty Program, 26 May 2026.

The picture shows who finds what. AI takes the wide base alone. The higher layers need a person to steer. My results change one detail: at the top, the person does not work alone. The person steers an agent. It is the same split as the automatic harness and the hand-steered hunt above.

Proof can solve the triage problem. A report that carries a reproduction can be checked by a machine before a person reads it. One reply in that thread asks for exactly this: exploit the bug on a private test network first. In an AI factory, triage is one more station on the line: hunt, prove, triage, patch.

The keys beside the code

Everything above is about your code. But the agent, the tools and you all work on a laptop. That laptop holds keys: one for your code host, one for your secrets, one for your deploys.

We believe that most devices are more likely to be compromised than not. One bad package is enough. One bad extension is enough. One agent that reads the wrong page is enough.

Most defences ask "how do I keep an attacker out?" A useful question is "what does the attacker get when they are in?"

Today the answer is usually "everything". Many developer tools keep a long-lived key in a file, and any program that runs as you can read it. A password manager is better, but one unlock opens the whole vault for a period.

We think every key needs three limits:

  • One device. The key is locked to the hardware of one machine. A copy on another machine is useless.
  • One program. The key goes only to the program that you named. Another program that asks gets nothing.
  • One approval. A person approves each key, each time. No single unlock opens them all.

Then a compromised laptop gives an attacker one action that you approved. It does not give them your company.

That is a subject for another article. We are building for it, and we will open the work later. It is here for one reason: a hunt finds the holes in your code, and it does nothing for the keys that sit beside it.

The full skill file

The images contain core components of the skill file formatted and the gist is the full skill.

Found something on Farcaster? Submit here.

10 likes1 recasts0 replies

0 replies

No replies yet