Budget the verification before the hunt
Cloudflare’s security audit skill can be useful to a business with custom software, but an open-ended request to audit everything is a poor buying decision. Start with a defined part of the application, a spending limit and enough time to verify what the agents report. The deliverable should help an engineer fix a demonstrated problem.
“Token monster” is our shorthand for a workflow that can grow through repeated investigation and review. It is not a published benchmark or a claim about what your audit will cost. The important question is whether the work produces evidence worth acting on.
Cloudflare’s open-source project is a workflow run by a coding agent. It is not a hosted vulnerability scanner that comes with your Cloudflare account. It organizes investigation and independent review, then produces findings and coverage records. Your choice of model, execution environment and scope determines much of the operating cost.
Choose a boundary the business cares about
Consider a hypothetical distributor with a customer portal. Customers can download invoices and invite colleagues to their account. A sensible first review could ask whether one customer can reach another customer’s invoices, including through exports and background jobs. That is a concrete question with a business consequence.
Write down the repository revision, the relevant routes and workers, and the systems excluded from the pass. Include the people who can explain access rules. If the answer depends on an identity-provider setting outside the repository, record that dependency for its owner to check.
Avoid buying a broad assessment while only funding a narrow one. A scoped pass can be worthwhile, provided everyone understands what remains unexamined. The invoice review says little about payroll, supplier integrations or the rest of the application.
A promising finding creates more work
The skill separates discovery from verification. Its validation procedure asks a fresh reviewer to challenge each candidate and distinguishes confirmed findings, questions needing specific validation, and rejected claims. Standard and deep profiles include another independent check of the final records; the quick profile combines those review stages.
That structure explains why cost can rise after the first apparent success. A reviewer may need to reconstruct the access checks, build a bounded local test and discover that a control already prevents the proposed attack. Rejecting that claim is useful work. It saves the team from planning a fix for a problem that was never established.
Cloudflare’s budget guidance counts agent invocations and reserves capacity for reconnaissance, coverage criticism and verification before assigning hunting work. That is a planning mechanism, not a dollar ceiling: two agent calls can use different amounts of context, output and execution time.
For a paid run, record actual model usage and engineering time alongside the agent count. Check the provider’s current charging rules for the models you select. Enforce a spend ceiling in the surrounding system where available; a requested budget in a prompt is not a substitute for a billing control.
Agree on the stop condition before starting
We would put a short work order in front of the engineer running the audit. It should make the decision to continue explicit, especially when a promising lead needs more validation.
| Decision | What to record |
|---|---|
| Scope | The business boundary, source revision, included paths and exclusions. |
| Resources | Model and tool budgets, maximum concurrent work, elapsed-time limit and a named approver for expansion. |
| Verification reserve | Capacity kept available to check candidates before buying another hunting pass. |
| Execution | The isolated environment, synthetic test data and permitted local effects. |
| Handoff | Evidence, unresolved questions, engineering owner and the next decision for each result. |
Suppose the hunting budget is nearly spent and several candidates remain unverified. Stop opening new lines of investigation. Finish what the remaining budget can support and label the rest as incomplete. A shorter, honest handoff is more valuable than a long list that mixes evidence with speculation.
Concurrency also needs a limit. Starting more investigators at once can shorten elapsed time while increasing simultaneous spend and the volume of results waiting for review. Choose a level your team can supervise and reconcile.
Keep the audit away from live customers
The current skill’s execution rules require target-controlled code to run inside an operating-system-enforced sandbox, with external networking disabled, a sanitized environment, restricted writes and resource limits. If those controls are unavailable, the workflow should retain the validation blocker instead of running the code.
In the invoice example, use dummy customers and synthetic invoices. The question is whether the control fails under the test conditions. There is no reason to prove it by downloading a real customer’s documents or probing a shared production service.
Treat the audit output as sensitive engineering material too. Give the people fixing the issue access to the evidence, and keep private traces, credentials and customer records out of public reports. Agree on retention and access before generating a directory full of findings.
Count resolved decisions, not alarming paragraphs
Track cost per verified, actionable finding, but keep it beside coverage and severity. A run with no confirmed findings makes that ratio undefined; it does not automatically make the review worthless or prove that the application is secure. You still need to know what was examined and what could not be checked.
Useful operational measures include the cost of investigation and review, engineer time spent on rejected candidates, unresolved validation blockers, time to an accepted remediation plan, and whether the relevant regression test passes after the fix. Keep model cost separate from the full cost of remediation.
The audit and the repair are separate pieces of work. A confirmed issue needs an owner, a narrow change, a regression check and an appropriate release plan. Expanding the audit indefinitely while known problems sit unfixed is rarely the best use of the next dollar.
For a lean team running important business software, start where a failure would hurt and where someone can act on the result. Our platform and cloud work connects that review to the release and recovery process. Talk with us about the application boundary you want to examine.
Written by the dplyz team
Senior engineers who work inside client companies to take AI from pilot to production: agentic workflows, AI integration and the platform work around them.
