What exactly may the agent commit the business to?
1. Describe the action in terms a customer or employee would recognize
“Manage service requests” is too broad. Break it into proposing a visit, reserving a slot, confirming the price, assigning a crew and changing an existing commitment. Each deserves its own decision. Permission to prepare one step should not silently grant permission to finish the rest.
For our hypothetical company, let the agent organize incoming requests and prepare options. Decide separately whether it may confirm an appointment, promise a discount or move someone else’s job. An instruction to improve customer satisfaction is not permission to waive invoices.
This is consistent with the NIST AI RMF 1.0 core: its Map function calls for documenting business context, risk tolerance and intended application scope. Use those questions to settle the purpose before buying more capability.
| Proposed action | Illustrative starting authority | Decision the business must make |
|---|---|---|
| Suggest appointment options | Draft for a dispatcher | Which availability and service rules must the suggestion respect? |
| Tag a request for internal review | Limited automatic action | Which queue, records and categories are allowed? |
| Change a confirmed appointment | Review before execution | Whose commitments and travel plans could change? |
| Offer a service credit | Review before execution | Who owns the financial concession and its limits? |
| Rewrite the company price list | Outside this agent’s job | Does the business want this capability at all? |
These are suggested starting positions, not a universal risk classification. An internal tag can still matter if another system treats it as an instruction. Follow the effect through to the end: who will see it, what might run next, and what obligation could it create?
Keep a short record for each permitted action: purpose, affected people, permitted records, accountable owner and evidence required to proceed. A useful review asks whether this particular action should happen under these conditions. Asking whether the whole assistant is “safe” is too vague to settle anything.
Make the limits enforceable outside the conversation. OWASP recommends checking authorization in downstream systems, rather than leaving that decision to the model. Ask the engineer to demonstrate a rejected action as well as a successful one. The business brief should explain what must be refused; the software must actually refuse it. This gives the owner a useful acceptance question: can the assistant change this record when the required condition is absent? A reassuring explanation from the assistant is not the evidence. The observed result in the connected system is.
How far can one wrong decision spread?
2. Limit the total effect, not just the size of one action
A small mistake repeated across every branch is a large mistake. In our example, a wrong service category could affect one request or reroute a whole morning’s enquiries. Write down the permitted reach: one location or several, one customer or a list, one run or continuous operation.
The same applies to money. A modest credit may be acceptable on one eligible account. A sequence of modest credits can exceed the budget. We recommend defining both individual limits and an aggregate limit over a useful business period, with one owner responsible for changing them.
OWASP describes rate limits and monitoring as ways to limit damage from excessive agency, rather than guarantees that prevent it. That distinction matters: a slower wrong action is still wrong. Use limits alongside the checks that establish whether an action is allowed.
Start with a population the team can inspect. A single branch with known processes may give you a clearer trial than a company-wide launch with conflicting rules. Expansion should follow evidence that the controls hold and staff can handle exceptions. More connected systems do not automatically justify more authority.
Read access also deserves a boundary. An assistant preparing a schedule should receive the information needed for that task. It should not inherit every customer note simply because the connection makes those notes available. Keep credential handling and access enforcement in the engineering scope; our guide to credentials outside agent workspaces covers that separate design decision.
Can the business genuinely undo the action?
3. Judge recovery by the customer impact
“We can restore the record” is an incomplete answer. If a customer has already rearranged their day around a confirmed visit, changing the booking back does not undo the inconvenience. If a crew has left for the wrong job, the old calendar entry will not recover the travel time.
For every proposed permission, ask what recovery would involve after someone has acted on the result. Could the business correct it quietly, or would it need to contact customers, renegotiate a commitment or involve finance? Who would do that work, and how quickly could they notice the problem?
The OWASP AI Agent Security Cheat Sheet recommends explicit approval for high-impact or irreversible actions and separating a proposed action from its execution. Apply that principle to the real commitment, even when the underlying database change is easy to reverse.
In the service example, drafting an alternative appointment is a useful place to begin. Reassigning an occupied slot deserves a different discussion because it affects another customer. If the cost of recovery is unclear, keep the action in review while the business works it out.
Time changes the calculation too. A scheduling error noticed before confirmation is different from the same error found after the crew is on the road. Set the review and monitoring expectations around the point where harm begins. A daily report may be adequate for some internal housekeeping; it is a poor substitute for a check before a consequential promise.
This is also where the commercial case becomes more honest. Include the likely review work, exception handling and recovery effort when comparing options. The cheapest automated step can create the most expensive manual cleanup. A narrower workflow that finishes reliably may be the better investment.
What should happen when the agent lacks a reliable answer?
4. Make uncertainty a supported outcome
An assistant should be able to return a useful partial result. “I found two possible customer records; please choose one” is often better than a confident update to the wrong account. Design the handoff so the person receives the relevant evidence and the unresolved question.
For our hypothetical dispatcher, define cases that require a pause: conflicting availability, an unclear service address, an exception to the pricing policy or a request outside the normal service area. A fluent explanation from the model should not override missing business evidence.
OWASP’s agent security testing guidance calls for checking that unauthorized tools are denied even when a model requests them confidently. Test your business boundaries with the same discipline: show that a confident recommendation cannot enlarge the agent’s permission.
Build the trial around the decisions the company actually faces. Include ordinary requests, incomplete requests and cases where two reasonable goals conflict. Record whether the system completed the right job, stopped at the right boundary, or handed over enough information for a person to continue.
Look at the workload left behind. If staff must reconstruct every case before approving it, the assistant may have moved work rather than reduced it. If it always escalates a particular situation, fix that situation’s information or policy before increasing permissions. Sometimes the business needs a clearer rule before the model needs better instructions.
FAQs
Does a more accurate model make an agent safe enough to act?
Accuracy is relevant evidence, but it does not decide the consequence your company will accept. Evaluate the model on the actual job, then assess the permissions, recovery options and human workload around it. Keep the business authority decision explicit.
Does every action need a person to approve it?
No. We recommend limited automatic authority for well-defined actions with acceptable consequences and enforceable controls. High-impact actions deserve independent review. The choice should follow the specific action and operating context; calling a workflow routine is not enough.
Can an agent follow the rules and still create a business problem?
Yes. A permission may be too broad, or a permitted action may have consequences the team overlooked. That is why the assessment should include normal mistakes, repeated actions and effects on other workflows, alongside deliberate attempts to misuse the system.
Is following NIST AI RMF a compliance certification?
No. NIST describes the AI Risk Management Framework as voluntary guidance. Using it can structure your risk work; it does not establish that a particular deployment satisfies every law, contract or industry requirement. Those obligations need their own review.
What should we put in the first agent safety brief?
List the permitted commitments, affected records and people, cumulative limits, pause conditions, recovery steps and accountable owners. Include examples the agent should finish and examples it should hand over. Use that brief to judge the trial before widening access.
Written by the dplyz team
Senior engineers who work inside client companies to take AI from pilot to production: agentic workflows, AI integration and the platform work around them.








