What we do

Services
Ways to work with us

Industries

Industries
More industries

Company

Company
Case studies

Insights

Insights
Latest
Work with us

AI safety starts with what the agent is allowed to change

Our view: define what an agent may commit your business to before expanding its autonomy. Five questions make that decision practical.

By the dplyz team10 min read

What should business AI agent safety actually protect?

Our view: start with the commitments an agent can make on your behalf. Before giving it access to a CRM, booking system or finance tool, decide whose records it may change, what consequences you will accept, and who can stop it. Those decisions belong in the business brief.

Consider a hypothetical home services company. An assistant suggests an appointment the crew cannot reach. In a draft, that is an error a dispatcher can catch. Sent as a confirmed booking, it becomes a customer promise and a problem for the afternoon schedule. The suggestion is identical. The authority changes the consequence.

A pink and teal spiral of light contained by a luminous circle against a deep black field.

OWASP’s 2025 guidance on excessive agency identifies three recurring causes: too much functionality, too many permissions and too much autonomy. It covers damaging actions caused by ordinary model errors as well as malicious manipulation. A system does not need to be hacked to make an expensive mistake.

For a growing business, this makes agent safety an operating decision. You want the assistant to move routine work forward without giving it room to improvise company policy. The five questions below turn that ambition into a brief an operations lead, a finance lead and an engineer can agree on.

What exactly may the agent commit the business to?

1. Describe the action in terms a customer or employee would recognize

“Manage service requests” is too broad. Break it into proposing a visit, reserving a slot, confirming the price, assigning a crew and changing an existing commitment. Each deserves its own decision. Permission to prepare one step should not silently grant permission to finish the rest.

For our hypothetical company, let the agent organize incoming requests and prepare options. Decide separately whether it may confirm an appointment, promise a discount or move someone else’s job. An instruction to improve customer satisfaction is not permission to waive invoices.

This is consistent with the NIST AI RMF 1.0 core: its Map function calls for documenting business context, risk tolerance and intended application scope. Use those questions to settle the purpose before buying more capability.

Proposed actionIllustrative starting authorityDecision the business must make
Suggest appointment optionsDraft for a dispatcherWhich availability and service rules must the suggestion respect?
Tag a request for internal reviewLimited automatic actionWhich queue, records and categories are allowed?
Change a confirmed appointmentReview before executionWhose commitments and travel plans could change?
Offer a service creditReview before executionWho owns the financial concession and its limits?
Rewrite the company price listOutside this agent’s jobDoes the business want this capability at all?

These are suggested starting positions, not a universal risk classification. An internal tag can still matter if another system treats it as an instruction. Follow the effect through to the end: who will see it, what might run next, and what obligation could it create?

Keep a short record for each permitted action: purpose, affected people, permitted records, accountable owner and evidence required to proceed. A useful review asks whether this particular action should happen under these conditions. Asking whether the whole assistant is “safe” is too vague to settle anything.

Make the limits enforceable outside the conversation. OWASP recommends checking authorization in downstream systems, rather than leaving that decision to the model. Ask the engineer to demonstrate a rejected action as well as a successful one. The business brief should explain what must be refused; the software must actually refuse it. This gives the owner a useful acceptance question: can the assistant change this record when the required condition is absent? A reassuring explanation from the assistant is not the evidence. The observed result in the connected system is.

How far can one wrong decision spread?

2. Limit the total effect, not just the size of one action

A small mistake repeated across every branch is a large mistake. In our example, a wrong service category could affect one request or reroute a whole morning’s enquiries. Write down the permitted reach: one location or several, one customer or a list, one run or continuous operation.

The same applies to money. A modest credit may be acceptable on one eligible account. A sequence of modest credits can exceed the budget. We recommend defining both individual limits and an aggregate limit over a useful business period, with one owner responsible for changing them.

OWASP describes rate limits and monitoring as ways to limit damage from excessive agency, rather than guarantees that prevent it. That distinction matters: a slower wrong action is still wrong. Use limits alongside the checks that establish whether an action is allowed.

Scope the permission before increasing volume
One actionWhat may change on this request?
One populationWhich customers, records or locations?
One operating periodHow much total activity is acceptable?

Start with a population the team can inspect. A single branch with known processes may give you a clearer trial than a company-wide launch with conflicting rules. Expansion should follow evidence that the controls hold and staff can handle exceptions. More connected systems do not automatically justify more authority.

Read access also deserves a boundary. An assistant preparing a schedule should receive the information needed for that task. It should not inherit every customer note simply because the connection makes those notes available. Keep credential handling and access enforcement in the engineering scope; our guide to credentials outside agent workspaces covers that separate design decision.

Can the business genuinely undo the action?

3. Judge recovery by the customer impact

“We can restore the record” is an incomplete answer. If a customer has already rearranged their day around a confirmed visit, changing the booking back does not undo the inconvenience. If a crew has left for the wrong job, the old calendar entry will not recover the travel time.

For every proposed permission, ask what recovery would involve after someone has acted on the result. Could the business correct it quietly, or would it need to contact customers, renegotiate a commitment or involve finance? Who would do that work, and how quickly could they notice the problem?

The OWASP AI Agent Security Cheat Sheet recommends explicit approval for high-impact or irreversible actions and separating a proposed action from its execution. Apply that principle to the real commitment, even when the underlying database change is easy to reverse.

In the service example, drafting an alternative appointment is a useful place to begin. Reassigning an occupied slot deserves a different discussion because it affects another customer. If the cost of recovery is unclear, keep the action in review while the business works it out.

Time changes the calculation too. A scheduling error noticed before confirmation is different from the same error found after the crew is on the road. Set the review and monitoring expectations around the point where harm begins. A daily report may be adequate for some internal housekeeping; it is a poor substitute for a check before a consequential promise.

This is also where the commercial case becomes more honest. Include the likely review work, exception handling and recovery effort when comparing options. The cheapest automated step can create the most expensive manual cleanup. A narrower workflow that finishes reliably may be the better investment.

What should happen when the agent lacks a reliable answer?

4. Make uncertainty a supported outcome

An assistant should be able to return a useful partial result. “I found two possible customer records; please choose one” is often better than a confident update to the wrong account. Design the handoff so the person receives the relevant evidence and the unresolved question.

For our hypothetical dispatcher, define cases that require a pause: conflicting availability, an unclear service address, an exception to the pricing policy or a request outside the normal service area. A fluent explanation from the model should not override missing business evidence.

OWASP’s agent security testing guidance calls for checking that unauthorized tools are denied even when a model requests them confidently. Test your business boundaries with the same discipline: show that a confident recommendation cannot enlarge the agent’s permission.

Build the trial around the decisions the company actually faces. Include ordinary requests, incomplete requests and cases where two reasonable goals conflict. Record whether the system completed the right job, stopped at the right boundary, or handed over enough information for a person to continue.

Look at the workload left behind. If staff must reconstruct every case before approving it, the assistant may have moved work rather than reduced it. If it always escalates a particular situation, fix that situation’s information or policy before increasing permissions. Sometimes the business needs a clearer rule before the model needs better instructions.

Who can withdraw permission when the system stops behaving as intended?

5. Assign ownership before increasing autonomy

Give the workflow a named business owner and a technical owner. The business owner defines acceptable commitments and exceptions. The technical owner makes those boundaries enforceable and keeps the system observable. Staff need to know whom to contact when the results stop making sense.

NIST AI RMF 1.0, Manage 2.4 calls for assigned responsibilities and mechanisms to override or deactivate systems whose outcomes depart from intended use. Manage 4 also addresses monitoring, incident response and recovery. The ability to withdraw authority belongs in the launch plan.

A recovery plan needs an owner at every step
Pause the actionStop further commitments.
Identify the effectFind affected records and people.
Correct and communicateGive the business a clear recovery task.
Review before resumingConfirm what changed and what was tested.

A stop control is only useful if someone can operate it during the hours the agent runs. Agree what continues while the action is paused: perhaps enquiries can still be collected, while confirmations return to a dispatcher. Exercise that fallback before depending on it.

Decide what evidence the owner will review. Completed jobs, corrections, missed boundaries and unresolved handoffs tell you more than a count of messages generated. Keep enough operational history to identify affected cases without turning every log into a copy of sensitive customer data.

Our position is that autonomy should be earned one action at a time and remain revocable. A stronger model may improve the work. It does not, by itself, answer whether the business should permit a broader commitment. Revisit authority when the process, customer population or operating conditions change.

Bring these five decisions into your AI agent development brief. They give the engineers something concrete to implement and the business something concrete to accept. If you have a workflow in mind, tell us what it needs to achieve; we can help define its first useful scope.

FAQs

Does a more accurate model make an agent safe enough to act?

Accuracy is relevant evidence, but it does not decide the consequence your company will accept. Evaluate the model on the actual job, then assess the permissions, recovery options and human workload around it. Keep the business authority decision explicit.

Does every action need a person to approve it?

No. We recommend limited automatic authority for well-defined actions with acceptable consequences and enforceable controls. High-impact actions deserve independent review. The choice should follow the specific action and operating context; calling a workflow routine is not enough.

Can an agent follow the rules and still create a business problem?

Yes. A permission may be too broad, or a permitted action may have consequences the team overlooked. That is why the assessment should include normal mistakes, repeated actions and effects on other workflows, alongside deliberate attempts to misuse the system.

Is following NIST AI RMF a compliance certification?

No. NIST describes the AI Risk Management Framework as voluntary guidance. Using it can structure your risk work; it does not establish that a particular deployment satisfies every law, contract or industry requirement. Those obligations need their own review.

What should we put in the first agent safety brief?

List the permitted commitments, affected records and people, cumulative limits, pause conditions, recovery steps and accountable owners. Include examples the agent should finish and examples it should hand over. Use that brief to judge the trial before widening access.

Written by the dplyz team

Senior engineers who work inside client companies to take AI from pilot to production: agentic workflows, AI integration and the platform work around them.

How we work

What’s slowing your team down?

Describe the bottleneck in a few lines. The people who would fix it reply with how they’d do it, what it connects to and how fast it can ship.

Describe your bottleneck