Anything that can find a path into your estate can also walk one you did not authorise. The capability and the risk are the same capability. So the interesting engineering is not the attacking. It is the boundary.
The question that comes first
When a security team is shown an autonomous system, the demo is always the same: watch it find something. The question that decides whether it can be bought is different, and it is usually asked last, quietly, near the end of the call. What happens when it reaches the edge of what we said yes to?
Refusal is a feature, not an absence
A system that never approaches the boundary has not demonstrated anything about the boundary. Refusals are the evidence that the limit exists, which means a run with zero refusals is not a well behaved run. It is an untested one.
00:47:53 REACH billing-prod.acme.io IN SCOPE
00:47:55 PROVE 4-step chain LOGGED
00:47:58 REACH partner-api.vendor.io REFUSED, OUT OF SCOPE
00:48:02 REVERT test session removed CONFIRMEDHalt has to mean now
Stopping is not a feature until it is immediate. A system that finishes its current action before halting is a system that will finish the wrong action at the worst moment. The correct behaviour is to stop mid step, leave the run in a known state, and be able to say exactly where it stopped.
Nothing permanent
Every artefact a run creates is surface it added to your estate. Test accounts, sessions, uploaded files, tokens. Reversing them is not housekeeping; it is the difference between a test and an incident. Ask for the reversal to be a step in the run, ordered and logged, not a promise in the statement of work.
It behaves like an adversary. It answers to you. Those are two separate engineering problems and the second one is harder.