When AI must not act alone: human-in-the-loop patterns for security automation

Most of the AI security work we do for clients ends up hinging on one question: what is this thing allowed to do on its own at 3am on a Sunday?

Everything else follows from the answer. Model choice, prompt design, which vendor’s agent framework you use, all of it matters less than where you draw the line between an action the system takes and an action a person takes. This post closes out Practical AI in Cyber Security with the governance layer that the earlier posts assumed was there.

Draw the line at reversibility

The clean split is not “smart tasks versus dumb tasks”. It is reversibility. If the worst outcome of the AI being wrong is a wasted five minutes of someone’s reading time, automate it. If the worst outcome is a production system offline, a mailbox item gone, or a clinician locked out of a roster, a human decides.

Safe to run unattended:

• Collection. Pulling logs, endpoint telemetry, sign-in events, ticket history, config snapshots into one place.
• Enrichment. Resolving an IP to an ASN, looking up a hash, attaching asset ownership, pulling the last three changes to the affected host.
• Triage and correlation. Grouping alerts, suppressing known-benign patterns, ranking by likely impact, drafting the “what we think happened” summary.
• Reporting. Weekly posture summaries, control-drift reports, evidence packs for an audit.

Behind a human decision:

• Blocking. Firewall rules, conditional access changes, disabling a user account.
• Deleting. Quarantining or purging mail, removing files, revoking tokens at scale.
• Isolating. Pulling a host off the network, suspending a VM, stopping a service.
• Anything that touches a system of record. Closing an incident, changing a risk rating, writing to an asset register.

Vendors are pushing hard in the other direction. Cloudflare’s own material on agentic AI describes systems that make decisions, execute code, query databases, send email and handle payments without constant human prompting. That is a fair description of the capability. It is also a precise list of the things you want fenced off in a security context, because each one of them is hard to undo.

The split is not new and it is not unique to security. Windmill’s engineering write-up on their support pipeline lands in exactly the same place: AI does the volume work, drafting replies and even drafting the fix, but a human still presses send on anything customer-facing. Their reasoning is worth stealing wholesale. The system only works while people trust the drafts, and one wrong auto-sent reply burns that trust for every message after it. A SOC analyst who has been burned once by an automated block on a payment gateway will start ignoring the tool, and then you have paid for AI and got nothing.

The pattern: agent proposes, human approves

The most useful working implementation we have seen of this contract is in a repo, not a security product. The kube-agents project documented a protocol for AI agents contributing code: an agent claims work by self-assignment, escalates anything it cannot resolve with a needs-human label, and cannot merge until a human lgtm gate is satisfied. Three primitives. Claim, escalate, approve.

That maps onto a security runbook almost directly:

claim    -> agent takes alert #4471, marks it in-progress so a second
            agent or analyst doesn't duplicate the work
enrich   -> pulls host, user, recent changes, threat intel, prior tickets
propose  -> writes the finding, the evidence and ONE recommended action
            ("isolate WKS-0412"), status = awaiting-approval
escalate -> if it cannot complete, status = needs-human with the reason
approve  -> named human approves, rejects or edits; only then does the
            action execute, under the human's authority

The kube-agents authors made the contract deliberately framework-agnostic. That is the detail to copy. Your SOC processes will outlive your current model vendor by years. Keep the escalation and approval contract in your own orchestration layer, expressed in your own states, so swapping Claude for Gemini next year is a config change rather than a governance rewrite.

A blind check must never look green

The failure mode that costs real money is not the AI recommending the wrong action. It is the AI reporting “all clear” on a check that never ran.

Alert and fault are different states and need different plumbing. An alert means the check ran and found something. A fault means the check could not run: the API token expired, the log source stopped shipping, the model call timed out, the agent hit a rate limit. Both are “not green”, but they need different handlers, different dashboards, and often different people.

Most monitoring code we inherit collapses the two. The check throws, something catches the exception, logs it at debug level and returns a default, and the tile stays green for eleven weeks while the log source has been dead the whole time. Adding an LLM to that pipeline makes it worse, because models are obliging. Ask one to summarise incomplete data and it will produce a confident, well-formatted summary of the subset it could see, with no flag that half the estate was missing.

What we build into these pipelines:

• Every check returns one of three states, never two: ok, alert, fault. There is no default value on the failure path.
• A fault renders as a distinct colour on the dashboard, not as a green tile with a footnote.
• Staleness is a fault. If a source has not reported inside its expected window, the check that depends on it is faulted, not passed.
• Coverage is reported alongside every AI-generated summary. “23 of 26 sources; 3 faulted” sits above the narrative, generated by the pipeline rather than written by the model.
• The model never sets its own coverage figure. That number comes from the orchestrator, which knows what it asked for and what came back.

The needs-human label is the same idea at the agent level. An agent that hits something it cannot resolve emits an explicit escalation state rather than quietly returning success.

The audit trail is the deliverable

When someone asks how your AI-assisted decisions are governed, the honest answer is whatever your logs can show. Governance is a document exercise at the point it gets tested.

That point is closer than it looks for organisations with any European exposure. The EU AI Act moved into enforcement on 31 August 2026 with the AI Office issuing formal requests for information to general-purpose model providers, covering model security, independent external evaluations and post-market monitoring. The requests went to providers rather than to deployers, so an Australian organisation using an API is not in the first wave. But the pattern is set: the questions are document-driven, the answers become part of a permanent supervisory record, and incorrect or incomplete replies carry penalties of up to 15 million euros or 3% of global turnover. Australian organisations serving EU users, or working under contracts that flow those obligations down, will be asked to evidence human oversight of AI-assisted decisions. You cannot backfill that evidence.

For each AI-assisted decision we record: what proposed the action, which model and version, what evidence it saw (the actual query results, not a paraphrase), what it recommended, who approved or rejected it, when, and what executed. In practice the evidence snapshot is the hard part, because inputs change under you. Store it with the decision.

One more control, learned from an open-source community rather than a compliance framework. The nixpkgs maintainers proposed closing any pull request that removes the AI disclosure checkbox, on the basis that merging one would be a serious policy violation. The interesting part is the failure mode they found. The control existed, but it was a checkbox the submitter could delete. So in a security pipeline, AI-provenance metadata must be system-generated and not editable by the thing that generated the content. If the agent can write its own “reviewed by a human” flag, the flag means nothing.

Roll it out in dry run, and stay there longer than feels necessary

Every automation we deploy runs in dry run first. The agent does the full job, including selecting the action, and writes what it would have done to a log and a channel humans read. Nothing executes.

We run this for a minimum of four weeks in most environments, longer for anything that touches identity or network paths. The dry-run period tells you three things you cannot get from testing: how often the agent recommends something an analyst would not do, how noisy it is at 2am when the batch jobs run, and whether the evidence it attaches is actually enough for a human to decide from. That last one fails more often than the first.

Promotion out of dry run is per action type, never for the whole system. Enrichment and triage might graduate in week two. Automated blocking might never graduate, and for most clients it should not.

An honest cost note. Human-in-the-loop is more expensive to run than full automation, because you are paying for the AI and keeping the analyst. The saving comes from what the analyst spends their time on: reviewing a drafted finding with evidence attached takes a couple of minutes, where building that finding from scratch takes twenty or thirty. The approval queue itself needs an owner and an SLA, or it becomes a place where alerts go to age quietly.

For anything clinically adjacent, the sign-off is part of the design rather than a caveat bolted on the end. An AI can assemble a suspected-breach pack, correlate the access logs, draft the notification timeline. A named clinician or privacy officer approves what leaves the building, and their name is on the record.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.