From runbooks to agents: automating ops toil without automating outages

Most IT teams keep a folder of runbooks that someone works through by hand: purge stale records, rotate logs, fix the config value that keeps drifting, restart the service that leaks memory. Nobody enjoys this work, and in 2026 the obvious move is to hand it to a scheduled job or an AI agent. The risk is easy to miss. A careful engineer doing a task at 10am becomes a script doing it at 3am with nobody watching, and one bad assumption can become an outage or an empty table.

This post is part of our Practical AI in DevOps series. It covers how we automate this kind of toil on our own platform and where we keep a person in the loop.

Pick runbooks that are boring and reversible

Good first candidates run often, have a clear definition of done and can be undone. Purging expired records qualifies if you have backups and a retention window. Changing firewall rules on a production edge is a poor first project. If a runbook step says “use judgement”, it needs either a person or a much tighter written rule before it goes on a schedule.

Dry-run first, and leave it there for a while

Every automated task we build starts with a dry-run mode. In that mode it does all the reading and deciding but none of the writing, and it logs what it would have done in enough detail that an engineer can check each decision.

The clearest example on our own platform is an automated data cleanup. It ran in dry-run for weeks before we trusted it to delete anything. Over that period we read its output and compared each proposed deletion with what we would have removed by hand. We switched it to live only after those lists matched run after run.

Weeks can feel slow for a small task. Keeping a dry-run going costs very little, while a wrong deletion means restore work and an uncomfortable conversation with whoever owns the data. The shape we use looks like this:

def run_cleanup(dry_run: bool = True, max_deletes: int = 500) -> None:
    candidates = find_expired_records()
    if len(candidates) > max_deletes:
        alert(f"cleanup wants {len(candidates)} deletes, limit is {max_deletes}")
        return
    for record in candidates:
        if dry_run:
            log.info("would delete %s (expired %s)", record.id, record.expired_at)
        else:
            delete_if_exists(record.id)

Dry-run is the default. Going live needs an explicit flag in the scheduler config, so the switch is a reviewed change with a name against it.

Make every step safe to run twice

Schedulers retry, runs overlap and jobs die halfway through. Write each step so that running it a second time does no harm. In practice that means “set the timeout to 30” instead of “add 10 to the timeout”, “create the folder if it is missing”, and “delete this record if it still exists”. Take a lock at the start so two runs can’t work on the same data at once, and give each run an ID that appears in every log line.

When a run fails partway, the next run should finish the job without anyone tidying up first. If you can’t make a step idempotent, leave it in the manual runbook.

Prefer string edits to model rewrites

Language models are useful in ops for summarising logs, sorting alerts and proposing a fix. They are a poor tool for applying the fix. Ask a model to change one timeout in a 300-line YAML file and you get back a whole file that looks right. A comment may be missing, keys may have moved, and somewhere a value you never mentioned may have changed.

A deterministic edit avoids that. Find this exact string, replace it with that one, and fail if the string doesn’t appear exactly once. The diff is a single line and a reviewer can check it in seconds. When we use a model to work out the change, we have it output the old and new strings, and plain code applies them.

The same idea is spreading through agent tooling. Tesseracted Labs wrote about moving coding-agent guardrails out of prompts and into runtime hooks, arguing that an invariant shouldn’t live inside the probabilistic system it is meant to constrain. Their example of two agents with GitHub access approving each other’s pull requests shows how a prompt-level rule fails without anyone noticing. For SQL, the open-source schema-guard checks agent-written queries against a snapshot of the real schema before they run, which catches invented column names before they reach a database.

Alert when the numbers look wrong

Give every automated task an expected range: how many rows it normally deletes, how many services it normally restarts, how long a run takes. When a run falls outside that range, the task stops and raises an alert, and a person decides what happens next.

The max_deletes check in the snippet above does this. If a cleanup that usually removes a few hundred rows suddenly wants to remove most of a table, something upstream has probably broken, such as a status field that was renamed so that no record looks active any more. Pushing on through is the wrong response. The same goes for agents watching monitoring data: an anomaly opens a ticket or pages someone. We allow automatic remediation only for actions that have already been through the dry-run and staging steps.

Give each task only the access it needs

Platform vendors now treat broad agent permissions as a real risk. In early October, Apple tightened macOS Full Disk Access controls, and Ars Technica reported that Apple’s stated reason was the growing risk of that level of access as AI agents become more capable and autonomous.

For scheduled ops tasks, the cleanup job’s database account can delete from the one table it cleans and can’t drop anything. Each task runs under its own identity, so the audit log shows which task did what. Where an agent proposes shell commands, something deterministic has to sit between the proposal and the server. The zero-trust-llm project shows one pattern: middleware intercepts the command, checks it against a read-only allow-list, and asks a human for a yes or no before anything else runs. AWS has also released Strands Box, an open-source local agent sandbox in developer preview, with per-tool policies such as “allow git push only if tests are passing”.

Roll out in stages

Once a task has earned its way out of dry-run, it goes live in a test environment, then on one server or tenant, then everywhere. We start with a low per-run cap and raise it as the run history builds up. The manual runbook stays current until the automation has a stable record, and the dry-run flag stays in place so that switching back takes one config change.

Budget for drift

Automations that work on day one break later because the systems around them change. One automation developer on Reddit described “maintenance drift” as the cost they had underestimated on every job, giving the example of a client renaming a HubSpot property or moving an intake form six weeks after delivery.

Build assertions into each task that check its assumptions before it acts. Confirm the columns it reads still exist and the API still returns the fields it expects. When an assertion fails, the task stops and alerts, through the same anomaly path as above.

What it costs

This approach takes more code than a straight script, and the dry-run period pushes back the payoff by weeks. It also has a running cost, because someone has to read alerts and update assertions when the systems around a task change. For a runbook that runs once a quarter, that overhead can be more than the toil it replaces, and we leave those runbooks manual. For daily and weekly chores, the trade has paid off for us.

PicNet builds production AI systems for Australian organisations. If you have a folder of runbooks you would like to stop doing by hand, talk to us about what a first project could look like.