One small team, 100+ automated checks: anatomy of an AI-assisted monitoring platform

A backup job that has not run for eleven days will not send you an email. Neither will a TLS certificate with nineteen days left on it. The failures that cost our clients money are almost never the loud ones; they are the ones where something stopped happening and no system was responsible for noticing. We built our own monitoring platform for that class of problem, after years of stitching together vendor dashboards that each covered one slice of an estate and none of which covered the gaps between them. This post is part of our Practical AI in DevOps series, and it describes what the platform is made of, how little of it is AI, and what it costs us to keep running.

How the checks are put together

Every check is a scheduled task written in code. It runs on a timer, talks to an API or a device, compares what it finds against an expectation, and returns a structured result. There is no agent, no DSL and no plugin marketplace. Across the estate we run more than a hundred of these, grouped roughly as follows.

• Backup checks confirm that a restore point exists, that its timestamp is inside the agreed window, and that the job covers the volumes the client thinks it covers.
• Certificate and domain checks watch expiry dates, registrar locks, nameserver drift and whether the certificate presented on port 443 matches the one we issued.
• Network device checks read firmware versions, configuration deltas against the last known-good config, and support or licence expiry on firewalls.
• Cloud posture checks look for publicly readable storage, admin accounts without MFA, elevated roles nobody has used in ninety days, and logging that has been switched off.
• SIEM review checks pull the last day of correlated alerts and reduce them to what a human should read.

The contract every check returns is deliberately boring:

public record CheckResult(
    string CheckId,
    string ClientId,
    CheckState State,      // Ok, Alert, Fault
    string Headline,
    IReadOnlyList<Finding> Findings,
    DateTime RanAtUtc,
    TimeSpan Duration);

Because the shape is fixed, a new check is usually an afternoon of work rather than a project, and the reporting, escalation and history layers get it for free.

OK, ALERT and FAULT

The three-state result is the design decision that has paid for itself most often. OK means the check ran and found what it expected. ALERT means the check ran and found a problem: an expired certificate, a backup outside its window, a storage bucket open to the internet.

FAULT means the check could not run. The credential expired, the API returned 500, the device refused the connection, the script threw. Our platform treats FAULT as its own state with its own escalation path, because a check in FAULT is blind and a blind check is more dangerous than a failing one. An ALERT tells you something true about the world. A FAULT tells you that you currently know nothing about that part of the world, while the dashboard beside it stays green.

Most tooling we have used collapses this distinction into “no data”, which sorts visually next to “fine”. Our rule is that two consecutive FAULTs on the same check escalate to a named engineer, and a FAULT older than a week is a ticket with a due date. The most common cause, by a wide margin, is a read-only credential that rotated or lapsed.

The scheduler watches itself

A monitoring platform that stops running is worse than no platform, because everyone has stopped looking for the things it used to catch. The scheduler writes a heartbeat to an endpoint hosted outside its own account and region. If that endpoint sees no heartbeat inside the expected interval, it emails us directly. The heartbeat watcher is intentionally the dumbest component we own: no queue, no database, no model, a timestamp and a comparison. It has never needed a change since we wrote it.

Where the LLM sits

The AI in this platform does one job: it turns structured findings into an email a person will read on a phone at 7am. A daily digest per client contains a machine-generated table of every check and its state, and above that, a short prose summary written by a model from the findings payload.

Two rules govern that step. The model never decides state, because state is computed in code from the check’s own data and is auditable. And the model never removes a finding; the table below the prose is generated without it, so a summary that misses something can be caught by reading three lines further down.

We keep the summarisation call behind a thin interface with one method, because the vendor surface underneath it moves on a monthly cadence. In September alone, Anthropic merged its Cowork agentic product back into a single Claude product (Anthropic), and TypeSafe launched a model family built for fast structured decisions at a fraction of per-token cost (TypeSafe). Cheap structured classification is tempting for triage work, and we may use it for SIEM noise reduction. We are not moving state determination out of code for it, because an Australian client asking why a backup was reported healthy needs an answer that points at a timestamp comparison rather than a probability.

Why we own it rather than buying six tools

The argument for point tools is that each one is better at its slice than anything we would write. That is true. The argument against is that the slices are where the problems live. A backup tool knows the jobs it runs; it does not know about the VM that a cloud posture check found last week and that nobody added to the backup policy. Answering that question needs both result sets in one store, which means owning the store.

There are second-order reasons. Per-endpoint SaaS pricing across a managed client base gets expensive fast, and it charges you most for the clients with the most infrastructure, which are the ones where an extra check costs us an afternoon. Our escalation rules encode agreements we have made with specific clients, not a vendor’s severity taxonomy. And a small senior team can now maintain considerably more code than it could five years ago: agent orchestration is a shipped product category rather than a demo, with tools built to run several coding agents together on one codebase (Castforge). The Servo project’s one-year retrospective on funded part-time maintenance is a useful analogue for the ownership question, with one engineer’s sustained attention producing 1,150 reviewed pull requests and eight new maintainers over twelve months (Servo). Sustained attention from a small number of people is what makes platform ownership work, and it is also the thing most organisations underestimate when they decide to build.

What it costs to run

The infrastructure is a single small VM, a queue, a database and an object store for check history. That bill is smaller than one seat of most commercial monitoring products. Model inference is the smallest line item on the platform, because we only send the findings, not raw logs: a daily digest is a few thousand tokens of input.

The real cost is engineering attention, and it runs at roughly a day a month in steady state, with spikes when a vendor changes an API or a device family gets a new firmware scheme. The first six months cost considerably more than the SaaS tools we replaced. If your team does not have someone who will own this at that level for years, buy the tools instead.

Where it falls short

Checks run on schedules between fifteen minutes and daily, so this is not uptime monitoring and it is not EDR. It sits alongside both and answers a different question: is the configuration of this estate still what we agreed it would be. It has nothing to say about a site that went down four minutes ago.

The summaries flatten nuance sometimes, which is why the generated table stays in the email. And every check is only as trustworthy as the credential behind it, which is the argument for taking FAULT seriously. Under the OAIC’s Notifiable Data Breaches scheme the clock starts when you suspect a problem, so the time you spend discovering that a check has been blind for three weeks is time you cannot get back.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.