AI in a real-time payments world: monitoring and anomaly detection that keeps up

A payment on the New Payments Platform clears and settles in seconds. The Reserve Bank’s RTGS system settles high-value payments one at a time throughout the day. Suppose your monitoring runs reconciliation overnight and emails a report at 7am. Then you hear about a duplicate payment almost a working day after the money left, and often after the customer has already rung to ask about it.

This post is part of our Practical AI in Banking series. It describes how we layer monitoring for real-time payments. Deterministic checks and statistical anomaly detection do the detection, and an LLM sits on top to triage operational alerts, so a small team works through a short, ordered queue. The design comes from a real-time banking platform PicNet has been building for an Australian payments provider over several years. The client stays anonymous here. The design choices are the ones we work with every day.

Why overnight monitoring falls behind

Batch reconciliation still has a job. It is the authoritative check that your ledger matches scheme settlement reports and your settlement account position. It makes a poor early warning system, because by the time it runs, a faulty retry loop or a stuck message queue has had hours to do damage.

Attackers have also sped up. Wiz’s incident response team describes AI-driven intrusions where activity reaches “hundreds or thousands of actions per hour” without the pauses of a human operator. It also notes that the footprint is spread across identity providers, cloud control planes, CI/CD systems and workload logs, so that “no single alert tells the full story” (Wiz). A payments platform runs on the same cloud plumbing, and detection that runs once a night gives that kind of attacker most of a day.

Customer-side traffic is changing as well. US banks have reported unvetted AI agents accessing customer accounts. Commenters argued for properly scoped agent tokens instead of customers handing their passwords to bots that screen-scrape (r/technews). Your monitoring now has to tell that traffic apart from ordinary customer behaviour.

Layer one: invariants checked on every payment

The first layer has no AI in it. Every payment event passes through deterministic checks that run in the stream within seconds:

• Each end-to-end identifier produces exactly one ledger posting.
• Debits and credits balance per batch and per settlement cycle.
• Status only moves through allowed transitions, so a settled payment can never return to pending.
• Every outbound payment gets a scheme response within its timeout, and every timeout is counted. Nothing is retried silently.

Duplicate postings need the most care. The FaultFence test harness for financial AI agents ships with 17 failure scenarios. One of them, PAY-004, simulates a payment that succeeds but loses its response, then checks whether the retry creates a duplicate ledger effect (r/AI_Agents). The author admits the scenario doesn’t yet check where idempotency is enforced. A commenter points out that you can’t properly simulate duplicate retries without handling idempotency keys. We agree with the commenter. Test the idempotency key directly, both in pre-production and as a live check.

The live version can be a simple windowed rule:

-- Any end-to-end id with more than one ledger effect in the last 10 minutes
SELECT end_to_end_id, COUNT(*) AS postings, MIN(posted_at), MAX(posted_at)
FROM ledger_postings
WHERE posted_at > now() - interval '10 minutes'
GROUP BY end_to_end_id
HAVING COUNT(*) > 1;

A hit on this rule pages the on-call engineer immediately and skips the triage layer described below.

Layer two: anomaly detection on payment flows

Invariants catch failures you can name in advance. Anomaly detection catches drift in measures that have no fixed correct value. Examples are volumes per channel per minute, rejection rates by reason code, latency to the scheme, value distributions per participant, and the share of payments going to newly created payees.

Baselines have to follow the calendar. Australian payment volumes move with pay cycles and month end. Public holidays differ by state, and business payments surge at the start of each day. A detector that ignores this will page someone every Monday morning.

Simple methods cover most of the ground. Two that work are seasonal decomposition with a rolling threshold on the residual, and quantile bands per series keyed by hour of week. In a recent r/sre discussion, practitioners argued that anomaly detection is still relevant now that AI agents exist. They were also surprised that tabular and time-series foundation models aren’t more widely used for it (r/sre). We would trial a foundation model on high-cardinality series, where hand-tuned baselines get unwieldy. We would keep the simple methods wherever they work, because an engineer can explain a quantile band to an auditor in two minutes.

Each detector emits an event with the series, the observed value, the expected range and a score. These events go to the triage layer. Only the invariant checks page anyone directly.

Layer three: LLM triage of operational alerts

A small team can’t read hundreds of anomaly events and infrastructure alerts a day. They need related alerts grouped and given context, and that is the job we give the LLM. For each cluster of alerts in a time window, the triage service assembles:

• the alerts with their scores
• recent deployments and configuration changes
• the matching runbook section
• current scheme and participant status notices

It then asks the model for a structured result that the service validates against a schema:

{
  "summary": "Rejections for one participant above expected range since 10:42",
  "likely_cause": "participant-side degradation; no matching internal change",
  "suggested_priority": "P2",
  "related_alerts": ["a-1931", "a-1934", "a-1940"],
  "runbook": "participant-degradation"
}

These rules are part of the design from the start:

• The model can group alerts, summarise them and suggest a priority. It can’t close or suppress an alert, and it has no tool access to anything that moves money.
• Invariant breaches never go through the model.
• An engineer acknowledges every P1 and P2. The model’s suggested priority is logged next to the engineer’s choice, so you can measure how often the model is wrong.
• Alert content counts as untrusted input. Payment references and payee names are free text written by outsiders, so the prompt treats them as data and the output must pass schema validation. FaultFence includes tool poisoning among its scenarios for the same reason.

Architecture sketch

• The core platform publishes payment events to a durable stream such as Kafka or a managed equivalent, with the end-to-end identifier on every message.
• Stream processors run the invariant checks and write breaches straight to the paging system.
• Per-minute aggregates land in a metrics store, and detectors run against them every minute.
• A triage service clusters alerts, calls an LLM hosted in an Australian region, validates the response, and posts to the team’s chat channel and ticket queue.
• Every alert, model input, model output and human decision goes to an audit store.

Lock down the admin tooling around the stream as tightly as the stream itself. One of the confirmed entry points in the Wiz investigation was a remote code execution flaw in an exposed Kafka UI instance.

Costs and limitations

• Anomaly detection produces false positives, especially in the first months while baselines settle. Budget engineering time for threshold tuning, and retire any detector nobody acts on.
• LLM triage costs money for every alert cluster, and that cost grows with alert volume. FinOps practitioners are still asking how to track the real cost of each AI agent workflow (r/FinOps), so tag every triage call with a workflow and cluster ID from day one.
• The model will sometimes be confidently wrong about the cause. The design accounts for this, which is why the model only suggests and the engineer makes the call.
• For APRA-regulated entities, CPS 230 brings the monitoring stack and the LLM provider into operational risk and service provider management. The audit trail and the hosting location both need to stand up to review.

Where to start

Start with invariants. They cost little to run and catch the failures that hurt most. Next, add anomaly detection on a handful of flow metrics. Put an LLM in front of the alert queue once alert volume is high enough that people have started ignoring it. Before you let the model’s suggestions order the queue, check them against what your engineers decided.

If your real-time payments still rely on an overnight batch for monitoring, talk to us about what a first project could look like. PicNet builds production AI systems for Australian organisations.