Scanning public health signals: outbreak and drug safety news monitoring

The first post in this series described a daily digest: a scheduled job that reads a wide set of sources, summarises them, and emails one message to one group. It works because nobody is relying on it for anything time critical. If it misses something, a person notices next week.

Surveillance is a different job with the same plumbing. You are watching a small number of official feeds where almost nothing happens, and the item that matters is buried among a hundred routine ones. Miss the recall notice for a product your pharmacy stocks and the cost is real. Send twelve false alarms a week and people stop reading, which produces the same outcome by a slower route. This post is part of our Practical AI in Health series, and it covers the version of the digest we build when the output has to be trusted.

The feed list is the hard part

Most of the effort in these projects goes into sources, not models. A typical starting list for an Australian health service or aged care provider:

• WHO Disease Outbreak News, for international events that may affect returning travellers or supply chains.
• TGA safety alerts, recalls and medicine shortage notices.
• The TGA’s Database of Adverse Event Notifications, if you want to watch specific products over time rather than react to published alerts.
• Communicable disease and health alert pages published by state and territory health departments.
• Sponsor and manufacturer recall notices for the products you actually stock.
• Professional college and peak body bulletins, which often carry practice advice before it reaches a government page.

Some of these publish RSS. Some publish an HTML list that changes structure once a year without warning. A few give you nothing but a page you have to scrape. Budget for that: feed breakage is the main ongoing maintenance cost of a monitor like this, well ahead of anything to do with the model. We use Centazio, our open source integration platform, for the fetch and normalise layer, mostly so that each source is an independently testable reader rather than a script that quietly dies.

Pipeline

The shape is deliberately boring:

• Fetch each source on a schedule. Hourly is fine for almost everything here. Nothing in this domain moves fast enough to justify polling every minute.
• Normalise to a common record: source, url, published date, title, body text, retrieved date.
• Deduplicate. Hash on url, then fuzzy match titles within a rolling window, because the same recall appears on three sites in slightly different wording.
• Pre-filter cheaply. Keyword and embedding similarity against your own product list, service list and site locations. This throws away the clearly irrelevant before you spend a model call on it.
• Classify what survives, with a structured output schema.
• Route to a named owner queue.
• Log everything, including the items you discarded, with the reason.

That last point matters more than it sounds. The discard log is what lets you answer “why didn’t we hear about this?” three months later, and it is the only way to measure the failure mode you cannot see.

The classifier returns something like this:

{
  "category": "medicine_recall",
  "relevance": "act | review | log",
  "confidence": 0.0,
  "affected_products": ["..."],
  "affected_sites": ["..."],
  "evidence_quote": "verbatim sentence from the source",
  "source_url": "https://..."
}

The evidence_quote field is not decoration. Every alert that reaches a human carries the sentence it was based on and a link to the original page, so the reviewer can confirm or dismiss in about ten seconds without opening a browser tab. Summaries that cannot be traced back to a source sentence do not get sent.

Two thresholds, not one

The instinct is to tune one threshold until the alerts feel about right. That never settles, because the two errors have different owners. A missed recall is a clinical governance problem. A noisy inbox is an operational one, and the people who suffer are not the people who set the threshold.

So we grade into three buckets rather than two. Items above the high bar page the owner. Items in the middle land in a review queue that somebody works through once a day. Everything else is logged and searchable. GitHub took the same approach with npm publish time malware scanning, where packages are either published, held for manual review, or blocked, with a separate metadata category for legitimate code that looks malicious to a scanner. Ambiguous hits get labelled and routed instead of being forced into a yes or a no.

Run the recall side high and you will get odd results. Apple’s on device iMessage classifier, tuned to catch nudity, flagged a video of a dog lying on its back as sensitive content. A classifier looking for medicine names in news text will do the equivalent: flag an article about a share price, a sponsor’s marketing announcement, an overseas alert for a product with the same trade name and a different formulation. None of those are bugs you can fix with a better prompt. They are what high recall costs, and the review queue is where you pay it.

Where the thresholds sit is an operational decision, not a technical one. Practitioners tuning scanners in build pipelines end up in the same argument about how much noise a team will tolerate. Ask who is on the receiving end and how many items a day they can genuinely triage, then set the numbers to fit that person’s capacity.

Routing and sign off

Category maps to owner. Medicine recalls to pharmacy. Device recalls to biomedical engineering or procurement. Communicable disease alerts to infection prevention. Anything unmatched goes to a named fallback owner, never to a shared mailbox that belongs to nobody.

Two design requirements we hold to on this kind of build. First, the system never contacts a patient, a clinician or a ward directly. It drafts and it routes. A named person reads the source, decides, and sends. Second, nothing the model produces is treated as a clinical judgement. This is administrative monitoring: has something we hold, dispense or use been the subject of an official notice. Whether that changes anyone’s care is a decision for the clinician who signs off, and the workflow is built so it cannot be skipped rather than warned against in a policy document.

One useful consequence of the design: the inputs are all public web pages. No patient information enters this pipeline, which removes most of the privacy work that dominates other health AI projects and makes it a reasonable first build for an organisation that has not run one before.

Measuring it

Build a small gold set before you go live. Twenty or thirty past notices your team should have acted on, plus a hundred or so items they should have ignored, all with the correct answer recorded. Replay it whenever you change a prompt, a threshold or a model version. It takes a couple of days to assemble and it is the only thing standing between you and a monitor whose accuracy nobody can describe.

Then sample the discard log weekly. Twenty random items, read by a human, looking for anything that should have surfaced. False positives announce themselves. False negatives only appear if you go looking.

Costs and limits

Inference is the cheap part. A pre-filter that drops most items before the model sees them keeps volume in the tens or low hundreds of classifications a day, and the model bill sits well below the cost of the engineer maintaining the source readers. Expect the real budget to go on the initial source mapping, the gold set, and the ongoing review time you are asking somebody to spend.

The limits are worth stating plainly. This monitors published notices, so it is always downstream of whoever publishes. It cannot tell you about an emerging problem nobody has written up. It will misread ambiguous product names, and it degrades quietly when a source changes its page layout, which is why every reader needs a heartbeat check that alerts when a feed goes silent for longer than usual. Silence from a feed is not the same as good news.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Tagged: #public-health-surveillance #drug-safety #alerting #llm-classification #integration