Summarising clinical and referral documents with AI

Every health service we talk to has the same pile: referrals arriving as faxed PDFs, scanned handwriting and email attachments, plus discharge summaries written at the end of a long shift. Someone has to read all of it and pull out the same dozen facts every time. It is slow, it is boring, and things get missed.

The missing part is documented. A retrospective audit of a tertiary medical oncology unit against the Australian National Guidelines found that “procedures” were absent from 82% of discharge summaries, “ceased medicine” from 38% and “information provided to the patient” from 8%. Follow-up arrangements were recorded in 91% of summaries, but exact appointment dates were often missing. That is the problem worth attacking, and it is an administrative one.

This post is part of our Practical AI in Health series. Here I want to cover how we build document summarisation that survives contact with a clinical governance committee.

Extract first, then write prose

The most common mistake is to hand the whole document to a model and ask for a summary. You get fluent paragraphs that nobody can check against anything.

The alternative is to treat the job as structured extraction. Break the document into segments, pull named entities and relations out of each one, map the results to a schema you already use, and only then generate readable text from the structured record. A published pipeline doing exactly this (OCR, chunking, entity and relation extraction, then mapping to FHIR R4 and OMOP) reported roughly 95% accuracy and F1 across seven medication data fields covering 5,789 extracted medications, though validation was on only 34 patients. Small cohort, but the design point stands: fields you can validate beat prose you cannot.

A working architecture looks like this:

• Ingest and OCR, with the original page image retained for every extracted value.
• Deterministic segmentation into sections before any model sees the text.
• An extraction pass that returns a fixed schema and refuses to emit a value without a supporting span from the source.
• Validation against terminology sets, date logic and internal consistency (a ceased medicine that never appears in the admission list is a flag, not a fact).
• A completeness check that lists which required fields were not found, because absence is the failure mode you will hit most.
• A narrative pass restricted to the validated structure, never the raw document.
• A review screen where the clinician or intake officer sees each field next to the quoted source text, approves or corrects, and signs off.

Our extraction output looks something like this:

{
  "field": "ceased_medicines",
  "value": ["metoprolol 25 mg BD"],
  "evidence": [{ "doc_id": "dc-2291", "page": 2,
                 "quote": "Metoprolol ceased 14/06 due to bradycardia" }],
  "confidence": 0.61,
  "status": "needs_review"
}

No span, no field. It is a boring rule and it removes an entire class of argument later.

What actually goes wrong

Fabrication gets the headlines; omission is the more frequent problem. A vendor-neutral evaluation covering 18 experimental configurations and 12,999 clinician-annotated sentences measured a 1.47% hallucination rate and a 3.45% omission rate. The same paper’s baseline is worth keeping in mind when someone demands perfection: human-written notes average about 1 error and 4 omissions each, and an earlier BART-based model produced 3.9 errors and 6.6 omissions per note. Your review process should ask “what did it drop?” at least as often as “what did it invent?”

Grounding is not a nice-to-have. When researchers built 300 doctor-designed vignettes and seeded each with a single fabricated lab value, sign or disease, models repeated or elaborated the planted error in up to 83% of cases, with an overall hallucination rate of 65.9% under a default prompt. A mitigating prompt brought that down to 44.2%. Two things follow. Prompt engineering helps and is nowhere near sufficient. And referrals contain errors already, so any pipeline that free-runs over the source text will confidently launder those errors into your record.

Confidence flags earn their place, but be honest about what they are. A model’s self-reported confidence number is close to meaningless on its own. What works in practice is agreement between two independent extraction passes, whether a matching source span was found, and whether the value validates against a terminology set. Combine those, set a threshold, route everything below it to a human. Everything above it also goes to a human, just with less friction.

Structure the referral, let a clinician triage it

There is a temptation to go one step further and have the system assign urgency. Queensland Health researchers tried automated categorisation of 17,378 ENT referrals from two hospitals between 2019 and 2022 against the Clinical Prioritisation Criteria, and achieved 53.8% agreement with the actual assigned triage category. Extraction of referral content is a solved-enough engineering problem. Assigning urgency is not, and the clinician making that call needs to be a design requirement, not a disclaimer in the footer.

Keeping health information onshore

The OAIC’s October 2024 guidance recommends as best practice that organisations do not enter personal information, particularly sensitive information, into commercially available AI products, and flags APP 3, 6, 8 and 10 as live obligations. Health information is sensitive information. An offshore API call is a cross-border disclosure that the health service remains accountable for. The ACSQHC’s ambient scribe scenario is blunter still: the data must be stored and processed in Australia unless the patient has given explicit consent for offshore handling.

That leaves three practical options. A commercial API deployed in an Australian region with contractual guarantees on retention and training is the cheapest to run and the heaviest on procurement and privacy paperwork. Open-weight models on infrastructure you control give you residency, version stability and no vendor changing the model underneath you, in exchange for fixed GPU spend and owning your own evaluation. A hybrid puts a smaller local model on extraction and de-identification, with anything larger only ever seeing de-identified text, which adds moving parts and relies on de-identification you should not assume is perfect.

On cost, per-token spend is usually the smallest line item. For a few thousand documents a month it is rounding error next to the clinician review time and the validation work. Self-hosting shifts cost to fixed GPU hours, which pays off at volume or when residency requirements make the decision for you. Budget most of your money for evaluation, not inference.

Validation does not travel

Victoria’s May 2025 sector advisory on ambient AI scribes sets a minimum implementation standard and states plainly that a solution evaluated in one context cannot be assumed to transfer between clinical settings, or from an overseas training context to Australian practice. Good accuracy on GP referrals tells you little about oncology or maternity. Budget for per-specialty validation with your own documents and your own clinicians.

Watch your roadmap too. TGA regulation is technology-agnostic and turns on intended purpose. A tool that drafts documentation is generally not a medical device, but the TGA’s own worked example is a scribe that later gains a feature suggesting diagnoses or treatments not mentioned in the consultation, at which point it is regulated. The TGA published findings on strengthening medical device software regulation in July 2025 and has further consultation running through 2025 and 2026, so the boundary is being actively redrawn.

The ACSQHC guidance also asks for consent processes, summaries that meet medico-legal requirements, records labelled to show AI was involved, and regular performance review. Build the labelling and the audit log in from the first sprint. Retrofitting them is miserable.

Australian deployment evidence exists but is early. Gold Coast Hospital and Health Service ran a 16-week ambient scribe trial across outpatient specialties in 2024, evaluated through patient and staff surveys, interviews, scribe outputs and eMR review, and reported positive staff and patient experience alongside note quality and efficiency gains. Useful, and outpatient-weighted. Treat it as a reason to run your own measured pilot, not as proof the problem is solved.

Start with one document type and one clinic. Measure omissions against a clinician-reviewed sample before and after. If it does not reduce the fields that go missing, it is not worth deploying.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Tagged: #clinical-documentation #referrals #data-residency #llm #health-informatics