Mining patient feedback and complaints for themes

Most health services in Australia are sitting on several years of patient comments that nobody has read from start to finish. The Likert scores get tabulated and put in the quarterly pack. The free text gets skimmed by whoever is assembling that pack, and the rest is archived. This post is part of our Practical AI in Health series, and if you are looking for a first AI project that is cheap, visible to your board and nowhere near a clinical decision, this is the one we usually point people to.

The data is already sitting there

If you run an Australian hospital you are probably already collecting AHPEQS, the ACSQHC’s 12-item patient experience question set, which includes a free-text comment field. One implementation study covered 86,180 surveys across 36 private hospitals over 18 months. Nothing new needs to be collected for a theme-mining project. The corpus exists.

The volumes are worth knowing before you scope anything. In a provincial health system’s inpatient survey analysis, 43.4% of adult patients and 46.9% of paediatric caregivers left a free-text comment, and topic modelling on those comments produced 86 adult and 35 paediatric topics, including elements of care that no existing survey question asked about. That last part is the argument for doing this at all. Closed items can only measure what someone thought to ask.

A single hospital’s 2023 feedback corpus held 126,134 entries, 92,578 of them with free text, totalling about 1.57 million tokens at an average of 16.9 words per comment. Short comments, high volume. Trivial for a machine to process, impractical for a person to read.

Complaints are a separate stream and a more sensitive one. Ahpra received 13,327 notifications about practitioners in 2024/25, up 19% on the year before, with clinical care at 35% of all notifications and communication the second-largest issue at 4,313. Communication problems are exactly the kind of signal that shows up in free text months before it shows up anywhere formal.

Why this is the low-risk starting point

The Commonwealth’s Safe and Responsible AI in Health Care legislation and regulation review found that consultees considered the current regulatory system not fit for purpose, and canvassed mandatory guardrails for high-risk settings. Analysing what patients wrote about waiting times and car parking is not a high-risk setting. There is no diagnosis, no triage, no treatment recommendation, and no patient-facing output.

It also lands on an accreditation hook you already have to satisfy. NSQHS Action 1.13 requires you to seek patient and carer feedback regularly and use it to improve safety and quality, and companion Action 1.14 requires complaints analysis to be reported to the governing body, to drive improvement, and to have identified risks recorded in the risk management system. A theme-mining project produces evidence for all of that as a by-product.

Start with a taxonomy, not a topic model

The tempting approach is to run unsupervised clustering and see what falls out. Do that second. Start with the Healthcare Complaints Analysis Tool (Gillespie and Reader, BMJ Qual Saf 2016), which was built from a systematic review of 59 studies covering 88,069 complaints and distils 729 codes into three domains and seven problem categories: quality, safety, environment, institutional processes, listening, communication, and respect and patient rights. Each is graded across four severity levels, plus stage of care and harm.

Two practical consequences. First, raters found an average of 1.94 problems per complaint letter, so your classifier has to be multi-label. Single-label classification will quietly lose half the signal. Second, a published taxonomy means your numbers can be compared against other sites and across years, which an invented cluster set cannot.

The closest published analogue to what you would be building is a 2025 cross-sectional study where 1,816 anonymous complaints from seven public primary care clinics were coded by trained humans against HCAT(GP), then classified by GPT-3.5 turbo, GPT-4o mini and Claude 3.5 Sonnet. Accuracy ranged from 58.4% to 95.5% depending on the field, agreement with human coders varied a lot (Cohen from 0.114 to 0.623), and the authors concluded further fine-tuning was needed. Thematic analysis of the same corpus found long wait times (21.6%), staff attitudes (15.8%) and appointment booking issues (10.5%) accounted for nearly half of all complaints. Read that result as: good enough to prioritise reading order, not good enough to publish unreviewed.

The architecture

• Ingest from the survey platform, the complaints register and the feedback inbox into a staging store. We built Centazio for this kind of multi-source integration work, but any pipeline that gives you an immutable staging copy is fine.
• De-identify before anything leaves your network. Strip or pseudonymise patient names, clinician names, dates, room and ward identifiers, and keep the mapping inside your environment.
• Classify each comment against the HCAT labels with structured output, multi-label, with a confidence value and the text span the model relied on.
• Run unsupervised clustering as a second pass over comments the taxonomy handles poorly. That is where the categories nobody thought to ask about turn up.
• Aggregate into theme counts by site, ward and quarter, with sentiment trend lines and a simple alert when a theme’s share moves outside its historical band.
• Route low-confidence and high-severity items to a human review queue.

A per-comment output looks something like this:

{
  "comment_id": "c-2026-04-118322",
  "domain": "management",
  "categories": ["institutional_processes", "communication"],
  "severity": "medium",
  "stage_of_care": "in_practice",
  "harm": "minimal",
  "confidence": 0.62,
  "evidence": ["waited over two hours past my appointment time"]
}

Human sign-off is a design requirement here, not a caveat. No item reaches the risk register, a named individual, or a governing body report without a quality officer confirming the classification. The model decides what gets read first. A person decides what it means.

Privacy and governance

The OAIC’s guidance on privacy and the use of commercially available AI products, published in October 2024, is the document to work from. The Privacy Act 1988 and the APPs apply wherever personal information is used to train, test or run an AI system, APP 10 accuracy obligations are heightened in an AI context, and the OAIC recommends a Privacy Impact Assessment before you introduce a new system, along with alignment to the Voluntary AI Safety Standard. Re-identification through cross-matching datasets is called out as a key risk, which matters because patients routinely name a clinician, a ward and a date in a two-sentence comment.

Budget for this from day one. The NIHR co-design study demonstrated automated free-text analysis with near real-time dashboards, and then the planned NHS-wide roll-out stalled on information governance rather than technical failure. That is the most likely way this project dies here too.

Costs and honest limits

The model bill is the smallest line item. A corpus of a million or two tokens costs very little to classify at current API prices, even if you re-run it several times while tuning prompts. The real cost is human: a few hundred comments double-coded by two staff to build a validation set, which is two to three weeks of a quality officer’s time, plus the PIA and vendor review.

Some limits worth stating up front. Sentiment on 17-word comments is noisy and should be read as a trend, never as a per-comment score. Feedback volumes skew toward strong opinions, so theme counts are not incidence rates for the patient population. Severity classification is where model agreement is weakest, so treat it as a sorting aid only. And if you change model versions, re-run the back history, otherwise your trend line is measuring the model rather than the hospital.

Done properly, what you end up with is a quarterly view of what patients actually said, ranked by volume and severity, traceable back to individual comments, and defensible in an accreditation conversation.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Tagged: #patient-feedback #natural-language-processing #clinical-governance #privacy #large-language-models