Predicting appointment no-shows: what actually works in an Australian clinic

Every conversation about AI in health at the moment is about language models. Meanwhile the problem that most reliably wastes clinic capacity, the patient who does not turn up, is best solved with a technique that has been sitting on the shelf since about 2015: gradient boosted trees on a table of appointment records.

We have been building these models at PicNet since well before the current wave. Old posts on this blog about XGBoost still get fetched by AI assistants, which is mildly funny, because the underlying advice has not changed much. Tabular data, careful feature engineering, honest evaluation. What has changed is that health services now have the appointment history and the reporting infrastructure to make it worth doing. This post is part of our Practical AI in Health series, and it is the one where the answer is deliberately unfashionable.

Measure your own base rate before you model anything

The single most common mistake is assuming a no-show rate from the literature. Australian did-not-attend rates vary enormously by service and location. A regional medical imaging department recorded 724 failures to attend across 13,458 appointments, a rate of 5.4%. A surgical outpatient clinic in Mount Isa recorded 18.3% across 6,267 appointments. Those two services need completely different models, different thresholds and, frankly, different business cases.

There is a related trap in the published benchmarks. A review of 52 no-show modelling papers from 2010 to 2025 found that logistic regression appears in 68% of them, that the best models land between AUC 0.75 and 0.95, and that a large number of studies re-use the same 2015-16 Brazilian dataset (Sciencedirect). If a vendor quotes you an accuracy figure without naming the population it was measured on, assume it does not transfer.

So step one is not a model. It is a query against your PAS or booking system that gives you attendance by clinic, by modality, by referral source and by month for the last two or three years. If that query is hard to write, fix that first. It is also the data you need for the business case.

Features that earn their place

The useful features are mostly logistical rather than clinical. The MRI deployment we keep coming back to below used 21 categorical and numeric fields pulled straight from the radiology information system and appointment system, with no patient medical conditions at all (AJR). Typical shape:

• Lead time between booking and appointment, which is usually the strongest single signal
• Prior attendance history for the patient, including counts of previous no-shows and cancellations
• Number of times the appointment has been rescheduled
• Appointment day, hour and session
• Modality or procedure type, and the ordering department or referring practice
• Appointment modality, since telephone and telehealth consultations were associated with better attendance in the Mount Isa cohort (adjusted OR 0.57 and 0.62)
• A travel proxy such as distance band or Modified Monash Model category

Two notes on the last one. Distance genuinely predicts non-attendance, and it is also a measure of how hard your service is to reach. In the Mount Isa data, Indigenous status was the strongest independent predictor of non-attendance at an adjusted OR of 3.72 (95% CI 3.22 to 4.29), which the authors attribute to systemic and logistical barriers rather than anything about the patients. A model trained on those fields will happily learn that access inequity is patient risk and present it back to you as a risk score. Decide up front what you will do with that, because it determines whether the project improves equity or entrenches it.

Class imbalance: do less than you think

At a 5% base rate the instinct is to reach for SMOTE or class weights. Resist it. A real-world evaluation of imbalance correction found that cost-sensitive learning, random over- and under-sampling and SMOTE did not improve AUROC or AUPRC, degraded calibration, and inflated the proportion of patients classified as high risk by up to 62.8%. The naturally trained model had the highest net benefit in decision curve analysis across clinically relevant thresholds (medRxiv).

Calibration is what you actually need, because the output feeds a capacity decision. Report AUPRC and calibration measures such as Brier score and log loss alongside AUROC, and set the operating threshold from the cost ratio of the intervention rather than defaulting to 0.5. Something like this, with your own numbers:

# your finance team's numbers, not ours
cost_of_reminder_call = 5.00     # a few minutes of admin time
cost_of_empty_slot    = 250.00   # idle clinician and equipment

# indifference threshold, before capacity limits
threshold = cost_of_reminder_call / cost_of_empty_slot   # 0.02

In practice the threshold you deploy is set by how many calls your admin team can make in a day, not by the arithmetic. Which is fine. Rank by predicted probability, take the top N you can service, and use the cost ratio as a sanity check on whether N is too small.

What good accuracy actually looks like

A deployed outpatient MRI model built with XGBoost on 32,957 appointments, at a 17.4% no-show rate, achieved a ROC AUC of 0.746 with an optimised F1 of 0.708, precision 0.606 and recall 0.852 (AJR). That is a long way from the 0.90+ figures you see quoted in richer-data studies, and it was enough. Human attendance behaviour is noisy and a lot of the variance is unrecorded: a lift home fell through, a shift changed, the letter went to an old address.

If your first model lands around 0.72 to 0.78 AUC, you have a normal, usable model. Chasing 0.85 with more features is usually worse value than spending the same effort on what happens after the score is produced.

The model is half the system

The same MRI team used the score to rank appointments and place a phone reminder to the top 25% one business day out, run over six months. No attempt to classify every appointment as show or no-show. That framing matters because it turns a probabilistic output into a fixed daily work queue for the people who actually make the calls.

On which intervention to attach, the JAMIA rapid review of model-triggered interventions is the most useful source available. It found high-certainty evidence that model-triggered SMS reminders reduce no-shows (median RR 0.91), moderate certainty for phone calls (median RR 0.61) and patient navigators (RR 0.55), and uncertain evidence for model-driven overbooking. The same review found almost no full cost-effectiveness analyses, no formal acceptability evaluation, and only half of trials reporting race or ethnicity, so effect variation across equity-relevant groups could not be assessed.

Overbooking deserves a specific warning. Samorani and colleagues show what happens when you pair an accurate classifier with a cost-minimising scheduler: the highest-risk group is systematically assigned the slots with the longest waits, and dropping demographic fields does not fix it, because postcode, language and prior utilisation act as proxies (PMC). Their recommendation is to decouple the prediction from the scheduling objective.

There is a workable alternative. A US health system running a commercial EHR’s built-in no-show predictor rejected the overbooking action as ethically problematic and used the score to trigger patient-positive responses instead: flexible appointment times, telehealth options, help with transport and childcare. Across a 12-clinic pilot they saw a 9% mean reduction in no-shows (Health Affairs). Same model, different action, different distributional consequences.

Whatever the action, a person stays in the loop. The score selects who gets contacted and offered support. It never cancels, deprioritises or reassigns an appointment on its own, and any output that touches a clinical pathway goes to a named human for sign-off. Build that in as a design requirement rather than adding it as a caveat later.

The regulatory boundary

Keep the scope administrative. Software intended for the prediction or prognosis of disease can meet the definition of a medical device under s41BD of the Therapeutic Goods Act, and the Clinical Decision Support Software exemption (in force since 25 February 2021, Schedule 4 Part 2 of the Therapeutic Goods (Medical Devices) Regulations 2002) requires all three of its criteria to be met, plus notification to the TGA within 30 working days of supply and ongoing compliance with the Essential Principles (TGA guidance). A model that predicts whether someone will attend an appointment is not predicting disease. A model that starts inferring clinical deterioration from attendance patterns is a different product, and you should get advice before you build it.

What a first project looks like

Six to ten weeks is realistic for one clinic or modality. Two weeks on data extraction and base-rate analysis, two to three weeks building and validating the model, and the remainder on the daily work queue, the calibration monitoring and the operational rules for who calls whom. The build is not expensive. The part that decides whether it works is the intervention design and the willingness to look at how the benefit is distributed across patient groups after six months.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Tagged: #machine-learning #xgboost #patient-scheduling #predictive-analytics #health-it