Governing AI in Australian health: privacy, PHI and safe adoption

Most AI conversations we have with health organisations stall at the same point. Not “does it work”, but “where does the patient data go, who is accountable when it goes somewhere it shouldn’t, and who signs the output”. Those are the right questions, and “the vendor says it’s encrypted” is not an answer that survives an audit or a board paper. This post, part of our Practical AI in Health series, sets out how we approach governance on projects that handle protected health information, including the parts that are genuinely awkward.

Start with the data, not the model

Before anyone compares models, write down what the system will actually send. Which fields, from which source, about whom, and to which processor. We do this as a table on a whiteboard, and it usually shrinks the problem.

A surprising share of health AI work is administrative and needs far less identifiable data than the first design assumes. Sorting a referrals inbox, drafting a discharge summary skeleton from structured fields, matching duplicate patient records, extracting values from a scanned form, coding free text against a reference list. Several of these can run on a de-identified payload, with identifiers rejoined inside your own network after the model returns.

Under the Privacy Act and the Australian Privacy Principles, health information is sensitive information, and sending it to a model provider is a disclosure, not merely storage. If that provider processes offshore, APP 8 and the cross-border rules come into play and you remain accountable for what the recipient does with it. Ask your privacy officer to write the organisation’s position down before the proof of concept starts. Retro-fitting a privacy impact assessment onto a pilot that clinicians already like is the worst possible sequence, and we have watched it happen.

Residency and model choice

Three broad options, with honest trade-offs.

• Hosted model API in an Australian region. Cheapest per unit of work, best quality, fastest to build. The catch is that your account being in Sydney does not guarantee inference runs in Sydney. Ask which region serves the model, whether prompts and outputs are logged, for how long, and whether zero retention is contractual or a toggle someone can flip back.
• Hosted API offshore. Sometimes the only way to get a specific model. Requires an explicit APP 8 position, disclosure to patients where relevant, and a clear view of sub-processors. We would not put identified clinical records through this without legal sign-off in writing.
• Open weight models on infrastructure you control, in your cloud tenancy or on premises. Full control of residency and logging. You pay for it with a fixed GPU bill whether the system is busy or idle, plus the ops work of running inference yourself, plus more prompt engineering and more human review because smaller models need it. In our experience this only makes financial sense at high, steady volume, or when the residency requirement is absolute.

The middle path we use most often is the de-identification boundary: strip identifiers before the call, keep a mapping table in your own database, rejoin afterwards. It is cheap and it works well for structured data. It is not a silver bullet, because clinical free text is full of names, addresses and dates that no regex will reliably catch. Where free text is involved, either accept that the payload is PHI and govern it accordingly, or keep the model inside your boundary.

The disclosure nobody puts in the risk register

The model call is not the only place data leaves the building. A worked example from an open source medication tracking app is instructive: alongside rewriting its privacy policy to cover the Australian Privacy Act, GDPR Article 9 health data, PIPEDA and others, and to call out international AI data transfers as a separate disclosure item, the same change removed fonts loaded from Google’s CDN, which had been sending every visitor’s IP address to a third party without disclosure.

Fonts. In a health app. Nobody puts that on a risk register, and yet it was a real undisclosed transfer that contradicted the app’s own privacy policy. The lesson generalises: your privacy impact assessment should inventory every outbound network call the application makes, including analytics, error reporting, session replay, map tiles and CDN assets, not just the AI endpoint. That same project also treats international AI data transfers as a distinct thing patients are told about, which is a reasonable pattern for Australian providers to copy given APP 8.

Human sign-off is an architectural requirement

We keep our health AI work administrative and operational. It does not diagnose, triage clinical urgency, or produce anything a patient sees without a person in between.

Where output is clinically adjacent, for example a drafted summary that will eventually sit in a record, the review step has to be built into the system rather than written into a policy document. In practice that means the model output is created in a draft state that cannot be transmitted, filed or actioned until a named person approves it. The approver’s identity and timestamp are stored with the record. Rejecting or editing is as easy as accepting, and the prompt, the input and the raw output are logged so an incident can be reconstructed months later.

Then measure the review. If your audit log shows reviewers approving in under two seconds with a zero per cent edit rate, you do not have oversight, you have a click-through. We treat edit rate as a live quality metric, and a sudden drop is a signal to go and talk to the users.

What to ask a vendor before you sign

These are the questions we get asked, and the ones we ask on our clients’ behalf when we are reviewing someone else’s product:

• In which country is our data processed, and in which country is it stored at rest? Not “where is the company”, where does inference physically run?
• Is our data, including prompts and outputs, used to train or improve any model? Is that a contractual commitment or a configuration setting?
• What is the retention period for prompts, outputs and logs, and how do we trigger deletion?
• Who are your sub-processors, and how are we notified when the list changes?
• How does the product enforce human review, and can that enforcement be turned off by an administrator?
• What is logged for audit, how long is it kept, and can we export it into our own SIEM?
• What is your notification timeframe and process for a suspected breach, and does it meet our obligations under the notifiable data breaches scheme?
• If the tool influences clinical decisions, what is your regulatory position on medical device classification? Worth raising with your regulatory advisor early, because it changes the project shape.
• What happens to our data if we terminate, and what does the exit look like?

A vendor who cannot answer the first three in a sentence each is not ready for health data.

Where PicNet sits

We are a small senior team in Sydney, and every manager here is an engineer, which means the person answering these questions has usually built the thing. We have delivered large traditional systems for some government contracts under strict constraints, including work where AI was excluded entirely, so we are comfortable building inside tight boundaries rather than treating them as an obstacle. Our open source integration platform, Centazio, handles the unglamorous plumbing that most of these projects turn out to be: moving data between systems reliably, with an audit trail.

The honest limitations. A contract clause does not make an offshore API compliant, it only allocates blame. De-identification reduces risk, it does not eliminate it. Human review costs clinician time, and if the workflow does not save more time than the review consumes, the project is not worth doing. We would rather tell you that in week one than in month six. Start with one narrow administrative process, measure it, and decide from evidence.

PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.

Tagged: #privacy #governance #health-it #data-residency #procurement