Entity resolution at scale: deterministic first, LLM for the tail

Most organisations we work with hold the same customer three or four times. The CRM has one copy under a trading name. The finance system has another under the registered company name, and a third came in from a web form with a typo in the email address. Any AI project that reads across those systems picks up the duplicates. Ask a model to summarise “the customer’s history” and it will give you a confident summary of a third of it.

This post is part of our AI-Ready Data and Integration series. It describes the matching pipeline we recommend. Exact and rule-based matching handles most records, and an LLM proposes matches for the ambiguous pairs left over. Before any merge above a set risk level is applied, a person approves it.

Why rules go first

A rule runs in microseconds and costs almost nothing. When a rule merges two records you can point to the reason, and someone will eventually ask why their account history changed. An LLM call is slower and costs money every time. It can also give a different answer to the same pair on a different day.

Volume is the bigger problem. Comparing every record with every other record means n(n-1)/2 comparisons, which comes to roughly 500 billion pairs for a million records. No model budget covers that. The deterministic stages exist to cut that number down to something a model and a review team can handle.

Stage one: normalise, block, match

Most matching failures come from formatting, so normalise before you compare anything:

• Names: case, whitespace and punctuation, with legal suffixes removed (“Pty Ltd”, “Pty. Limited”, “P/L”).
• Addresses: standard street types (St, Street), unit and level notation, and “c/-” care-of lines moved out of the street field.
• Phone numbers: converted to E.164 (+61…), which makes landlines and mobiles comparable across systems.

Next, match exactly on strong identifiers. For organisations that usually means the ABN. One ABN can cover several branches or trading names, though, so we pair it with a name or address rule and don’t merge on ABN alone. For individuals, a verified email or a customer number shared between systems does the same job.

For the remainder, blocking keeps the comparison count manageable. You only compare records that share a cheap key, such as postcode plus the first few letters of the normalised name. Within each block, you score pairs with string similarity on names and token overlap on addresses. Open-source tools such as Splink, which implements the Fellegi-Sunter probabilistic model, and Zingg handle this layer well. Neither needs an LLM.

The output falls into three bands. Pairs in the confident-match and confident-non-match bands are settled. Only the grey band between them goes to the next stage.

Stage two: an LLM for the grey band

The grey band holds the pairs rules handle badly: “Bob’s Plumbing” and “R. Smith Plumbing Services” with the same mobile number, a nickname against a full name, a business that moved two suburbs over. An LLM reads both records side by side the way a person would. It can also use free-text notes and order descriptions that are hard to write rules for.

We ask for structured output with three possible decisions (match, no_match or unsure), the fields the decision relied on, and a one-line reason. The model only proposes. It never writes to the master record. The routing logic stays deterministic:

def route(pair, verdict):
    if pair.abn_match and pair.name_score > 0.9:
        return "auto_merge"
    if pair.score < LOWER_BAND:
        return "no_match"
    if verdict.label == "match" and pair.risk_tier == "low" and band_precision(pair) >= 0.98:
        return "auto_merge"
    if verdict.label == "no_match" and pair.risk_tier == "low":
        return "no_match"
    return "human_review"

Look at band_precision. Don’t route on the model’s own stated confidence, because nothing guarantees that a “0.9” from an LLM means 90 per cent. Route on how accurate the model has proven to be for that score band against your labelled data (covered below).

Stage three: people confirm risky merges

Set the risk threshold by consequence, not by score. Merging two prospects on a marketing list does little harm if it’s wrong. Merging two customer accounts with credit balances, or two people’s personal details, is another matter. Australian Privacy Principle 10 requires reasonable steps to keep personal information accurate. A wrong merge that shows one person’s details to another can turn into a privacy incident, and you may have to assess it under the Notifiable Data Breaches scheme.

Design the review step as part of the pipeline from the start:

• A queue that shows both records, the fields that differ, and the model’s reason.
• Merges that can be undone. Keep the source records and a crosswalk of system IDs, and log every merge decision with who or what made it.
• Reviewer decisions fed back into the labelled set, so each week of review improves your measurements.

Measure on your own data before trusting any matcher

Precision is the share of proposed merges that are correct. Recall is the share of true duplicates the pipeline finds. Published benchmarks and vendor figures come from someone else’s data. Your names, your legacy systems’ quirks and your data entry habits will all differ from theirs.

Build a labelled set before go-live. Sample a few hundred pairs across every score band. A purely random sample will be almost all obvious non-matches and won’t tell you much. Have two people label each pair independently, then settle the disagreements. Expect a few: some pairs are hard for humans as well, and those belong in review permanently.

Score each stage separately against that set. Then pick operating points: very high precision for anything that auto-merges, and higher recall where missing a duplicate is the expensive mistake, such as screening for duplicate supplier payments. Re-run the set whenever you change a rule, a prompt or a model version. Treat it like a regression suite.

What it costs at volume

The LLM cost comes down to one line:

cost  grey_band_pairs  tokens_per_pair  price_per_token

Here’s a worked example with assumed numbers. Two million records produce 10 million candidate pairs after blocking. Rules settle all but 2 per cent, which leaves 200,000 pairs for the model. At around 800 tokens per pair, input and output combined, that’s 160 million tokens. Plug in your provider’s current rate card. At mid-sized model prices, the model bill is usually smaller than the human one.

Say 5 per cent of those pairs go to review. That’s 10,000 decisions, and at 30 seconds each it adds up to about 83 hours of staff time. Reviewer hours are the cost to plan and budget for.

To keep both numbers down:

• Tighten blocking, which shrinks every stage after it.
• Start with a small model and escalate only the unsure pairs to a larger one.
• Cache verdicts keyed on a hash of the two normalised records, so unchanged pairs never get judged twice.
• Run matching incrementally, comparing only new and changed records each night.
• Use batch endpoints for the initial backfill. Most providers price them below interactive calls.

Limitations

The same model can give different verdicts across runs and versions. Pin the model version, set temperature to zero, and lean on the labelled set to catch drift. If you send personal information to a model hosted overseas, APP 8 on cross-border disclosure applies. Check whether your provider offers the model in an Australian region, or run a smaller model in your own environment. Rules need maintenance as source systems change. Finally, no matcher fixes data that was wrong when it was entered. Entity resolution finds the duplicates, and stopping new ones is a data entry and integration problem.

PicNet builds production AI systems for Australian organisations. If you have duplicate records across your systems, talk to us about what a first matching project could look like.