First decide whether rows represent entities, events or versions. Repeated names can be legitimate, while different-looking rows can describe the same entity. Resolve duplicates with explicit matching and preservation rules rather than deleting rows solely because they look similar.
Understand what one row represents
A duplicate rule begins with the grain of the dataset. A customer table may intend one row per customer, while an order table legitimately contains many rows for the same customer. A history table may contain several versions of one customer record. The same repeated identifier can be a defect in one table and a necessary relationship in another.
Write the grain in ordinary language and ask the source owner to confirm it. Inspect a few normal examples and a few suspected duplicates before choosing a rule. This small exercise can prevent an automated cleaning step from removing genuine business activity.
Build a matching rule that reflects uncertainty
An exact, stable identifier is often more useful than a name alone, but its quality still needs checking. Identifiers can be reused, imported incorrectly or absent. Names, addresses and contact fields may change or be shared by different people and organisations.
Separate confident matches from cases needing review. Define which fields are used, what transformations are allowed and which disagreements should stop an automatic merge. Avoid assuming that a similarity score is proof of identity. A threshold needs validation against the population and the consequence of an incorrect decision.
Decide which information survives
A merge is not only a decision about which row to keep. Different records may contain valid information about contact preferences, historical transactions or previous names. Agree how conflicting values will be handled and what provenance should remain.
Preserve a link between original records and the resolved entity where appropriate. That allows later investigation and makes it possible to understand how a report changed. A reversible, reviewed decision is often safer than permanently removing evidence during an early investigation.
Inspect downstream effects
Deduplication can change counts, relationships and totals. Before releasing a revised dataset, check whether transactions still attach to the correct entity and whether the reporting denominator has changed. A customer count may decrease while transaction revenue should remain the same.
Compare representative before-and-after cases, including an incorrect match if one is found. Record the change in interpretation. Do not present a lower row count as a quality improvement unless the business meaning supports that conclusion.
Repair the source of repetition
Duplicates may arise from repeated imports, unclear creation rules or separate systems that lack a common identifier. Cleaning the report each week can hide a recurring collection problem. Use the investigation to identify where prevention or a controlled resolution process belongs.
Choose an owner for the matching rule and a route for disputed cases. New evidence can reveal that an earlier merge was wrong. A useful process anticipates that possibility instead of treating the cleaned dataset as permanently settled.
Illustrative worked example
A fictional scenario to explain the method, not a customer result or a market benchmark.
An illustrative customer export contains “North Coast Studio” and “Northcoast Studio Ltd”. They share a registration identifier, but their contact records refer to different departments. A separate row with the same company name belongs to an unrelated organisation in another region.
The team reviews the shared identifier, confirms the first two rows describe one organisation and preserves both departmental contacts under the resolved entity. It keeps the unrelated organisation separate. Original record identifiers remain in a mapping table so previous transactions can be traced.
The reported organisation count falls from three to two, while the total value of their legitimate transactions remains unchanged. These invented records illustrate entity resolution, not a universal matching rule. Actual identifiers, privacy requirements and the consequences of a mistaken merge must be assessed in their own context.
TAKE THIS INTO YOUR NEXT CONVERSATION
A practical next-step checklist.
- Write the intended grain of each dataset.
- Distinguish exact matches from review candidates.
- Preserve valid history and original-record links.
- Reconcile downstream counts and relationships after resolution.
Common questions.
Can repeated email addresses establish that two rows are duplicates?
Not on their own. Addresses can be shared, reused or entered incorrectly. Use an agreed combination of evidence appropriate to the entity and the risk of a mistaken match.
Should duplicate events be treated like duplicate customer records?
No. An event-level check must distinguish a repeated import from a genuine second event. The meaning of the row determines the correct rule.
Further reading & source context.
These primary references provide background. They are not evidence for the fictional example above and do not endorse Accuralytix.
Found an error or a missing qualification? Send an editorial correction.
Check my data readiness.
Use a practical tool to turn this idea into a clearer starting point.
Check my data readiness