700k-Scale Identity Resolution

A social account is not a creator.

A platform gives us an account. The business needs a person or organization. Those are different objects. An Instagram profile can be renamed. A YouTube creator can run two channels. A TikTok handle can differ from the legal name, the display name, and the name used on a campaign brief. One person can therefore show up many times in the source data, with no single field that safely joins every record.

Picture one person, Maya, with Instagram @maya.creates, TikTok @mayacreates, and a YouTube channel under a brand name. Three rows. One performer. If reporting sums each row as a separate creator, Maya’s program looks smaller than it is. If a bad rule merges Maya with someone who shares a first name, two careers get mixed. The job is to build a durable identity reporting can trust after handles, names, and source files change.

This is an entity-resolution problem: decide which records describe the same real thing. Here the real thing is a creator, and the records are social channels. The job is not “find rows that look alike.” It is “build one identity the warehouse and dashboards can reuse.”

Start by naming the grain.

Grain means what one row represents. The raw table has one row per social account (channel). The creator table has one row per resolved person. A bridge table connects the two. Keep those grains separate. If channel and creator fields are mixed into one wide table, a creator with three accounts can triple a campaign total. A clear bridge lets the warehouse count accounts when the question is about accounts, and humans when the question is about humans.

No single field deserves complete trust.

Exact matching is tempting because it is easy to explain: if two handles are equal, merge them. Real social data makes that rule unsafe. Handles can be reused, copied, changed, or entered with different punctuation. Display names are worse. Two humans can share a name, while one creator can use a brand name on one platform and a personal name on another — like Maya’s brand-named YouTube channel.

The safer method is to combine several clues. The implementation uses a strong third-party creator key when it is usable, exact normalized handles across platforms, platform context, and name similarity. Each clue has a weight. Evidence can add together, but weak evidence cannot win merely because it exists.

Normalization removes noise, not meaning.

Before comparison, handles are put into a common form by removing case and punctuation. That lets @creator.name and Creator_Name meet on neutral ground. Normalization should stop there. It should not guess that two different words are equivalent. The more meaning a cleanup rule invents, the more quietly it can merge the wrong humans.

Common and empty values are blocked before they can create links. A value such as “unknown” is missing data, not a shared identity. Very short or unusually common handles are also poor evidence. A basic rule: a matching value becomes less useful as more unrelated records share it.

A graph can use evidence that a normal join cannot.

A database join asks whether two fields meet a fixed condition. A graph asks how a set of records is connected. Each account becomes a node — a point in the graph. A match clue becomes an edge — a line between two points. The edge stores its weight and the reason it exists. The library we used for this is NetworkX, a standard Python toolkit for graph work.

This matters because identity can be transitive. Maya’s Instagram may share a provider key with her TikTok. That TikTok may share a handle pattern with her YouTube. Instagram and YouTube may have no direct match, yet the full evidence can still place all three in one group. The software finds these connected groups. Each group becomes a candidate creator.

Weights make the decision open to inspection.

The point of a weight is not decoration. It records how much trust the system gives a clue. In the current setup, a clean creator key carries more weight than a matching handle. A similar name can add support. A weak edge is removed before groups are formed. The exact numbers can be tuned, but the rule should stay visible: which evidence joined these two records, and why was that evidence strong enough?

Every edge is written to an audit log with its source, target, signal, weight, and name score. That log is essential. Without it, a wrong merge becomes a mystery. With it, a reviewer can see the path that joined two accounts and decide whether the rule or the source data needs correction.

One weak link should not merge two strong groups.

Graph matching has a dangerous failure mode. Imagine three accounts that clearly belong to Maya and three that clearly belong to someone else. If one weak edge connects the groups — maybe a shared first name scored too generously — a basic “connected group” algorithm will call all six one person. The graph is technically connected. The identity is wrong.

In graph terms, a bridge is an edge that, if removed, splits a group into separate parts. The pipeline checks bridges inside larger groups. If a bridge is also weak, it is cut before final creator IDs are created. I call this workflow BridgeCut Creator Graph Resolution. The name is specific to this project; the ideas come from standard record linkage and graph analysis.

False merges usually cost more than missed matches.

A missed match splits one creator into two rows. That is inconvenient, but the source records stay separate and can be joined later. A false merge combines two humans. Their performance, campaign history, and ownership can become mixed. Undoing that error is harder because downstream systems may already trust the shared ID.

That cost difference should shape the threshold. The system is conservative when evidence is weak. It keeps a real single-platform creator as a valid result instead of forcing every record into a multi-platform group. Automation is useful here because it can clear strong cases quickly — not because it can erase uncertainty.

A correct match can still break reporting if its ID changes.

Identity resolution runs more than once. New accounts arrive, handles change, and better evidence appears. If every run creates brand-new creator IDs, historical dashboards lose continuity. The same person may appear under one ID last month and another this month even when the match itself improved.

The pipeline solves this with a separate historical remap. After the new graph is resolved, each group is compared with the prior master list. A clear match keeps its old master creator ID. Conflicts are not guessed away. Saved review decisions say whether to keep one old ID, split a group by its earlier ownership, or hold the case for review.

This separation keeps two questions apart. The graph asks, “Which accounts belong together now?” The remap asks, “Which durable business identity should this group keep?” Combining those questions in one fuzzy rule would make both harder to test.

Human review belongs at the uncertain edge.

Manual linking failed because humans were asked to inspect everything. Fully automatic matching would fail the other way by hiding hard choices. The useful middle is selective review. The system handles routine evidence and sends only risky groups to a person.

Groups are flagged when they are unusually large, have low average edge strength, contain names that strongly disagree, or still include a weak link. The review file shows the accounts, platforms, names, links, confidence, and reason for the flag. A reviewer can keep the group or split it. That decision is saved so the next run does not ask the same question again.

The system changes how much work humans must do.

A manual process grows with every account. If the network doubles, the queue roughly doubles, and hard cases can take even longer because reviewers must search more history. That model had already stopped keeping pace. A base of about 700,000 creators plus tens of thousands of monthly additions made complete hand-linking impossible.

The graph pipeline does not compare every account with every other account. That would approach O(n²): double the records and the possible pair checks grow about four times. Instead, it first groups records by useful keys — a normalized handle or creator key — then builds candidate edges only inside those smaller buckets. The exact run time depends on bucket sizes, but this blocking step removes most impossible comparisons before graph work begins.

Very common values are blocked, and groups above a safe size are skipped during pair generation. These limits protect both speed and quality. A key shared by hundreds of records is expensive to compare and unlikely to describe one person. The same rule solves a computer problem and a data problem at once.

Design the output for reporting, not just matching.

The main output is a lookup from each source account key to one master creator ID. It also carries the original platform, handle, URL, match status, confidence, group size, evidence, and review state. Reporting can use the two IDs it needs: the account key for platform facts and the creator ID for cross-platform totals. Analysts do not need to rerun the matching logic inside every dashboard.