How to Deduplicate IPv6 Addresses When One Host Has Many Spellings

A duplicate list of email addresses is easy to clean: Alice@Example.com and alice@example.com differ only by case, you fold the case, and the duplicate disappears. IPv6 is not that forgiving. The same host can appear a dozen ways in one file, and most of those ways are not just upper versus lower case. A plain dedup that compares strings character by character keeps every spelling as if it were a different machine.

This is the trap I want to walk through, because it bites everyone the first time they merge two routing exports.

One Address, Many Legal Spellings

IPv6 packs 128 bits into eight 16-bit groups. The format allows several shortcuts, and any combination of them produces the same host:

• Leading zeros in a group can be dropped: 0db8 becomes db8.
• One run of all-zero groups can collapse to ::.
• Hex letters can be upper or lower case: DB8 equals db8.

So these three strings all point to the exact same machine:

2001:db8::1
2001:0db8::1
2001:0db8:0000:0000:0000:0000:0000:0001

A human reads them as one address. A naive deduplicator reads them as three, because the byte sequences are different. Run a sort -u or a spreadsheet "Remove Duplicates" over a file mixing those forms and you keep all three rows. Your count is wrong, your firewall rule list is bloated, and the audit reviewer asking "why is this host listed three times?" has a fair question.

Meaningful IPv6 deduplication has to compare addresses by what they mean, not how they are typed. The standard answer is RFC 5952: expand or compress every address to one canonical form first, then dedupe. In canonical form, 2001:db8::1 is the single representation, and the two longer spellings above both reduce to it. Compare the canonical strings and the duplicates collapse.

What This Tool Actually Does (and Does Not Do)

I will be straight about the IPv6 Address Deduplicator rather than oversell it, because the line matters for your data.

The tool extracts IPv6-shaped tokens from whatever you paste, then builds a dedup key for each one. That key is the address with surrounding punctuation stripped and the hex letters lowercased. So it does fold case and trim stray brackets or trailing commas. These two rows merge into one:

2001:DB8::1
2001:db8::1

What the dedup key does not do is expand :: or strip leading zeros before comparing. It does not rebuild each address into RFC 5952 canonical form. That means the compression and zero-padding variants stay distinct. Feed it the three-spelling example from above and it keeps all three rows, because after lowercasing they are still three different strings:

2001:db8::1
2001:0db8::1
2001:0db8:0000:0000:0000:0000:0000:0001

I tested this directly while writing, and the count came back as three unique values, not one. So treat the deduplicator as a case-insensitive, whitespace-tolerant collapser — excellent for the "same spelling typed twice, one in caps" problem that logs and copy-paste create constantly, but not a full canonicalizer on its own. The validator inside it still flags genuinely broken rows (five hex digits in a group, two :: markers), which is a separate and useful job.

The Two-Step Fix

The honest workflow is to canonicalize first, then dedupe. The IPv6 Address Normalizer is the partner tool for the first step: it rewrites every address into one consistent form so the compression and zero-padding noise is gone before comparison.

Here is the worked example end to end. Start with this messy paste, the kind you get from concatenating two access logs:

Input

2001:0db8::1
2001:db8::1
2001:DB8:0:0:0:0:0:1
fe80::a00:27ff:fe4e:66a1
FE80::A00:27FF:FE4E:66A1

If you drop that straight into the deduplicator, the case-only pairs merge but the zero-padded 2001:DB8:0:0:0:0:0:1 and the compressed forms stay separate:

Deduplicator only — output

2001:0db8::1
2001:db8::1
2001:db8:0:0:0:0:0:1
fe80::a00:27ff:fe4e:66a1

Four rows, and the first three are still the same host. Now run the normalizer first to collapse every 2001:db8 spelling to canonical form, then deduplicate:

Normalize, then dedupe — output

2001:db8::1
fe80::a00:27ff:fe4e:66a1

Two rows, which is the true unique count. That is the result you want feeding a firewall rule set, an allowlist, or a SQL IN clause.

Keep the Evidence, Not Just the Answer

When you do deduplicate, do not throw away where the duplicates came from. The tool keeps a count per value and the first source line for each, and it exports CSV, JSON, Markdown, SQL IN, TypeScript union, or plain lines. I always export CSV with the line numbers when the list is going to a review, because "this host appeared 4 times, first on line 212" is a sentence a reviewer can act on, while a bare deduplicated list is not. The "include invalid rows" option is worth leaving on for the same reason: a row with five hex digits in a group is data you want to see and fix at the source, not silently drop.

Everything runs in the browser. Nothing you paste — and no file you load with the local File API — leaves the tab, which is the right default when the addresses are internal infrastructure.

A Quick Checklist

Before you trust a deduplicated IPv6 list:

1. Normalize first if your data mixes compression or zero-padding styles. Case folding alone is not enough.
2. Extract cleanly from logs and copied pages. If your source is HTML or a dump full of other text, pull the addresses with the IPv6 Address Extractor before deduplicating.
3. Keep counts and line numbers so the dedup is auditable, not a black box.
4. Do not treat valid format as a live host. A well-formed address is not proof the machine answers; the validator checks syntax, not reachability.

The same discipline applies to IPv4 work, where leading zeros and odd notations cause their own confusion — the IPv4 Address Extractor handles the pull step there. IPv6 just has more legal ways to write one thing, so the "many spellings of one address" problem is sharper. Canonicalize, then dedupe, and the list finally tells the truth.

---

Made by Toolora · Updated 2026-06-13