Skip to main content

How to Deduplicate DOIs Without Letting Prefix and Case Variants Slip Through

A plain text dedup keeps duplicate DOIs because the resolver prefix and letter case differ. Here is how to collapse those variants into one clean citation.

Published By Li Lei
#doi #citations #deduplication #bibliography #research-tools

How to Deduplicate DOIs Without Letting Prefix and Case Variants Slip Through

A reference list looks tidy until you actually count it. You merge two export files, run them through whatever dedup your editor offers, and the line count drops by a satisfying amount. Then a reviewer points out that the same paper is cited twice, three pages apart. The DOIs were identical the whole time. Your dedup just could not see it.

This happens because a DOI travels in several disguises. The same identifier shows up as a full resolver link, as a doi: shorthand, and as a bare string, and the DOI standard treats letters as case-insensitive on top of that. A character-by-character dedup compares the disguises, not the identity underneath, so the copies survive.

Why a plain dedup keeps duplicate DOIs

Take one real paper: the 2020 Nature article on array programming. Across a single bibliography you might find all three of these:

  • https://doi.org/10.1038/s41586-020-2649-2
  • doi:10.1038/s41586-020-2649-2
  • 10.1038/s41586-020-2649-2

These are one citation. The DOI is 10.1038/s41586-020-2649-2. Everything before the 10. is a resolver wrapper that tells a browser where to send the request, not part of the identifier. But to a string comparison, the first entry is 41 characters and the third is 25, so they are obviously "different" and both stay.

Case makes it worse. The DOI handbook states that DOIs are case-insensitive, so 10.1145/3375637.3375842 and 10.1145/3375637.3375842 written with a stray capital in the suffix point to the same record. Many publishers print suffixes in mixed case, and citation managers do not always normalize them. A literal dedup that respects case will keep both.

So the rule is simple: a meaningful DOI dedup has to strip the resolver prefix and case-fold the value before it compares anything. A dedup that skips those two steps is comparing wrappers, and wrappers are exactly the part that varies.

What this tool actually does

I checked the behavior of the DOI Deduplicator against its parser rather than trusting the marketing line, because "deduplicate" is a word people use loosely. Here is what it really does, and where the limits are.

It strips the resolver prefix at extraction. The parser scans your pasted text for the bare DOI pattern, anchored at 10. followed by a registrant code and a slash-separated suffix. When it meets https://doi.org/10.1038/s41586-020-2649-2, it pulls out 10.1038/s41586-020-2649-2 and leaves the https://doi.org/ behind. The same is true for the doi: form. The match starts at the 10., so the wrapper never enters the comparison. This is the prefix-stripping step, done implicitly by where the extraction begins.

It case-folds before comparing. Every extracted DOI is normalized to lowercase, and the deduplication key is built from that normalized value. Two suffixes that differ only by case collapse to one row. That matches the DOI standard's case-insensitivity, so you get the behavior a librarian would expect.

It keeps the evidence. The output is one canonical row per DOI, with a duplicate count and the first source line, so you can explain to a co-author where the repeats came from instead of just deleting them silently.

A couple of honest limits. The tool normalizes case but does not rewrite a DOI into a canonical display form beyond lowercasing, so if you want a consistent presentation style across a manuscript you may still want a pass through a dedicated normalizer. And it validates format only: a well-formed DOI that resolves to nothing still counts as valid here, because nothing in the browser can confirm a record exists without a network call, and this tool stays fully local. Treat validation as "this is shaped like a DOI," not "this paper exists."

A worked example

Paste this messy block, the kind you get after merging a Zotero export with a few lines copied off a journal page:

References:
https://doi.org/10.1038/s41586-020-2649-2
doi:10.1038/S41586-020-2649-2
See also 10.1038/s41586-020-2649-2 (duplicate)
10.1145/3375637.3375842

A naive dedup leaves four lines, because the prefixes and the capital S make each row look unique. Run it through the DOI Deduplicator and you get two rows:

10.1038/s41586-020-2649-2   (count 3, first line 2)
10.1145/3375637.3375842     (count 1, first line 5)

The three Nature variants merge into a single canonical entry with a count of three, and the ACM paper stands on its own. Export that as CSV, JSON, Markdown, or a plain line list and hand it straight to a co-author or a script. The count column is the part I lean on most: it tells me at a glance that one citation was sitting in my list three times under three masks.

How I use it in practice

When I assemble a bibliography from more than one source, I do not trust my eyes anymore. I once spent twenty minutes manually scanning a 180-entry reference list for duplicates before a submission, found three, and felt confident. A later automated pass found two more that I had read straight past, both of them resolver-link versions of papers I had already added as bare DOIs. My eyes were matching the visible text; the duplicates were hiding in the prefix. Now my first move with any merged list is to paste the whole thing in, let the parser strip and fold, and read the count column. It takes about ten seconds and it has caught a duplicate in nearly every multi-source list I have thrown at it.

Fitting it into a cleanup pipeline

DOI cleanup rarely lives alone. A few neighboring steps that pair well:

  • If your source text is full of full URLs and you want every link first, pull them with the URL Extractor before isolating the DOIs.
  • If you are scraping references out of Markdown notes, the Markdown Link Extractor gets you the raw links to feed in.
  • When the final list has to go into a document table, the CSV to Markdown Table turns the exported CSV into something you can drop into a README or a wiki.

The order that works for me is extract, then dedupe, then format. Each step is local, nothing is uploaded, and the duplicate counts survive the whole way through so the audit trail stays intact.

The takeaway

A duplicate DOI is almost never a typo. It is the same identifier wearing a different coat: a resolver prefix, a doi: label, or a capital letter in the suffix. Any dedup that compares the coat instead of the identity will let those copies through. Strip the prefix, fold the case, and compare what is left. That is the whole trick, and it is exactly what a DOI-aware dedup is for.

Made by Toolora · Updated 2026-06-13