Why Two Identical Lines Aren't Always Duplicates, and How I Fixed It

You merge two customer lists, remove the duplicates, and “Café Riche” is still there twice. You compare the two lines letter by letter and they look exactly the same. They aren’t the same to a computer: one was typed on a keyboard and the other was copied from a PDF, and the é is stored differently in each.

That’s the case that shaped ToolOnTap’s remove duplicate lines tool. Removing exact repeats is easy. Deciding which lines a person would call “the same” takes a few more rules, and this article goes through them.

The basic idea: one key per line

The tool reads the text once, from top to bottom. For each line it builds a key, the version of the line used for comparison, and looks it up in a Map. A key it hasn’t seen yet is stored with the line, its position and a count of 1. A key it has seen just adds 1 to the count.

The key is where all the rules live:

  const keyOf = (line: string) => {
    const k = (o.trim ? line.trim() : line).normalize('NFC')
    return o.ignoreCase ? k.toLocaleLowerCase() : k
  }

So two lines are duplicates when they’re equal after three optional changes: spaces at the ends removed, Unicode normalized, and case folded. The line you get back is always one of your own lines (trimmed, if that option is on), never the key. With “Ignore case” on, “Apple” and “apple” merge, and the result shows whichever spelling the tool kept, not a lowercased copy.

Because each line is one Map lookup, the work grows in step with the length of the list. In a test with 200,000 email addresses, 50,000 of them unique, removing the duplicates took about 70 ms in Node 20 on my laptop. With “Ignore case” and A–Z sorting on, it was under 100 ms.

Same letters, different bytes

Unicode often has more than one way to store the same visible text. The é in “Café” can be:

• one character, U+00E9 (Latin small letter e with acute), which is what most keyboards type; or
• two characters, a plain e followed by U+0301 (combining acute accent), which some apps produce and which can come along when you copy text from a PDF.

On screen they’re identical. As strings they’re different lengths, and === says they’re not equal. Arabic has the same issue. أ can be stored as one character, U+0623, or as alef followed by a hamza mark, U+0627 U+0654.

normalize('NFC') fixes this. NFC is one of the normalization forms defined by the Unicode standard: it rewrites every letter-plus-mark sequence that has a single-character equivalent into that single character. After it, both versions of “Café” and both versions of أحمد give the same key, and the tool removes the copy.

I chose NFC over NFKC on purpose. NFKC also folds “compatibility” characters: the ligature fi becomes f and i, full-width digits become ordinary digits, a superscript ² becomes 2. That would merge “file” and “file”, which is probably what you want, but it would also merge “m²” and “m2”, which is not. NFC only merges text that is meant to be the same, so it never removes a line you’d call different. The cost is that a line with the fi ligature stays separate from one with plain “fi”.

Invisible spaces

The other common reason for “duplicates” that survive is whitespace you can’t see. “Ignore spaces at the start and end” is on by default, and it uses JavaScript’s trim(). That removes more than spaces and tabs. It also removes the non-breaking space (U+00A0), which often sticks to the end of text copied from web pages and Word, and the byte order mark (U+FEFF), which some programs put at the start of an exported file. So “apple” followed by a non-breaking space merges with plain “apple”.

There’s one character trim() doesn’t remove: the zero-width space, U+200B. It isn’t classed as whitespace in Unicode, so a line with one inside it stays separate. I left that as it is. A zero-width space can also be part of the data, and silently stripping characters from inside a line goes further than this tool should.

With the option on, the lines you get back are trimmed too. Turn it off if leading spaces mean something in your list, such as indented code.

Line endings are handled before any of this. Windows (\r\n), old Mac (\r) and Unix (\n) line breaks are all recognized, so a list pasted from a Windows export doesn’t end up with an invisible \r on every line. The result always uses \n.

Keep first or keep last

By default each line keeps its first position and later copies are removed, so the list keeps its original order. “Keep last” does the opposite. Each line moves to the position of its last copy:

This matters for logs and exports sorted by date. If a list of order numbers runs from oldest to newest, “Keep last” leaves each one where it last appeared, which is its most recent position. With “Ignore case” on, it also keeps the last spelling, so “Apple” then “apple” gives “apple”.

Empty lines get their own rule. With “Remove empty lines” on (the default), they’re taken out and counted separately from duplicates. With it off, they stay exactly where they were and are never merged, because two blank lines in a document usually mean a paragraph break, not a repeated item.

Sorting so that 2 comes before 10

A plain A to Z sort compares text character by character, so “item 10” comes before “item 2”, because 1 comes before 2. Many spreadsheets sort text columns this way, and it’s rarely what you want. The tool sorts with the browser’s own collator in numeric mode:

const collator = new Intl.Collator(undefined, { numeric: true, sensitivity: 'base' })

numeric: true compares runs of digits as numbers, so “item 2” sorts before “item 10”. undefined means the browser’s language, so Arabic and accented names follow that language’s alphabetical order instead of raw character codes. sensitivity: 'base' makes the comparison ignore case and accents, so “Item 1” sits next to “item 2” by its number. Empty lines are left out of a sorted list. “Z–A” is the same sort reversed. Sorting happens after duplicates are removed, so it never affects which copy is kept.

The “Repeated most” list

Removing duplicates answers one question. Often the more useful question is which lines were repeated, and how often. Because every key already has a count, the tool gets this for free: the lines that appeared more than once, sorted by count, are shown under “Repeated most”. The page lists the top 8 and says how many others there are.

That turns the tool into a checker as well as a cleaner. Paste an order export and the order numbers that appear twice are at the top. Paste two keyword lists one after the other, and as long as neither list repeats items on its own, every line under “Repeated most” is in both lists.

You can paste your own list into remove duplicate lines. The other text tools work the same way, in the browser, one paste at a time.