[{"data":1,"prerenderedAt":38},["ShallowReactive",2],{"$fklUy1jz3LJX1QE1obW4JQyj8FySV9OokjN2Jp4eP5x0":3},{"slug":4,"title":5,"description":6,"date":7,"tags":8,"tools":13,"draft":15,"html":16,"toc":17,"minutes":37},"removing-duplicate-lines","Why Two Identical Lines Aren't Always Duplicates, and How I Fixed It","Unicode normalization, invisible spaces, keep first or last, and natural sorting, explained through the code behind ToolOnTap's remove duplicate lines tool.","2026-09-21",[9,10,11,12],"unicode","text","javascript","algorithms",[14],"remove-duplicate-lines",false,"\u003Cp>You merge two customer lists, remove the duplicates, and “Café Riche” is still there twice. You compare the two lines letter by letter and they look exactly the same. They aren’t the same to a computer: one was typed on a keyboard and the other was copied from a PDF, and the é is stored differently in each.\u003C\u002Fp>\n\u003Cp>That’s the case that shaped ToolOnTap’s \u003Ca href=\"\u002Fremove-duplicate-lines\">remove duplicate lines\u003C\u002Fa> tool. Removing exact repeats is easy. Deciding which lines a person would call “the same” takes a few more rules, and this article goes through them.\u003C\u002Fp>\n\u003Ch2 id=\"the-basic-idea-one-key-per-line\">The basic idea: one key per line\u003C\u002Fh2>\n\u003Cp>The tool reads the text once, from top to bottom. For each line it builds a \u003Cstrong>key\u003C\u002Fstrong>, the version of the line used for comparison, and looks it up in a \u003Ccode>Map\u003C\u002Fcode>. A key it hasn’t seen yet is stored with the line, its position and a count of 1. A key it has seen just adds 1 to the count.\u003C\u002Fp>\n\u003Cp>The key is where all the rules live:\u003C\u002Fp>\n\u003Cpre class=\"shiki shiki-themes github-light github-dark\" style=\"--shiki-light:#24292e;--shiki-dark:#e1e4e8;--shiki-light-bg:#fff;--shiki-dark-bg:#24292e\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">  const\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#6F42C1;--shiki-dark:#B392F0\"> keyOf\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\"> =\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> (\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#b35309;--shiki-dark:#FFAB70\">line\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">:\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#005CC5;--shiki-dark:#79B8FF\"> string\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">) \u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">=>\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">    const\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#005CC5;--shiki-dark:#79B8FF\"> k\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\"> =\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> (o.trim \u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">?\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> line.\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#6F42C1;--shiki-dark:#B392F0\">trim\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">() \u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">:\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> line).\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#6F42C1;--shiki-dark:#B392F0\">normalize\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">(\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#032F62;--shiki-dark:#9ECBFF\">'NFC'\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">)\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">    return\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> o.ignoreCase \u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">?\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> k.\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#6F42C1;--shiki-dark:#B392F0\">toLocaleLowerCase\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">() \u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">:\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> k\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">  }\u003C\u002Fspan>\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>So two lines are duplicates when they’re equal after three optional changes: spaces at the ends removed, Unicode normalized, and case folded. The line you get back is always one of your own lines (trimmed, if that option is on), never the key. With “Ignore case” on, “Apple” and “apple” merge, and the result shows whichever spelling the tool kept, not a lowercased copy.\u003C\u002Fp>\n\u003Cp>Because each line is one \u003Ccode>Map\u003C\u002Fcode> lookup, the work grows in step with the length of the list. In a test with 200,000 email addresses, 50,000 of them unique, removing the duplicates took about 70 ms in Node 20 on my laptop. With “Ignore case” and A–Z sorting on, it was under 100 ms.\u003C\u002Fp>\n\u003Ch2 id=\"same-letters-different-bytes\">Same letters, different bytes\u003C\u002Fh2>\n\u003Cp>Unicode often has more than one way to store the same visible text. The é in “Café” can be:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>one character, U+00E9 (Latin small letter e with acute), which is what most keyboards type; or\u003C\u002Fli>\n\u003Cli>two characters, a plain e followed by U+0301 (combining acute accent), which some apps produce and which can come along when you copy text from a PDF.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>On screen they’re identical. As strings they’re different lengths, and \u003Ccode>===\u003C\u002Fcode> says they’re not equal. Arabic has the same issue. أ can be stored as one character, U+0623, or as alef followed by a hamza mark, U+0627 U+0654.\u003C\u002Fp>\n\u003Cp>\u003Ccode>normalize('NFC')\u003C\u002Fcode> fixes this. NFC is one of the normalization forms defined by the Unicode standard: it rewrites every letter-plus-mark sequence that has a single-character equivalent into that single character. After it, both versions of “Café” and both versions of أحمد give the same key, and the tool removes the copy.\u003C\u002Fp>\n\u003Cp>I chose NFC over NFKC on purpose. NFKC also folds “compatibility” characters: the ligature fi becomes f and i, full-width digits become ordinary digits, a superscript ² becomes 2. That would merge “file” and “file”, which is probably what you want, but it would also merge “m²” and “m2”, which is not. NFC only merges text that is meant to be the same, so it never removes a line you’d call different. The cost is that a line with the fi ligature stays separate from one with plain “fi”.\u003C\u002Fp>\n\u003Ch2 id=\"invisible-spaces\">Invisible spaces\u003C\u002Fh2>\n\u003Cp>The other common reason for “duplicates” that survive is whitespace you can’t see. “Ignore spaces at the start and end” is on by default, and it uses JavaScript’s \u003Ccode>trim()\u003C\u002Fcode>. That removes more than spaces and tabs. It also removes the non-breaking space (U+00A0), which often sticks to the end of text copied from web pages and Word, and the byte order mark (U+FEFF), which some programs put at the start of an exported file. So “apple” followed by a non-breaking space merges with plain “apple”.\u003C\u002Fp>\n\u003Cp>There’s one character \u003Ccode>trim()\u003C\u002Fcode> doesn’t remove: the zero-width space, U+200B. It isn’t classed as whitespace in Unicode, so a line with one inside it stays separate. I left that as it is. A zero-width space can also be part of the data, and silently stripping characters from inside a line goes further than this tool should.\u003C\u002Fp>\n\u003Cp>With the option on, the lines you get back are trimmed too. Turn it off if leading spaces mean something in your list, such as indented code.\u003C\u002Fp>\n\u003Cp>Line endings are handled before any of this. Windows (\u003Ccode>\\r\\n\u003C\u002Fcode>), old Mac (\u003Ccode>\\r\u003C\u002Fcode>) and Unix (\u003Ccode>\\n\u003C\u002Fcode>) line breaks are all recognized, so a list pasted from a Windows export doesn’t end up with an invisible \u003Ccode>\\r\u003C\u002Fcode> on every line. The result always uses \u003Ccode>\\n\u003C\u002Fcode>.\u003C\u002Fp>\n\u003Ch2 id=\"keep-first-or-keep-last\">Keep first or keep last\u003C\u002Fh2>\n\u003Cp>By default each line keeps its first position and later copies are removed, so the list keeps its original order. “Keep last” does the opposite. Each line moves to the position of its last copy:\u003C\u002Fp>\n\u003Cdiv class=\"table-wrap\">\u003Ctable>\u003Cthead>\n\u003Ctr>\n\u003Cth>Input\u003C\u002Fth>\n\u003Cth>Keep first copy\u003C\u002Fth>\n\u003Cth>Keep last\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>b, a, b, c, a\u003C\u002Ftd>\n\u003Ctd>b, a, c\u003C\u002Ftd>\n\u003Ctd>b, c, a\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\u003C\u002Fdiv>\u003Cp>This matters for logs and exports sorted by date. If a list of order numbers runs from oldest to newest, “Keep last” leaves each one where it last appeared, which is its most recent position. With “Ignore case” on, it also keeps the last spelling, so “Apple” then “apple” gives “apple”.\u003C\u002Fp>\n\u003Cp>Empty lines get their own rule. With “Remove empty lines” on (the default), they’re taken out and counted separately from duplicates. With it off, they stay exactly where they were and are never merged, because two blank lines in a document usually mean a paragraph break, not a repeated item.\u003C\u002Fp>\n\u003Ch2 id=\"sorting-so-that-2-comes-before-10\">Sorting so that 2 comes before 10\u003C\u002Fh2>\n\u003Cp>A plain A to Z sort compares text character by character, so “item 10” comes before “item 2”, because 1 comes before 2. Many spreadsheets sort text columns this way, and it’s rarely what you want. The tool sorts with the browser’s own collator in numeric mode:\u003C\u002Fp>\n\u003Cpre class=\"shiki shiki-themes github-light github-dark\" style=\"--shiki-light:#24292e;--shiki-dark:#e1e4e8;--shiki-light-bg:#fff;--shiki-dark-bg:#24292e\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\">const\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#005CC5;--shiki-dark:#79B8FF\"> collator\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\"> =\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#cb2431;--shiki-dark:#F97583\"> new\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> Intl.\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#6F42C1;--shiki-dark:#B392F0\">Collator\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">(\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#005CC5;--shiki-dark:#79B8FF\">undefined\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">, { numeric: \u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#005CC5;--shiki-dark:#79B8FF\">true\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\">, sensitivity: \u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#032F62;--shiki-dark:#9ECBFF\">'base'\u003C\u002Fspan>\u003Cspan style=\"--shiki-light:#24292E;--shiki-dark:#E1E4E8\"> })\u003C\u002Fspan>\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Ccode>numeric: true\u003C\u002Fcode> compares runs of digits as numbers, so “item 2” sorts before “item 10”. \u003Ccode>undefined\u003C\u002Fcode> means the browser’s language, so Arabic and accented names follow that language’s alphabetical order instead of raw character codes. \u003Ccode>sensitivity: 'base'\u003C\u002Fcode> makes the comparison ignore case and accents, so “Item 1” sits next to “item 2” by its number. Empty lines are left out of a sorted list. “Z–A” is the same sort reversed. Sorting happens after duplicates are removed, so it never affects which copy is kept.\u003C\u002Fp>\n\u003Ch2 id=\"the-repeated-most-list\">The “Repeated most” list\u003C\u002Fh2>\n\u003Cp>Removing duplicates answers one question. Often the more useful question is which lines were repeated, and how often. Because every key already has a count, the tool gets this for free: the lines that appeared more than once, sorted by count, are shown under “Repeated most”. The page lists the top 8 and says how many others there are.\u003C\u002Fp>\n\u003Cp>That turns the tool into a checker as well as a cleaner. Paste an order export and the order numbers that appear twice are at the top. Paste two keyword lists one after the other, and as long as neither list repeats items on its own, every line under “Repeated most” is in both lists.\u003C\u002Fp>\n\u003Cp>You can paste your own list into \u003Ca href=\"\u002Fremove-duplicate-lines\">remove duplicate lines\u003C\u002Fa>. The other \u003Ca href=\"\u002Ftext-tools\">text tools\u003C\u002Fa> work the same way, in the browser, one paste at a time.\u003C\u002Fp>\n",[18,22,25,28,31,34],{"id":19,"text":20,"level":21},"the-basic-idea-one-key-per-line","The basic idea: one key per line",2,{"id":23,"text":24,"level":21},"same-letters-different-bytes","Same letters, different bytes",{"id":26,"text":27,"level":21},"invisible-spaces","Invisible spaces",{"id":29,"text":30,"level":21},"keep-first-or-keep-last","Keep first or keep last",{"id":32,"text":33,"level":21},"sorting-so-that-2-comes-before-10","Sorting so that 2 comes before 10",{"id":35,"text":36,"level":21},"the-repeated-most-list","The “Repeated most” list",5,1792325805417]