Counting Arabic Words and Characters Correctly in JavaScript

Type مَرْحَبًا with full harakat into a lot of character counters and you get 9. Type مرحبا without them and you get 5. It’s the same word, read the same way, five letters long. When I built ToolOnTap’s word counter, I wanted Arabic to come out right, because the people I write for type Arabic every day, and most counters treat it as an afterthought.

This article goes through what “a character” and “a word” mean in code, and the choices I made for Arabic, emoji, links, SMS and X.

What .length actually counts

In JavaScript, "مَرْحَبًا".length is 9. .length counts UTF-16 code units, and harakat are separate Unicode characters: fatha, sukun, fatha again, and fathatan each take a code unit of their own on top of the five letters.

Emoji have the opposite problem. A family emoji such as 👨‍👩‍👧 is three people joined by invisible zero-width joiners, and its .length is 8. The Egyptian flag 🇪🇬 is two “regional indicator” letters, E and G, and its .length is 4.

What a reader calls a character is a grapheme cluster: a base character plus everything attached to it. The browser can split text that way with Intl.Segmenter, and the counter uses it:

const graphemes = (() => {
  const seg = typeof Intl !== 'undefined' && 'Segmenter' in Intl ? new Intl.Segmenter(undefined, { granularity: 'grapheme' }) : null
  return (text: string): string[] => (seg ? Array.from(seg.segment(text), s => s.segment) : Array.from(text))
})()

If Intl.Segmenter isn’t available, it falls back to Array.from, which splits by code point. That’s still better than .length for emoji outside the basic range, though a family emoji would count as more than one.

Here’s what that gives for a few test strings:

Harakat and tatweel

Harakat are combining marks in Unicode, so grapheme segmentation already attaches them to the letter they sit on. No special case is needed: مَ is one grapheme.

The tatweel (ـ, U+0640) is different. It’s the stretch that typographers use to widen a word, as in مـــرحبا. Unicode treats it as a letter in its own right, so a grapheme splitter counts each one. But nobody writing an essay thinks of it as a character, so the counter removes it before counting:

export function characters(text: string): number {
  return graphemes(text.replace(TATWEEL, '')).length
}

TATWEEL is just /ـ/g. It’s removed before counting words too, so مـــرحبا and مرحبا are the same word.

What counts as one word

A word is a run of letters, marks and digits. The interesting part is what joins two runs into one:

• an apostrophe or a hyphen between letters, so “don’t” and “e-mail” are one word each;
• a dot or comma inside a number, so “3.14” and “1,000” are one word each.

That’s how Microsoft Word counts too, which matters when a teacher checks your essay in Word.

Links and email addresses needed their own rule. A URL like https://toolontap.com/word-counter is full of dots and slashes, and a simple word pattern would count “https”, “toolontap”, “com”, “word” and “counter” as five words. So before counting, the counter replaces every link and email address with the placeholder 0. That counts as exactly one word, and because it’s a number, it’s never shown in the “Most used words” list. The sentence Visit https://toolontap.com/word-counter now counts as 3 words.

Chinese and Japanese don’t put spaces between words, so each Han, Hiragana or Katakana character counts as a word, as it does in Word and Google Docs.

Reading time for Arabic

Reading studies measure a lower speed in words per minute for Arabic than for English, so a single average would make Arabic reading times look too short. The counter uses two speeds from published studies:

• 238 words a minute for English non-fiction (Brysbaert, 2019, “How many words do we read per minute?”).
• 138 words a minute for Arabic (Trauzettel-Klosinski and others, 2012, from the IReST reading test).

The counter switches to the Arabic pace when more than half of the words contain Arabic letters, and it says so under the numbers. For 276 words, that’s the difference between “1 min” and “2 min”. Speaking time uses 140 words a minute for both, a typical presentation pace.

SMS: one Arabic letter cuts the limit from 160 to 70

SMS is where Arabic users pay for bad counting, literally. A text message is encoded in one of two ways:

The GSM alphabet has Latin letters, digits, common punctuation and some accented and Greek letters. Any Arabic letter, any emoji, and even the curly apostrophe (’) that phones type automatically switch the whole message to UCS-2. “Hello it’s me” with a straight apostrophe is GSM-7; “Hello it’s me” with a curly one is UCS-2. Parts of a long message hold a little less because each part carries a header that tells the phone how to join them.

Two more details are in the code. A few GSM characters, such as €, [, ], { and }, take two units because they’re sent with an escape character. And in UCS-2, an emoji outside the basic range takes two units, so it uses up two of the 70.

Here’s the decision, from smsInfo:

  const chars = Array.from(text)
  const gsm = chars.every(c => GSM.includes(c) || GSM_EXT.includes(c))
  if (gsm) {
    const units = chars.reduce((n, c) => n + (GSM_EXT.includes(c) ? 2 : 1), 0)
    const perPart = units > 160 ? 153 : 160
    return { encoding: 'GSM-7', units, parts: units ? Math.ceil(units / perPart) : 0, perPart }
  }

The Limits section shows the encoding and the number of parts, so you can see when a message is about to become two.

X counts harakat, even if the counter doesn’t

The X limit follows the counting rules of twitter-text, X’s open-source library (version 3 configuration). Most Latin, Arabic and other scripts count 1 per code point, Chinese and Japanese characters count 2, an emoji counts 2, and every link counts 23 however long it is.

“Per code point” has a consequence for Arabic. On X, each haraka and each tatweel is a code point of its own and adds to the count. So مَرْحَبًا is 5 characters in the counter but 9 towards the 280 on X. I kept the two numbers separate on purpose. The characters figure tells you how long your text is, and the X figure tells you whether X will accept it. If a post with full harakat is just over the limit, removing the decorative tatweel or some harakat gets it under.

Most used words, Arabic spelling included

The “Most used words” list helps you spot a word you’ve repeated too often. For Arabic, it folds the spelling differences people make all the time, so the same word isn’t split across three rows:

function wordKey(word: string): string {
  return word.toLocaleLowerCase().replace(ARABIC_MARKS, '').replace(/[أإآ]/g, 'ا').replace(/ى/g, 'ي').replace(/ة/g, 'ه')
}

That makes أحمد and احمد one word, and the same for words ending in ى or ي, or ة or ه. Harakat are removed for this comparison only; the counts above don’t change. Common short words are skipped, in English and Arabic, including Egyptian ones like ده, دي and اللي that a standard Arabic list wouldn’t have.

You can paste your own text into the word counter and watch the numbers change as you type. It sits with the other text tools.