remove invisible unicode characters from a text

npm install out-of-character

by Spencer Kelly and Adam Tsiopani

---

Unicode has a few-dozen characters that do not render anything, on purpose.

This is cool for cultural idiosyncracies in historical languages. More often though, their use is unintentional (or nefarious!), and these characters end-up causing problems parsing text formats.

• these are sometimes called 'zero-width', 'ignorable', or 'tag-characters' •

This library helps spot and remove these funboys, before they cause some trouble.

It also catches bidirectional control characters (the 'Trojan Source' trick), tag characters (invisible letters used for steganography), and stray variation selectors — while leaving legitimate emoji, keycap, CJK, and Mongolian sequences alone.

Please remember that some text is meant to have Khmer-vowels, or Kaithi-alphabet characters.

CLI

npm install -g out-of-character

detect invisible characters in all files in a directory

out-of-character ./path/to/dir

remove them from all files in a directory

out-of-character ./path/to/dir --replace

include files in nested sub-directories

out-of-character ./path/to/dir --replace --recursive

---

detect invisible characters in a file

out-of-character ./path/to/file.txt

remove invisible characters from a file

out-of-character ./path/to/file.txt --replace

(--remove is an alias for --replace, and glob patterns like ./src/**/*.js work too)

Javascript API

import {detect, replace} from 'out-of-character'

let str='noth­ing s͏neak឵y h᠎ere' //actually, there is.
console.log(detect(str))
/*  😮  😮  😮
[
  {
    name: 'SOFT HYPHEN',
    code: 'U+00AD',
    type: 'Invisible',
    offset: 4,
    replacement: ''
  },
  {
    name: 'COMBINING GRAPHEME JOINER',
    code: 'U+034F',
    type: 'Invisible',
    offset: 10,
    replacement: ''
  },
  {
    name: 'KHMER VOWEL INHERENT AA',
    code: 'U+17B5',
    type: 'Invisible',
    offset: 15,
    replacement: ''
  },
  {
    name: 'MONGOLIAN VOWEL SEPARATOR',
    code: 'U+180E',
    type: 'Invisible',
    offset: 19,
    replacement: ''
  }
]*/

// get rid of them!
let after = replace(str)
console.log(str !== after)
// true

detect returns null when the text is clean. Offsets are UTF-16 code-unit indices, like String.prototype.indexOf.

Both functions accept an options object, to leave some characters alone — by code, or by type:

// keep bidi controls, and the zero-width space
replace(str, { exclude: ['Bidi', 'U+200B'] })

// types are: 'Invisible', 'Whitespace', 'Bidi', 'Variation', 'Tag',
//   'Separator', 'Line Break', and 'Visible'

fixing/detecting in files can be done like:

const fs = require('fs')
const {detect, replace} = require('out-of-character')

let text = fs.readFileSync('./some-file.txt').toString()
console.log(detect(text))
// yikes.

// ok, fix it
fs.writeFileSync('./some-file.txt', replace(text))

// ok, double-check it.
let goodNow = fs.readFileSync('./some-file.txt').toString()
console.log(detect(goodNow))
// fhew.

Thank you to character.construction/blanks by Jan Lelis

and a tale of characters in Unicode by Stefan Judis

See also

• printable-characters - by Vit Gordon
• unzalgo - by kdex

MIT