feat: expose registry-free tokenize function

[danielroe]

Changes

we're adopting speed-highlight in nuxt/cli 🎉

we're trying to get the size of nuxt/cli as small as possible, and noticed that bundling highlightText from /terminal to highlight just JSON and HTML costs 22.7 kB, because the entry reaches the language registry and nothing tree-shakes.

this PR adds @speed-highlight/core/tokenize, which takes the definitions the caller already imported:

import { tokenize } from '@speed-highlight/core/tokenize'
import html from '@speed-highlight/core/languages/html.js'
import css from '@speed-highlight/core/languages/css.js'
import js from '@speed-highlight/core/languages/js.js'

tokenize(code, { sub: html }, (text, type) => { /* ... */ }, { languages: { css, js } })

it's synchronous, and a sub naming a language that isn't in languages is emitted as plain text rather than throwing. (nb. I added a try/catch as esbuild rewrites the dynamic import into a shim that fails synchronously...)

bundle	size
highlightText from /terminal	22.7 kB
tokenize + JSON + HTML (8 grammars)	5.6 kB
tokenize + json	1.3 kB

rather than fork the loop, I pulled it into a generator that yields the name of each language it needs and expects the definition back. the sync entry feeds it from the languages map, and src/index.js feeds it from the existing langs cache and dynamic import, so the async API behaves as before.

Note

this also adds ./common.js to the exports map, so callers importing from the language subpaths can resolve expand rules. let me know if you want me to split this into a separate PR or drop it entirely

a sync highlightText (#29) would be easy on top of this but I left it out to keep the change small. let me know if you have any better ideas, and feel free to push changes if 🙏

[matubu]

Awesome! Let me take a look...