locator-extraction

A Node.js CLI that loads pages in Playwright (Chromium), walks interactive elements across frames and open shadow roots, proposes ranked Playwright locator strings (getByTestId, getByRole, CSS, text, etc.), validates them with locator.count(), and writes a versioned JSON report. Optional ignore rules, human review (interactive or decisions file), and snippet export help you turn noisy UIs into stable automation hints.

Requirements

• Node.js 18 or newer
• npm (or another compatible package manager)

Installation

git clone <repository-url>
cd <repository-folder>
npm install
npm run build
npx playwright install chromium

The CLI entrypoint is dist/cli.js. After build you can run:

node dist/cli.js --help
node dist/cli.js extract --help

To use the package binary locally (optional):

npm link   # from project root; then `locator-extract` is on PATH

Quick start

Extract locators from one URL and write report.json:

npm run build
node dist/cli.js extract --url https://example.com -o report.json

Use a targets file (supports ignore rules in the same file):

node dist/cli.js extract --targets examples/targets.sample.json -o report.json

Targets configuration

Targets are defined in a single file in JSON, YAML, or TOML (detected by extension, or use --targets-format).

Each target must include:

Field	Type	Description
name	string	Stable id for logs and reports
url	string	Page URL to open, or a relative path/segment (see baseUrl below)
requiresAuth	boolean	If true, configure auth—see Authentication (--storage-state, optional --auth-script, and/or env vars for built-in login)

Optional fields include waitUntil, timeoutMs, scopeSelector, and per-target ignoreLocators.

Base URL (baseUrl / BASE_URL)

For object-root files only, you may set a single base URL for the whole file:

• baseUrl (recommended in JSON) or BASE_URL (handy in YAML) — must be an absolute URL with scheme (e.g. https://my.app).

If set, each target’s url may be:

• Absolute — contains a scheme (https://…, file://…, …); it is used as-is and ignores the base, or
• Relative — path or segment (e.g. /checkout, dashboard); it is resolved with the URL API against the base (same rules as a browser).

Relative URLs require a baseUrl / BASE_URL on the file. The JSON array shorthand cannot define a base; use an object root or keep every url absolute.

Examples: examples/targets.with-base.json, examples/targets.base-alias.yaml.

Root shape:

• Array — list of targets only (ignore rules at root are not supported in this shorthand; no file-level baseUrl).
• Object — { "targets": [...], "baseUrl"?: "...", "ignoreLocators": [...] } (alias: locatorIgnoreRules for file-level ignore lists; alias for base: BASE_URL).

See also examples/targets.sample.json, examples/targets.sample.yaml, and examples/targets.sample.toml.

Ignore patterns

File-level and per-target ignoreLocators use an ordered list of rules:

• kind: regex | glob | substring
• pattern: string to match against locator strings and fingerprint fields (data-testid, id, text snippets, etc.)
• reason (optional): audit note stored on ignored rows

Ignored elements get status: "ignored" in the report and are omitted from accepted-only exports. Use --no-ignore to bypass rules for debugging.

Authentication

For targets with requiresAuth: true, you must configure at least one of the following (combinations are allowed):

1. --storage-state <path> — Playwright storage state JSON (cookies / localStorage from a prior login).
2. --auth-script <path> — Your own ESM module for non-standard flows (runs before navigating to the target URL—see below).
3. Built-in login — Set credential environment variables so that after navigating to the target URL, the tool can try to fill a common login or sign-up form on whatever page you land on (often a login page after redirect).

You do not need --auth-script if storage state alone works, or if the built-in login matches your page.

Order of operations (per protected target)

1. Create the browser context (with --storage-state applied if you passed it).
2. Open a page.
3. If you passed --auth-script, run it first (your code typically navigates to a login URL and signs in).
4. page.goto(target.url) — this may redirect to a login or sign-up screen.
5. If you did not pass --auth-script, and AUTH_USER / AUTH_PASSWORD (or aliases) are set, run the built-in helper: it looks for visible password fields and typical email/username inputs, fills them, and clicks a matching submit button. Passing --auth-script turns the built-in helper off for that run (your script handles auth instead).
6. Run locator extraction on the current page.

Tip: Credential env vars trigger built-in login whenever --auth-script is omitted—including runs that use --storage-state. For cookie-only sessions, leave AUTH_USER, AUTH_PASSWORD, and their aliases unset.

Built-in login (no --auth-script)

Implemented in src/auth/defaultLogin.ts. It runs only when requiresAuth is true, --auth-script is omitted, and credentials exist in the environment.

Variable (use one per column)	Role
AUTH_USER, AUTH_EMAIL, LOCATOR_EXTRACT_AUTH_USER	Username or email
AUTH_PASSWORD, AUTH_PASS, LOCATOR_EXTRACT_AUTH_PASSWORD	Password

If no visible input[type=password] appears after navigation (for example, session cookies already logged you in), the helper skips without error.

Custom selectors or flows still require --auth-script.

Auth script (--auth-script)

Topic	Details
What it is	A JavaScript module (ESM) that the CLI import()s at runtime.
Where it lives	You choose the path—for example scripts/auth.mjs next to your project, or ../my-app/tools/auth.mjs. It is not baked into this repo except the template under examples/auth.example.mjs. Keep secrets out of git (use env vars and .gitignore for any local-only scripts).
File extension	Prefer .mjs, or .js in a package that has "type": "module" in package.json. CommonJS (.cjs / require) is not supported for this hook.
Exports	export default async function (page, context) => Promise<void>, or a named export auth with the same signature. See runExtraction.ts (default ?? auth).
Parameters	page: Playwright Page for the default tab in that context. context: BrowserContext if you need extra pages, grants, or storageState() later.
When it runs	Only when --auth-script is passed and requiresAuth: after context creation (and --storage-state if any), before page.goto(target.url). Use this when login must happen before hitting the target, or when the built-in helper is not enough.
Path resolution	The <path> you pass on the CLI is resolved with Node’s path.resolve() from the current working directory of your shell (where you run node dist/cli.js). Use absolute paths if cwd varies (CI vs laptop).

Minimal shape:

// scripts/my-auth.mjs
/** @param {import('playwright').Page} page */
/** @param {import('playwright').BrowserContext} context */
export default async function (page, context) {
  await page.goto("https://my.app/login");
  await page.getByLabel("Email").fill(process.env.AUTH_USER);
  await page.getByLabel("Password").fill(process.env.AUTH_PASS);
  await page.getByRole("button", { name: "Sign in" }).click();
  await page.waitForURL("**/dashboard**");
}

Example command:

export AUTH_USER=you@example.com
export AUTH_PASS='…'
node dist/cli.js extract \
  --targets targets.prod.yaml \
  --auth-script ./scripts/my-auth.mjs \
  -o report.json

A commented starter file is in examples/auth.example.mjs (copy it elsewhere and adapt).

Review workflow

• --review — prompts to accept or reject each non-ignored element (needs a TTY).
• --decisions-file <path> — JSON file { "decisions": { "<elementId>": "accepted" | "rejected" } }. See examples/decisions.sample.json.

If neither is used, non-ignored elements are auto-accepted for downstream fields.

Second pass: refine an existing draft report:

node dist/cli.js review report.json -o report.final.json

CLI reference (extract)

Option	Description
--targets <path>	Targets JSON / YAML / TOML
--targets-format <fmt>	json, yaml, or toml
--url <url>	Single URL (no targets file; no ignore rules from file)
--url-name, --requires-auth	Synthetic target name and auth flag for --url mode
--storage-state, --auth-script	Authentication—see Authentication; --auth-script disables built-in login; examples/auth.example.mjs is for custom --auth-script flows
--headed	Run Chromium visibly
--wait-until, --timeout-ms	Navigation behavior
--no-ignore	Skip ignore rules
--review, --decisions-file	Review behavior
-o, --out	Main report JSON (default report.json)
--final-out	Additional JSON with accepted locators only
--emit-snippets	Write a .ts file with generated page.* locator lines

Output

Reports include schemaVersion (currently 1—additive fields may still ship under this version until a breaking release), run metadata (tool version, Playwright version, targets file path, optional targetsBaseUrl when set in the targets file, auth/review mode), and pages with per-element elementId, fingerprints, ranked candidates with validation (unique / ambiguous / invalid), warnings, and optional status / reviewStatus.

Each element may also include automation-oriented suggestedKeys: camelCase and snake_case strings derived from the fingerprint (priority: data-testid, id, form name/type, aria-label, visible text, heading/href/placeholder, then role/tag fallback). Keys are prefixed with a slug of the target name, made unique per page with numeric suffixes when stems collide (…, …_2, …_3), and suffixed with an element kind: _link, _button, _text, _input, or _input_field (text-like <input> fields such as email/password/text—resolved before generic role=textbox so keys look like …_email_input_field / …PasswordInputField in camelCase). Headings, labels, and textareas use _text; selects and non-text inputs use _input. suggestedKeySource records which rule produced the stem (before prefix/collision/kind). Treat these as hints—dynamic copy and locale changes may still require renames or review.

Development

npm install
npm run build
npm test          # Vitest
npm run test:watch

Fixtures and samples live under fixtures/ and examples/.

Project layout

src/
  cli.ts              # Commander CLI (extract, review)
  config/             # Load & validate targets (JSON / YAML / TOML)
  extract/            # DOM collect, candidates, validation, suggested keys, run pipeline
  auth/               # Built-in login helper after navigation
  ignore/             # Ignore-rule matching
  review/             # Interactive review & decisions file
  report/             # Report types & JSON emission
tests/                # Vitest tests
examples/             # Sample targets and decisions
fixtures/             # Local HTML for tests
.cursor/              # Optional Cursor rules & skills for this repo

Documentation maintenance

When you change the CLI, targets/schema, report shape, or install/run steps, update README.md (and any markdown under docs/ if you add that folder). Cursor users: see .cursor/rules/documentation-sync.mdc (always-on reminder) and .cursor/skills/documentation-sync/ for a detailed sync checklist.

License

Add a LICENSE file in the repository if you intend to publish under a specific license.