Skip to content

APP.md.spec.md

Latest commit

 

History

History
348 lines (247 loc) · 21.5 KB

APP.md.spec.md

File metadata and controls

348 lines (247 loc) · 21.5 KB
Raw

APP.md Specification

Status: every field below is backed by the current code. Fields that are parsed but not yet read by any consumer are flagged inline as (parsed only) so you know not to rely on them.

APP.md is a per-app description the developer writes once to give mirroir a map before exploration. It is NOT generated by the explorer — it's the developer's source of truth for what the app looks like and how it behaves.

Conceptual Model

Before the field list, a word on vocabulary. APP.md talks about a few distinct window regions, and the word "tab" is ambiguous in iOS literature. This spec uses these terms precisely:

Term What it means in APP.md Examples
Status bar OS chrome at the top — time, battery, signal. Never tappable by the explorer. iOS status bar; iPadOS top strip.
Nav bar Per-screen bar at the top holding the screen title and navigation chrome (back chevron, edit buttons). One per screen. iOS UINavigationBar; "< Back Settings Edit" strip.
Tabs / edge controls A cluster of primary entry points arranged along one window edge. The explorer treats them as high-priority breadth targets — first things to try, known to switch screens or trigger core actions. iPhone bottom tab bar (Messages, Calendar, Safari); iPad sidebar; Tapo's right-side camera-control rail; Instagram's 5 bottom icons.
Content Everything else inside the window. Cards, rows, charts, chat threads, etc. The explorer plans these per-screen. Lists of chats, grids of photos, settings rows.

When APP.md says ## Tabs or ## Tab Layout, it's describing the edge-control cluster — not strictly an iOS UITabBar. The field covers:

  • A horizontal strip of 3–5 icons at the bottom (iPhone standard).
  • A vertical column of icons on a window edge (Tapo camera controls, iPad sidebar).
  • A top-edge toolbar with a fixed set of action buttons (some editing apps).

What it explicitly is not:

  • Arbitrary chevron rows in a settings list — those are content (table rows), detected per-screen.
  • The back button in the nav bar — that's chrome, not a tab.
  • Transient overlay actions (floating share sheet, keyboard quick-actions) — those aren't geometry-stable enough to name.

The explorer uses the declared edge-control cluster for two things:

  1. Priority targets — inject into the per-screen plan as breadth-navigation items so they're visited before deeper content.
  2. Geometric fill-in — when OCR/YOLO only detects a subset of the expected N icons, extrapolate the missing positions along the declared axis and edge.

If your app has no edge-control cluster (full-screen game, single-view utility), you omit ## Tabs and ## Tab Layout entirely.

File Format

UTF-8 Markdown with an optional YAML front-matter block. File name must be APP.md (case-insensitive match during loading, but APP.md is canonical). Recommended location:

  • Project-local: <cwd>/.mirroir-mcp/skills/patterns/apps/<AppName>/APP.md
  • Global: ~/.mirroir-mcp/skills/patterns/apps/<AppName>/APP.md
  • Sibling skills repo: <cwd>/../mirroir-skills/patterns/apps/<AppName>/APP.md

The loader matches the app: front-matter field against the app_name parameter of generate_skill using diacritics-insensitive, case-insensitive comparison ("Sante" → "Santé").

File Structure

---
# Front matter (YAML-ish, flat key:value)
app: AppName           # required
version: 1             # reserved for future versioning
locale: fr_CA          # optional — prefer matching locales
archetype: dashboard   # optional — screen recipe name
reset_before_explore: true   # optional — force-quit before explore
obstacle_mode: auto    # optional — auto | hint | off
---

## Structure            # required (or at least one of Obstacles / Skip)

Free-form human prose describing the app. Becomes AI context in generated skills.

## [TabName] Tab        # optional, one per tab

Per-tab description. Body is injected into the generated skill's context.

## Tabs                 # optional — explicit tab list, alternative to "N tabs: Tab1, Tab2"

- Tab One
- Tab Two

## Tab Layout           # optional — see §Tab Layout

- orientation: vertical
- edge: right

## Obstacles            # optional — auto-dismissal rules

- Permission dialog → tap "Allow"
- Paywall → dismiss via "Not Now"

## Skip                 # optional — element text never to tap

- Delete Account
- Sign Out

## Credentials          # optional — parsed only, no consumer today

- email: ${TEST_EMAIL:-test@example.com}
- password: ${TEST_PASSWORD}

## Tips                 # optional — AI guidance, goes into skill context

- Dashboard cards can show empty state on test devices
- Swipe down on chart detail view to return

Front-Matter Fields

Field Type Required Behavior
app string ✅ yes Display name. Matching is diacritics-insensitive + case-insensitive.
version int optional APP.md schema version (carried on AppDescription.schemaVersion). The only supported version is 1. A file declaring a higher version logs a warning at load time and is still parsed best-effort. Defaults to 1 when absent.
locale string optional e.g. fr_CA. When multiple APP.md files match the same app, a locale match takes priority over a locale-less file.
archetype string optional Name of a screen recipe (e.g. dashboard, social-feed, settings-list). When set, bypasses recipe auto-detection.
reset_before_explore bool optional true / yes / 1 → explorer force-quits the app via App Switcher before launching. Used for apps that retain stateful overlays (TikTok, Instagram). Default: false.
obstacle_mode enum optional auto (default): dismiss matching obstacles. hint: surface to AI but don't act. off: ignore rules.
spotlight_name string optional Simulator-only. What launch_app types into Spotlight to reach the app in FakeMirroring. Defaults to app. Read by SimulatorSpecParser only when ## Simulator … sections are present.
icon string optional Simulator-only. Single glyph rendered on the FakeMirroring home screen and App Switcher card. Defaults to the first character of app. Read by SimulatorSpecParser only when ## Simulator … sections are present.

Sections

All section headings use ## (H2). H1 (#) is treated as a title and skipped. Any unrecognized ## heading is ignored unless it matches a ## [Name] Tab pattern (see Tabs).

## Structure (required-ish)

Free-form prose. Included verbatim in the AI "App Context" block of generated skills. This is where you describe the app's overall shape: what screen appears on launch, what tabs exist, what kinds of content each tab holds.

If you use the inline convention tabs: A, B, C (or N tabs: A, B, C), mirroir extracts those names and treats them as high-priority navigation targets.

## [Name] Tab (optional, repeated)

Per-tab descriptions. Any heading that ends with Tab or tab contributes its body to AI context and its name to the tabs list (e.g., ## Résumé Tab → tab name Résumé).

## Tabs (optional, recommended over inline)

Explicit bullet list of tab names:

## Tabs
- Pour toi
- Explorer
- Profil

Equivalent to tabs: Pour toi, Explorer, Profil in the Structure section, but clearer.

## Tab Layout

Geometric hints for the edge-control finder. Without this section, the explorer assumes a horizontal bottom tab bar (iOS convention). Apps with a vertical side rail (Tapo camera controls, iPad sidebar) declare their geometry here.

## Tab Layout
- orientation: vertical     # vertical | horizontal (default: horizontal)
- edge: right               # top | bottom | left | right (default: bottom)

When declared, TabTargetInjector.findTargets (Sources/mirroir-mcp/TabTargetInjector.swift) uses the layout hint to pick which window edge to sample (a 12%-wide band) and which axis the bar runs along. Text-match remains the primary strategy; the layout hint only governs the geometric fallback when OCR text matching leaves tabs unmatched. The fallback does not sort detected candidates and ordinal-map them. Instead, synthesizeAnchors (TabTargetInjector.swift:124–156) computes N evenly-spaced anchor points from the declared tab count and the bar geometry — each anchor's main-axis position is (index + 0.5) × axis / tabCount, independent of which individual icons were detected. The cross-axis position comes from the median of the band's detected (text-less) icon points, falling back to a geometric edge offset. Synthesis is gated by a presence check: the declared band must hold at least EnvConfig.tabSynthesisMinZoneEvidence elements, so a detail screen whose band happens to hold a stray element gets no phantom anchors. BFSExplorerHelpers only calls TabTargetInjector.inject, which wraps findTargets.

For the tab-synthesis fallback the axis is fixed by the declared orientation (horizontal → main axis X, vertical → main axis Y); the geometry comes from APP.md, not from inspecting the detections. The separate general icon-detection helper IconDetector.interpolateEvenSpacing does auto-detect a dominant axis from raw detections (extrapolating vertically when detections span a larger Y range than X range, otherwise horizontally), but that step backfills the icon list earlier in the pipeline and does not decide which axis synthesizeAnchors walks.

## Obstacles (optional)

Rules for auto-dismissing interstitials (permission dialogs, paywalls, onboarding overlays):

- <trigger text> → <action text>

Both → and -> are accepted as separators. Common action prefixes (tap , dismiss via , dismiss ) are stripped. Quoted action text is unwrapped.

Example:

- Health Access permission → tap "Autoriser"
- Notification permission → tap "Ne pas autoriser"
- Paywall → dismiss "Not Now"

When obstacle_mode: auto (default), the explorer checks these rules after each OCR pass and taps the action before continuing. obstacle_mode: hint surfaces rules as AI context without acting; off disables the mechanism.

Explorer caveat: only the BFS explorer honors these app-specific obstacle rules — BFSExplorerExploring passes obstacles: appObstacles into ExplorerUtilities.dismissAlertIfPresent. The DFS explorer calls dismissAlertIfPresent without obstacle rules, so it dismisses generic system alerts only and ignores ## Obstacles.

## Skip (optional)

Element text patterns the explorer must never tap. Merged with permissions.json.skipElements into a single global skip list for the session.

- Delete Account
- Sign Out
- Réinitialiser les données

## Credentials (optional)

Key-value pairs with ${VAR} / ${VAR:-default} substitution from the process environment. AppDescriptionLoader.resolveVariables applies the substitution at load time. SkillMdGenerator reads the resulting map and emits a ## Required Credentials section into the generated skill listing the declared key names only (sorted). Resolved env-var values never leak into the skill file — the section tells the AI what the flow needs while keeping secrets out of the artifact.

- email: ${TEST_EMAIL:-test@example.com}
- password: ${TEST_PASSWORD}

## Tips (optional)

Free-form exploration hints. Appended to the generated skill's context so the AI has extra guidance.

- Swipe down on detail views to return to the summary
- Some cards show "Aucune donnée" on test devices — treat as normal

## Simulator … family (optional — FakeMirroring scene graph)

When an app is ported to the FakeMirroring multi-app simulator, its APP.md gains a declarative scene graph alongside the guidance fields above. On the mirroir-mcp side, SimulatorSpecParser.parse extracts these into a SimulatorSpec carried on AppDescription.simulator (AppDescription.swift:150), but no runtime consumer reads that field — it is parsed only there. FakeMirroring re-parses the same APP.md independently: AppPackLoader.makePack reads the file from disk and calls SimulatorSpecParser.parse again to build a runnable AppPack at startup (AppRegistry.swift:95–105). The two processes never exchange the SimulatorSpec value; each derives its own from the file. The parser returns nil — leaving simulator unset (and FakeMirroring skipping the file) — when none of these sections are present, so guidance-only APP.md files are unaffected.

Three H2 section shapes are recognized:

  • ## Simulator — root/metadata block. - root: <screenID> names the launch screen. When omitted, the lexicographically-first declared screen ID wins (else "main").
  • ## Simulator Screen <id> — one per screen. Properties via - key: value bullets (title, back → back-chevron destination screen ID, tab_bar: true → render the bottom bar). Elements via - element <kind>: … bullets, kept in declaration order. A destination is written with → or -> after the text. Element kinds: row: Text -> dest (disclosure chevron), button: Text -> dest (pill, destination optional), text: Text (non-interactive), tab: Text -> dest (bottom-bar entry, destination required), textfield: "placeholder" #fieldID, slider: "label" #fieldID = 0.5 (fraction clamped 0.0–1.0), placeholder: 370x175 (gray media block, width×height in points).
  • ## Simulator Obstacle <id> — one per modal. Carries a title, optional body, dismissal buttons, and a trigger: on_first_describe, after_n_taps:<count>, or never.

The spotlight_name and icon front-matter fields (above) feed the same SimulatorSpec and are only read when at least one ## Simulator … section exists.

## Simulator
- root: home

## Simulator Screen home
- title: Home
- tab_bar: true
- element row: Settings -> settings
- element tab: Home -> home
- element tab: Profile -> profile

## Simulator Obstacle welcome
- title: Welcome
- body: Thanks for trying the app.
- buttons: Continue
- trigger: on_first_describe

Loading & Resolution

  1. AppDescriptionLoader.load(appName:) is called when generate_skill(action:"explore", app_name:"X") starts.
  2. Search paths, in the order AppDescriptionLoader.searchPaths() walks them (recursive file search per path):
    1. <cwd>/.mirroir-mcp/skills/**/APP.md
    2. ~/.mirroir-mcp/skills/**/APP.md
    3. <cwd>/../mirroir-skills/patterns/apps/**/APP.md
    4. <cwd>/.mirroir-mcp/skills/patterns/apps/**/APP.md
    5. ~/.mirroir-mcp/skills/patterns/apps/**/APP.md
    6. <cwd>/../mirroir-skills/apps/**/APP.md (legacy fallback)
    7. <cwd>/../mirroir-skills/**/APP.md (legacy fallback)
  3. Each file is parsed and its app: field matched against the requested app name (diacritics-insensitive, case-insensitive).
  4. Locale precedence:
    • If any APP.md's locale: matches Locale.current.identifier exactly (case-insensitive), the first such match encountered during the walk wins and is returned immediately.
    • Otherwise the loader keeps a fallback: the first matching description is tentatively chosen, and any later locale-less description replaces a locale-specific fallback (so a generic APP.md wins over a mismatched localized one). All other cases keep the earliest fallback.
  5. No match → exploration proceeds without a description (no auto-obstacles, no tab hints, no archetype override).

Permission System Bridge

APP.md and permissions.json are complementary. permissions.json governs what the MCP server may do at all (which tools are exposed, which apps may be touched); APP.md describes what a specific app looks like. They overlap in exactly two places today.

Concern APP.md permissions.json
Global MCP tool allow/deny (tap, swipe, …) — ✅ allow / deny
Per-app tool allow/deny (during explore) — ✅ perApp
App-level kill-switch (no launch / no explore) — ✅ blockedApps
Element skip patterns (global) — ✅ skipElements
Element skip patterns (per-app) ✅ ## Skip —
Auto-dismiss obstacle rules ✅ ## Obstacles —
Force-quit before explore ✅ reset_before_explore —
Archetype recipe override ✅ archetype —
Edge-control layout hints ✅ ## Tab Layout —

The three enforced bridge points:

  1. Blocked-app guard — shared with launch_app. generate_skill(action:"explore", app_name:"X") calls policy.checkAppLaunch(appName) before doing anything with the target. If permissions.json.blockedApps contains the name, the call returns the same error launch_app would return. Any APP.md for a blocked app is loaded but never used because exploration refuses to start.

    • Scope note: generate_skill(action:"start") does not call this guard today. Only explore.

  2. Per-app tool gate at explore start. permissions.json.perApp can scope tool allow/deny to a specific app. Before exploration begins, policy.toolsDenied(for:requiredTools:) checks that the minimum explorer tools (tap, swipe, type_text, press_key) are permitted for the app. If any are denied, the call refuses with a message naming the specific denied tools.

    {
  "allow": ["*"],
  "perApp": {
    "Banking": { "deny": ["type_text", "open_url"] }
  }
}

    Precedence: per-app deny always wins, even over global allow. Per-app allow can open a tool that global rules would close.

  3. Skip-list merge with dedup. At explore start, GenerateSkillHandlers.mergeSkipPatterns flattens three layers into one ExplorationBudget.skipPatterns:

    • Per-app (APP.md ## Skip) — prepended first, so its casing wins on duplicates.
    • Global (permissions.json.skipElements).
    • Built-ins (ExplorationBudget.builtInSkipPatterns).

    Case-insensitive duplicates are collapsed; the first occurrence keeps its casing. Source counts are emitted to the debug log at session start.

Outside those three points, the systems are disjoint. AppDescription.credentials has no counterpart in permissions.json; its only consumer is SkillMdGenerator, which surfaces the declared key names (never values) in the generated skill (see the Credentials section).

Consumers (what today's code actually uses)

APP.md field Consumer
app, locale AppDescriptionLoader for matching
archetype GenerateSkillHandlers.handleExplore — looks up recipe by name, sets on session
reset_before_explore GenerateSkillHandlers.handleExplore — force-quit via App Switcher
obstacle_mode, obstacles BFSExplorerExploring — passed to ExplorerUtilities.dismissAlertIfPresent
skipElements GenerateSkillHandlers.handleExplore — merged into ExplorationBudget.skipPatterns
tabs TabTargetInjector.inject/findTargets — injected as high-priority breadth-navigation targets (called from BFSExplorerHelpers)
tabLayout TabTargetInjector.findTargets — picks which edge to sample and which axis the bar runs along for the geometric synthesis fallback
context SkillMdGenerator — written verbatim into generated skill's "App Context" section
credentials SkillMdGenerator — declared key names only rendered as a ## Required Credentials section in the generated skill. Resolved env-var values never leak into the skill file.
hints SkillMdGenerator — rendered as a ## Tips section in the generated skill (parsed from ## Tips; intentionally excluded from context to avoid duplicate rendering).
simulator Parsed-only on the mirroir-mcp side: AppDescriptionLoader carries it on AppDescription, but no runtime consumer reads it. FakeMirroring re-parses the same APP.md independently (AppPackLoader.makePack) to build its AppPack; the two never exchange the value. nil for guidance-only APP.md files not yet ported to the simulator.

Examples

See ../mirroir-skills/patterns/apps/ for live examples (Santé, Settings, Weather, TikTok, Instagram). A minimal example showing a vertical edge-control rail (Tapo's camera controls):

---
app: Tapo
archetype: dashboard
---

## Structure

Landscape camera-viewer app. The live feed fills the window; a vertical
column of 5 controls hugs the right edge (camera snapshot, video record,
pan joystick, mic, phone handoff).

## Tabs

- Camera
- Video
- Joystick
- Mic
- Phone

## Tab Layout

- orientation: vertical
- edge: right

## Obstacles

- Permission dialog → tap "Allow"

## Skip

- Delete device

What the code does with this file:

  • Text matching runs first. If any OCR element's text matches a tab name (e.g. "Camera") it becomes a breadth-navigation target.
  • For names that didn't match text, TabTargetInjector.findTargets samples the right-edge 12% band (the declared edge: right). When that band holds at least EnvConfig.tabSynthesisMinZoneEvidence elements, synthesizeAnchors computes 5 evenly-spaced anchors down the rail — anchor i sits at y = (i + 0.5) × height / 5, with the shared rail X read from the median of the band's detected icon points. The anchors come from the declared tab count, not from sorting individual detections, so noisy 3-of-5 icon detection does not mis-assign tabs.
  • The Permission dialog → tap "Allow" rule auto-dismisses the iOS privacy prompt on first launch via ExplorerUtilities.dismissAlertIfPresent.

Known Gaps

These are observations about the current code, not commitments to change it.

  • AppDescription.locale is used at load time for matching the system locale but is not inspected during exploration.