Flow Test Engine

A TypeScript-based API testing engine for writing rich, declarative flows in YAML. It supports request chaining, variable interpolation, flexible assertions, conditional scenarios, and clean reports in JSON and HTML.

Flow Test Engine is a language-agnostic runner for API suites written in YAML. You can bolt it onto any repository—regardless of the tech stack—to manage HTTP flows, capture variables, and publish reports.

📚 Quick Links

5-Minute Quick Start - Get running fast
Cheat Sheet - Quick syntax reference
Examples - Ready-to-use templates
Development Workflow - Best practices
Full Documentation - Complete guides

✨ Features at a Glance

✅ Simple YAML syntax - Write tests in plain YAML, no code needed
✅ Dynamic data generation - Built-in Faker.js for realistic test data
✅ Variable interpolation - Capture and reuse data across steps
✅ Comprehensive assertions - Validate status, headers, body with rich operators
✅ Conditional logic - Scenarios, branching, and dynamic flows
✅ Retry strategies - Configurable retries for resilient testing
✅ CI/CD ready - Works with GitHub Actions, GitLab CI, Jenkins, etc.
✅ HTML reports - Beautiful, detailed test reports
✅ Language agnostic - Works with any tech stack (Node, PHP, Python, Java, etc.)

1. Requirements

Node.js 16 or newer (runtime used by Flow Test Engine)
npm 8 or newer (ships with Node.js)
Docker Desktop (optional, only if you want the bundled httpbin mock server)

📌 Install Node.js even if your application is written in another language—Flow Test Engine runs beside your project and never touches your runtime code.

2. Quick Start with `npx` (no install required)

Use npx to try the engine without adding dependencies:

# Initialise configuration and sample suites in the current directory
npx --yes flow-test-engine init
# or using the short form:
npx --yes fest init

# Execute all discovered suites using the generated config
npx --yes flow-test-engine --config flow-test.config.yml
# or using the short form:
npx --yes fest --config flow-test.config.yml

The wizard creates:

flow-test.config.yml with discovery rules, retries, and reporting options.
tests/ containing starter YAML suites you can edit or replace.
results/ where execution artifacts are stored (results/latest.json).

3. Make Flow Test Part of Your Repository

For long-term use, keep Flow Test assets in their own workspace so teammates and CI can run them consistently.

mkdir flow-tests
cd flow-tests
npm init -y
npm install --save-dev flow-test-engine
npx flow-test-engine init
# or using the short form:
npx fest init

This produces a flow-tests/package.json similar to:

{
  "name": "flow-tests",
  "private": true,
  "scripts": {
    "flow-test": "fest --config flow-test.config.yml",
    "flow-test:verbose": "fest --config flow-test.config.yml --verbose"
  },
  "devDependencies": {
    "flow-test-engine": "^1.0.2"
  }
}

A minimal suite (flow-tests/tests/my-first-test.yaml) might look like:

suite_name: "Payment API Smoke"
base_url: "https://sandbox.my-api.com"

auth:
  type: "bearer"
  token: "{{$env.API_TOKEN}}"

steps:
  - name: "Get payment"
    request:
      method: "GET"
      url: "/payments/{{payment_id}}"
    assert:
      status_code: 200
      body:
        id: { equals: "{{payment_id}}" }
        status: { in: ["CREATED", "CONFIRMED"] }

Run it from anywhere in your repo:

npm --prefix flow-tests run flow-test
# or, with npx
npx --yes --package flow-test-engine flow-test-engine --config flow-tests/flow-test.config.yml
# or using the short form:
npx --yes --package flow-test-engine fest --config flow-tests/flow-test.config.yml

Cross-suite step calls (`call`)

Need to reuse an existing step that already lives in another YAML suite? Add a call block to the step instead of duplicating the HTTP definition:

steps:
  - name: "Seed reusable user"
    call:
      test: "../shared/setup-user.yaml"   # always relative to the current file
      step: "create-user"                 # matches step_id or name from the target suite
      variables:
        role: "admin"                     # optional values injected into the called step
      isolate_context: true                # default; restore local variables after the call
      on_error: "warn"                    # fail | warn | continue (continue marks the step as skipped)

Paths are sandboxed to the configured test_directory; absolute paths or .. escapes are rejected.
isolate_context defaults to true. When enabled, captured variables from the remote suite are returned with the node_id.variable namespace so you can safely consume them without leaking the called suite's runtime state.
Set isolate_context: false if you want the called step to mutate the current variable scope in place.
call steps cannot define request, iterate, input or scenarios—they delegate entirely to the target step.
Recursive chains are blocked (max depth 10) so you get a friendly error instead of an infinite loop.

4. Everyday CLI Patterns

fest --dry-run --detailed              # Discover tests and show plan
fest --suite auth,checkout             # Run specific suites
fest --priority critical,high          # Focus on high-impact scenarios
fest --tag smoke                       # Filter by YAML tags
fest --config ./flow-tests/staging.yml # Point to another config file
fest --report json,csv                 # Produce multiple report formats
fest graph mermaid --output discovery.mmd # Generate Mermaid discovery graph file
fest graph --direction LR --no-orphans    # Print graph left-to-right skipping orphans
fest schema --format json              # Export engine schema for IDE extensions

Schema Export for IDE Extensions

The Flow Test Engine exposes a complete schema catalog that IDE extensions can use to provide rich autocomplete, validation, and documentation:

# Export schema to file
fest schema --format json > flow-test-engine.schema.json

# Inspect specific structures
fest schema --format json | jq '.structures.TestSuite'

# View available examples
fest schema --format json | jq '.examples[].name'

The schema includes:

✅ Complete structure definitions (TestSuite, TestStep, Assertions, etc.)
✅ All supported properties with types, descriptions, and examples
✅ Interpolation patterns (Faker, JavaScript, environment variables)
✅ Complete YAML examples categorized by complexity
✅ CLI documentation (commands, flags, usage)

For extension developers: See guides/12.schema-catalog-guide.md for detailed integration examples.

Variable Interpolation System

Flow Test Engine supports powerful variable interpolation with two syntaxes:

# Template syntax (universal, works everywhere)
url: "{{$env.BASE_URL}}/api/{{version}}/users"
email: "{{$faker.internet.email}}"
timestamp: "{{$js:Date.now()}}"

# Direct syntax (new in v2.0, cleaner)
email: "#faker.internet.email"
timestamp: "$Date.now()"
query: "@response.data[0].id"

Key Features:

✅ Both syntaxes fully supported forever (no breaking changes)
✅ Environment variables: {{$env.VAR_NAME}} (reads FLOW_TEST_VAR_NAME)
✅ Faker test data: #faker.person.name or {{$faker.person.name}}
✅ JavaScript expressions: $Date.now() or {{$js:Date.now()}}
✅ JMESPath queries: @body.data[0] (extract from responses)
✅ Cross-suite variables: {{other-suite.auth_token}}

📖 Complete Reference: guides/interpolation-complete-reference.md 📝 See Example: tests/examples/syntax-showcase.yaml

5. Integration Recipes by Ecosystem

The commands below assume you created the dedicated flow-tests/ workspace.

Node.js / TypeScript projects

Install the engine next to your source:

npm install --save-dev flow-test-engine
npx flow-test-engine init
# or using the short form:
npx fest init

Add scripts to package.json:

{
  "scripts": {
    "flow-test": "fest --config flow-test.config.yml",
    "flow-test:ci": "fest --config flow-test.config.yml --report json"
  }
}

Run locally or in CI:
```
npm run flow-test
npm run flow-test:ci
```

PHP / Laravel (Composer)

Keep Flow Test files under flow-tests/ as described earlier.

Reference that workspace from composer.json:

{
  "scripts": {
    "flow-test": "npm --prefix flow-tests run flow-test",
    "flow-test:ci": "npm --prefix flow-tests run flow-test -- --report=json"
  }
}

Execute with Composer:

composer run flow-test
composer run flow-test:ci

In CI (GitHub Actions example):

- uses: actions/setup-node@v4
  with:
    node-version: '18'
- run: npm ci --prefix flow-tests
- run: composer run flow-test:ci

Java / Spring (Maven)

Ensure Node.js is available on the build agent.

Add a Maven execution in pom.xml:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>exec-maven-plugin</artifactId>
  <version>3.1.0</version>
  <executions>
    <execution>
      <id>flow-tests</id>
      <phase>verify</phase>
      <goals><goal>exec</goal></goals>
      <configuration>
        <workingDirectory>${project.basedir}/flow-tests</workingDirectory>
        <executable>npx</executable>
        <arguments>
          <argument>fest</argument>
          <argument>--config</argument>
          <argument>flow-test.config.yml</argument>
        </arguments>
      </configuration>
    </execution>
  </executions>
</plugin>

Run with Maven:
```
mvn verify
```

Python / Django

Add a Makefile (or extend an existing one) so everyone runs the same command:

# Makefile
flow-test:
	@npm --prefix flow-tests run flow-test

flow-test-ci:
	@npm --prefix flow-tests run flow-test -- --report=json

Usage:

make flow-test
make flow-test-ci

In a tox.ini you can also add:

[testenv:flow-tests]
commands = npm --prefix {toxinidir}/flow-tests run flow-test

Dart / Flutter

Create tool/flow_test.dart to bridge into Node.js:

import 'dart:io';

Future<void> main() async {
  final result = await Process.start(
    'npx',
    ['--yes', 'fest', '--config', 'flow-tests/flow-test.config.yml'],
    mode: ProcessStartMode.inheritStdio,
  );
  final exitCode = await result.exitCode;
  if (exitCode != 0) {
    exit(exitCode);
  }
}

Add a script to pubspec.yaml (if you use melos or simple make targets):

# pubspec.yaml
scripts:
  flow-test: dart run tool/flow_test.dart

Run:

dart run tool/flow_test.dart
# or with the scripts plugin
dart run flow-test

Pure Terminal / CI Pipelines

If your project has no build tool, keep it simple:

npm ci --prefix flow-tests             # install once in CI
npm --prefix flow-tests run flow-test  # execute suites

Or run ad hoc:

npx --yes flow-test-engine --config flow-tests/flow-test.config.yml
# or using the short form:
npx --yes fest --config flow-tests/flow-test.config.yml

If you rely on Docker for dependent services, start them before running Flow Test. The sample docker-compose.yml in the Flow Test repository spins up httpbin:

docker compose up -d httpbin
npm --prefix flow-tests run flow-test
docker compose down -v

6. Reporting and Dashboards

Every execution writes a JSON artifact at flow-tests/results/latest.json. When you need a shareable visualization, enable HTML output with the --html-output flag (optionally pass a subdirectory name) or by adding html to reporting.formats in flow-test.config.yml. The engine now emits a Postman-inspired HTML experience alongside the JSON, and the Astro dashboard packaged with this repository reads the same data, so keep it in sync before starting the UI.

Copy (or symlink) the latest results into the dashboard data folder:
```
mkdir -p report-dashboard/src/data
cp flow-tests/results/latest.json report-dashboard/src/data/latest.json
# or keep it synced:
ln -sf ../flow-tests/results/latest.json report-dashboard/src/data/latest.json
```
The dashboard looks for the browser-facing file at report-dashboard/src/data/latest.json, while server-side rendering also checks ../results/latest.json. If neither is present it automatically reads reporting.output_dir from your flow-test.config.* file to locate the latest report. Either path can be kept fresh in CI.
Launch the dashboard:
```
npm install --prefix report-dashboard        # first run only
npm run --prefix report-dashboard dev
```
Open http://localhost:4321/flow-test/ (Astro’s default port). The UI reloads automatically when src/data/latest.json changes.
- Prefer to serve it at the root path during local development? Run the command with PUBLIC_BASE_URL=/ npm run --prefix report-dashboard dev and open http://localhost:4321/ instead.
Generate a static bundle when you need to publish the report:
```
npm run --prefix report-dashboard build
```
The build lands in report-dashboard/dist/. Deploy that folder and ship the matching src/data/latest.json (or automate a copy step) so the hosted dashboard can load the latest results.

Example CI step (GitHub Actions):

- run: npm ci --prefix flow-tests
- run: npm --prefix flow-tests run flow-test
- run: npm ci --prefix report-dashboard
- run: cp flow-tests/results/latest.json report-dashboard/src/data/latest.json
- run: npm run --prefix report-dashboard build

🤝 Contributing

We welcome contributions! Whether it's bug reports, feature requests, or pull requests, your help makes Flow Test Engine better.

Quick links:

📖 Contributing Guide — How to set up your development environment and submit PRs
🗺️ Roadmap — Planned features and evolution strategy
🐛 Issue Tracker — Report bugs or request features
💬 Discussions — Ask questions and share ideas

Development Setup

git clone https://github.com/marcuspmd/flow-test.git
cd flow-test
npm install
npm run build
npm test

See CONTRIBUTING.md for detailed instructions.

📈 Roadmap

The project is actively evolving. Key focus areas:

🔌 VS Code Extension — IntelliSense and validation for YAML suites
🚀 Zero-Config Deploy — GitHub Action for automatic dashboard deployment
📊 Execution History — Trend analysis across test runs
🧪 Integrated Mock Server — Offline testing without Docker

See the full Roadmap for details and timeline.

License

MIT – see package.json for details.

Made with ❤️ by Marcus

Name		Name	Last commit message	Last commit date
.claude		.claude
.github		.github
docs		docs
examples		examples
guides		guides
public		public
scripts		scripts
src		src
tasks/architecture-improvements		tasks/architecture-improvements
tests		tests
.DS_Store		.DS_Store
.env.example		.env.example
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CHEATSHEET.md		CHEATSHEET.md
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
COVERAGE_TODO.md		COVERAGE_TODO.md
Dockerfile		Dockerfile
GEMINI.md		GEMINI.md
IMPROVEMENTS-SUMMARY.md		IMPROVEMENTS-SUMMARY.md
PROGRESS.md		PROGRESS.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
api-documenter.json		api-documenter.json
api-extractor.json		api-extractor.json
cli.new.ts		cli.new.ts
docker-compose.yml		docker-compose.yml
flow-test.config.yml		flow-test.config.yml
jest.config.js		jest.config.js
jsdoc.json		jsdoc.json
package-lock.json		package-lock.json
package.json		package.json
schema-catalog.json		schema-catalog.json
tsconfig.json		tsconfig.json

Navigation Menu

Repository files navigation

Flow Test Engine

📚 Quick Links

✨ Features at a Glance

1. Requirements

2. Quick Start with `npx` (no install required)

3. Make Flow Test Part of Your Repository

Cross-suite step calls (`call`)

4. Everyday CLI Patterns

Schema Export for IDE Extensions

Variable Interpolation System

5. Integration Recipes by Ecosystem

Node.js / TypeScript projects

PHP / Laravel (Composer)

Java / Spring (Maven)

Python / Django

Dart / Flutter

Pure Terminal / CI Pipelines

6. Reporting and Dashboards

🤝 Contributing

Development Setup

📈 Roadmap

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Flow Test Engine

📚 Quick Links

✨ Features at a Glance

1. Requirements

2. Quick Start with npx (no install required)

3. Make Flow Test Part of Your Repository

Cross-suite step calls (call)

4. Everyday CLI Patterns

Schema Export for IDE Extensions

Variable Interpolation System

5. Integration Recipes by Ecosystem

Node.js / TypeScript projects

PHP / Laravel (Composer)

Java / Spring (Maven)

Python / Django

Dart / Flutter

Pure Terminal / CI Pipelines

6. Reporting and Dashboards

🤝 Contributing

Development Setup

📈 Roadmap

License

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

2. Quick Start with `npx` (no install required)

Cross-suite step calls (`call`)