Interactive demo for the ai-form-response-extractor library — extract structured data from scanned/photographed paper forms and digital PDFs using multimodal LLMs.
- Multi-step wizard — Configure provider, upload images/PDFs, paste a SurveyJS form definition
- Built-in sample data — Predefined test datasets with scanned form images and SurveyJS definitions
- Multi-provider support — OpenAI, Anthropic, and local inference via Ollama
- Real-time validation — Basic JSON validation and runtime checks for provider configuration
- Rich extraction results — Overall and per-field confidence scores, extracted JSON output, SurveyJS form preview with pre-filled data
Note: Feature details depend on the configured LLM provider and model capabilities.
npm installCopy the example environment file:
cp .env.example .env.localThen edit .env.local to configure at least one provider:
# OpenAI
OPENAI_API_KEY=sk-your-openai-api-key-here
# Anthropic
ANTHROPIC_API_KEY=sk-ant-your-anthropic-api-key-hereOllama lets you run vision-capable models locally without API keys:
- Install from ollama.ai
- Start the local server:
ollama serve
- Pull a vision model (example):
ollama pull llama3.2-vision
The demo auto-detects whether Ollama is running on startup.
npm run devOpen http://localhost:3000.
- Provider setup — Select an LLM provider and model. API keys are read from server-side environment variables and are never exposed to the client.
- Input selection — Choose a built-in test dataset or provide your own files and SurveyJS schema.
- Upload step — Upload scanned form images or digital PDFs, or use preloaded samples.
- Schema definition — Provide a SurveyJS JSON definition describing expected fields, or use a sample schema.
- Processing — The backend sends the input to
ai-form-response-extractor, which uses a vision-language model to extract structured data. - Review — The output includes extracted JSON, confidence metrics, and a SurveyJS form preview populated with the extracted values.
Some PDF workflows may require image conversion capabilities (commonly provided via optional dependencies such as sharp in Node.js environments). If you encounter errors related to PDF processing dependencies, ensure required native modules are installed in the runtime environment where the application is executed.
The repository follows a typical Next.js application structure:
app/
layout.tsx – Root layout
page.tsx – Main client page (state machine)
globals.css – Tailwind CSS imports
api/
providers/route.ts – GET: detect available LLM providers
process/route.ts – POST: run extraction via ai-form-response-extractor
test-data/route.ts – GET: serve sample test images and JSON
components/
NoProviders.tsx – Setup instructions when no providers found
SetupWizard.tsx – SurveyJS-powered multi-page wizard
Navigation.tsx – Tab bar (Setup | Result | SurveyJS Form)
Processing.tsx – Full-screen loading spinner
ResultView.tsx – Metrics cards + extracted JSON
SurveyFormView.tsx – Pre-filled SurveyJS form preview
ErrorView.tsx – Error display with retry
data/ – Sample datasets (see below)
types/
index.ts – Shared TypeScript types
This demo includes multiple built-in test datasets in the data/ directory. These datasets let you try the extraction pipeline without supplying your own files.
When you select a dataset in the wizard, the corresponding document or image(s) and SurveyJS JSON definition are loaded automatically. While a dataset is selected, the upload and JSON inputs are read-only.
To add your own datasets, create a subfolder in data/ that contains a form.json file with the SurveyJS schema and the document or image(s) to process that correspond to this schema.
| Variable | Description |
|---|---|
OPENAI_API_KEY |
OpenAI API key |
ANTHROPIC_API_KEY |
Anthropic API key |
OLLAMA_BASE_URL |
URL of local Ollama instance (default: http://localhost:11434) |