Skip to content
BranchesTags

Repository files navigation

Genius Indexer

Blockchain indexing and portfolio analytics platform for trading terminals. Indexes on-chain transactions for EVM chains (Polygon by default), tracks user portfolios in real-time, calculates analytics (P&L, volume, fees), and serves data via REST + WebSocket APIs.

Architecture Document: See docs/ARCHITECTURE.md for the full system design, database schemas, data flow diagrams, and trade-off analysis.

Tech Stack

Layer Technology Rationale
Runtime Node.js 20 + TypeScript Rich blockchain tooling ecosystem, excellent async I/O
HTTP Fastify ~75k req/s throughput, native JSON Schema validation, first-class WebSocket
ORM Drizzle SQL-like API with full type inference, zero codegen, TimescaleDB compatible
Database PostgreSQL 16 + TimescaleDB Relational core + time-series hypertables in a single engine
Queue BullMQ Redis-backed job queues with retries, rate limiting, repeatable jobs
Blockchain viem Type-safe ABI decoding, native batch transport, fallback providers
Cache / PubSub Redis 7 Price cache, balance cache, tracked wallet Set, Pub/Sub for real-time
Monorepo Turborepo + pnpm Incremental builds, shared types, strict dependency isolation

Architecture Overview

The system uses a hybrid architecture: a single API server plus three dedicated worker processes, communicating through BullMQ job queues and Redis Pub/Sub.

System Architecture

Open in Excalidraw | Full details in docs/ARCHITECTURE.md

Services:

  • API Server -- Fastify REST endpoints + WebSocket subscriptions. Read-only database access, serves portfolio/analytics/transaction data to the frontend.
  • Indexer Worker -- Polls for new blocks, filters transactions for tracked wallets, decodes ERC20 Transfer events, persists to database, dispatches downstream jobs. Also processes backfill jobs for historical data.
  • Portfolio Worker -- Applies balance deltas, calculates portfolio valuations using cached prices, generates snapshots, publishes real-time updates via Redis Pub/Sub.
  • Analytics Worker -- Computes P&L (FIFO cost basis), trading volume, and gas fee aggregations.

Setup

Prerequisites

  • Node.js >= 20.0.0
  • pnpm >= 9.x
  • Docker (for PostgreSQL + Redis)
  • Alchemy account (free tier works for basic usage)
  • CoinGecko API key (optional, free tier works without one)

If you don't have pnpm installed:

corepack enable && corepack prepare pnpm@latest --activate

1. Install dependencies

pnpm install

2. Configure environment

cp .env.example .env

Edit .env and fill in the required values:

# Required - get a free key at https://www.alchemy.com
ALCHEMY_RPC_URL=https://polygon-mainnet.g.alchemy.com/v2/YOUR_KEY

# Pre-configured defaults (work with Docker setup below)
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/genius_indexer
REDIS_URL=redis://localhost:6379

# Optional - get a free key at https://www.coingecko.com/en/api
COINGECKO_API_KEY=

3. Start infrastructure

docker compose up -d

This starts PostgreSQL (with TimescaleDB) on port 5432 and Redis on port 6379.

Troubleshooting: port 5432 already in use

If you have a local PostgreSQL running, it will conflict with the Docker container. Stop it first:

# Find which PostgreSQL is running
lsof -i :5432

# Stop it (Homebrew example)
brew services stop postgresql@15   # or postgresql@16, postgresql, etc.

4. Run database migrations

pnpm db:migrate

5. Seed initial data

pnpm db:seed

This creates the Polygon chain configuration and seeds well-known tokens (MATIC, USDC, USDT, WETH, WMATIC, DAI).

6. Set the indexer start block

The indexer needs to know from which block to start scanning. By default it starts from block 0, which would try to process the entire chain history. Set it to a recent block:

# Get the current Polygon block number
curl -s https://polygon-mainnet.g.alchemy.com/v2/YOUR_KEY \
  -X POST -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'

# Update the chains table (replace BLOCK_NUMBER with a recent block)
PGPASSWORD=postgres psql -h localhost -U postgres -d genius_indexer \
  -c "UPDATE chains SET last_indexed_block = BLOCK_NUMBER WHERE chain_id = 137;"

7. Start all services

pnpm dev

This starts 4 processes concurrently via Turborepo:

Service Description Port
API Server REST + WebSocket http://localhost:3000
Indexer Worker Block scanning + backfill -
Portfolio Worker Balance + valuation -
Analytics Worker P&L, volume, fees -

Environment Variables

Variable Required Default Description
DATABASE_URL Yes postgresql://postgres:postgres@localhost:5432/genius_indexer PostgreSQL connection string
REDIS_URL Yes redis://localhost:6379 Redis connection string
ALCHEMY_RPC_URL Yes - Alchemy RPC endpoint for Polygon
COINGECKO_API_KEY No - CoinGecko API key (free tier works without)
COINGECKO_BASE_URL No https://api.coingecko.com/api/v3 CoinGecko base URL (change for Pro plan)
CHAIN_ID No 137 EVM chain ID (137 = Polygon)
START_BLOCK No 0 Initial block for scanning
API_PORT No 3000 API server port
API_HOST No 0.0.0.0 API server host
ENABLE_BACKFILL No false Enable backfill feature flag
ENABLE_PRICE_FETCH No true Enable price fetching

Alchemy API Key

  1. Create a free account at alchemy.com
  2. Create a new app and select Polygon Mainnet
  3. Copy the API key and set it as ALCHEMY_RPC_URL in .env

Note on free tier limits: Alchemy's free tier limits eth_getLogs to 10 blocks per request. This makes backfill slower but functional. Upgrading to PAYG or Growth plan allows up to 2,000 blocks per request.

CoinGecko API Key (Optional)

  1. Create a free account at coingecko.com/en/api
  2. Generate a Demo API key from the dashboard
  3. Set it as COINGECKO_API_KEY in .env

Without a key, price fetching still works but with lower rate limits (~10-30 calls/min). DefiLlama acts as a free fallback if CoinGecko fails.

API Reference

Base URL: http://localhost:3000

Health Check

curl http://localhost:3000/health
{ "status": "ok", "timestamp": "2026-03-26T15:31:51.299Z" }

Wallets

Track a new wallet

Registers a wallet for indexing. Automatically triggers a backfill job to fetch historical transactions.

POST /api/v1/wallets
Body Field Type Required Description
address string Yes Ethereum address (0x...)
chainId number Yes Chain ID (137 for Polygon)
label string No Human-readable label
startBlock number No Block to start backfill from. If omitted or 0, uses default (82522143 for Polygon) 
# Track a wallet with default backfill start
curl -X POST http://localhost:3000/api/v1/wallets \
  -H "Content-Type: application/json" \
  -d '{
    "address": "0xda7Dc3113f3F2376ce8d132D09D5CDE5e0792eCd",
    "chainId": 137
  }'

# Track a wallet with custom start block (recent blocks only)
curl -X POST http://localhost:3000/api/v1/wallets \
  -H "Content-Type: application/json" \
  -d '{
    "address": "0xda7Dc3113f3F2376ce8d132D09D5CDE5e0792eCd",
    "chainId": 137,
    "startBlock": 84700000
  }'

Response:

{
  "data": {
    "id": "0731a3bd-33bc-4743-8eba-ab1943ec5e18",
    "address": "0xda7dc3113f3f2376ce8d132d09d5cde5e0792ecd",
    "chainId": 137,
    "label": null,
    "isActive": true,
    "firstIndexedBlock": null,
    "lastIndexedBlock": null,
    "createdAt": "2026-03-26T15:35:06.394Z",
    "updatedAt": "2026-03-26T15:35:06.394Z"
  }
}

List tracked wallets

GET /api/v1/wallets

curl http://localhost:3000/api/v1/wallets

Response:

{
  "data": [
    {
      "id": "0731a3bd-33bc-4743-8eba-ab1943ec5e18",
      "address": "0xda7dc3113f3f2376ce8d132d09d5cde5e0792ecd",
      "chainId": 137,
      "label": "My Wallet",
      "isActive": true,
      "firstIndexedBlock": null,
      "lastIndexedBlock": null,
      "createdAt": "2026-03-26T15:35:06.394Z",
      "updatedAt": "2026-03-26T15:35:06.394Z"
    }
  ]
}

Get wallet details

GET /api/v1/wallets/:id

curl http://localhost:3000/api/v1/wallets/0731a3bd-33bc-4743-8eba-ab1943ec5e18

Update wallet

Update label, active status, or trigger a backfill. When reactivating a wallet that was never indexed, a backfill job is automatically triggered.

PATCH /api/v1/wallets/:id
Body Field Type Required Description
label string No Update the wallet label
isActive boolean No Activate or deactivate the wallet
startBlock number No Block to start backfill from (used when reactivating) 
# Update label
curl -X PATCH http://localhost:3000/api/v1/wallets/0731a3bd-33bc-4743-8eba-ab1943ec5e18 \
  -H "Content-Type: application/json" \
  -d '{"label": "My Main Wallet"}'

# Reactivate a wallet (triggers backfill if never indexed)
curl -X PATCH http://localhost:3000/api/v1/wallets/0731a3bd-33bc-4743-8eba-ab1943ec5e18 \
  -H "Content-Type: application/json" \
  -d '{"isActive": true, "startBlock": 84700000}'

Delete wallet (soft delete)

Deactivates the wallet and removes it from the tracked set. The wallet data is preserved in the database.

DELETE /api/v1/wallets/:id

curl -X DELETE http://localhost:3000/api/v1/wallets/0731a3bd-33bc-4743-8eba-ab1943ec5e18

Response:

{ "data": { "id": "0731a3bd-33bc-4743-8eba-ab1943ec5e18", "deleted": true } }

Portfolio

Get current portfolio

Returns the current portfolio with token breakdown and USD valuations.

GET /api/v1/portfolios/:walletId

curl http://localhost:3000/api/v1/portfolios/0731a3bd-33bc-4743-8eba-ab1943ec5e18

Response:

{
  "data": {
    "walletId": "0731a3bd-33bc-4743-8eba-ab1943ec5e18",
    "totalValueUsd": 1250.50,
    "tokenCount": 3,
    "tokens": [
      {
        "symbol": "USDC",
        "balance": "500000000",
        "balanceFormatted": "500.00",
        "priceUsd": 1.0,
        "valueUsd": 500.0
      }
    ],
    "updatedAt": "2026-03-26T15:31:56.033Z"
  }
}

Get portfolio history

Returns historical portfolio value snapshots.

GET /api/v1/portfolios/:walletId/history

curl http://localhost:3000/api/v1/portfolios/0731a3bd-33bc-4743-8eba-ab1943ec5e18/history

Response:

{
  "data": {
    "walletId": "0731a3bd-33bc-4743-8eba-ab1943ec5e18",
    "snapshots": [
      {
        "time": "2026-03-26T15:30:00.168Z",
        "totalValueUsd": "1250.50",
        "tokenCount": 3
      }
    ]
  }
}

Analytics

Get P&L summary

Returns realized and unrealized profit/loss per token using FIFO cost basis.

GET /api/v1/analytics/:walletId/pnl

curl http://localhost:3000/api/v1/analytics/0731a3bd-33bc-4743-8eba-ab1943ec5e18/pnl

Response:

{
  "data": {
    "walletId": "0731a3bd-33bc-4743-8eba-ab1943ec5e18",
    "totalRealizedUsd": 150.25,
    "totalUnrealizedUsd": 80.00,
    "totalPnlUsd": 230.25,
    "tokens": [
      {
        "tokenId": "...",
        "symbol": "WETH",
        "realizedPnlUsd": 150.25,
        "unrealizedPnlUsd": 80.00,
        "avgBuyPriceUsd": 2800.00
      }
    ]
  }
}

Get trading volume

Returns trading volume over time, bucketed by time interval.

GET /api/v1/analytics/:walletId/volume

curl http://localhost:3000/api/v1/analytics/0731a3bd-33bc-4743-8eba-ab1943ec5e18/volume

Response:

{
  "data": {
    "walletId": "0731a3bd-33bc-4743-8eba-ab1943ec5e18",
    "totalVolumeUsd": 5000.00,
    "totalTxCount": 42,
    "totalFeesUsd": 12.50,
    "buckets": [
      {
        "time": "2026-03-26T00:00:00.000Z",
        "volumeUsd": 1200.00,
        "txCount": 10,
        "feesUsd": 3.20
      }
    ]
  }
}

Get gas fees

Returns gas fees paid by the wallet.

GET /api/v1/analytics/:walletId/fees

curl http://localhost:3000/api/v1/analytics/0731a3bd-33bc-4743-8eba-ab1943ec5e18/fees

Response:

{
  "data": {
    "walletId": "0731a3bd-33bc-4743-8eba-ab1943ec5e18",
    "totalFeesUsd": 12.50,
    "fees": [
      {
        "time": "2026-03-26T00:00:00.000Z",
        "feesUsd": 3.20,
        "feesNative": "0.008"
      }
    ]
  }
}

Transactions

Get transaction history

Returns paginated transaction history with cursor-based pagination.

GET /api/v1/transactions/:walletId

curl http://localhost:3000/api/v1/transactions/0731a3bd-33bc-4743-8eba-ab1943ec5e18

Response:

{
  "data": [
    {
      "txHash": "0xabc...",
      "blockNumber": 84709500,
      "blockTimestamp": "2026-03-26T15:00:00.000Z",
      "fromAddress": "0xda7dc3...",
      "toAddress": "0x3c499c...",
      "value": "0",
      "gasUsed": 52000,
      "status": 1
    }
  ],
  "meta": {
    "hasMore": true,
    "cursor": "eyJ..."
  }
}

To get the next page, pass the cursor:

curl "http://localhost:3000/api/v1/transactions/0731a3bd-33bc-4743-8eba-ab1943ec5e18?cursor=eyJ..."

Prices

Get current prices

Returns current USD prices for all tracked tokens.

GET /api/v1/prices/current

curl http://localhost:3000/api/v1/prices/current

Response:

{
  "data": {
    "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359": {
      "symbol": "USDC",
      "priceUsd": 1.0,
      "change24h": 0.01
    },
    "0x7ceB23fD6bC0adD59E62ac25578270cFf1b9f619": {
      "symbol": "WETH",
      "priceUsd": 3200.50,
      "change24h": -2.5
    }
  }
}

Get price history

Returns historical price data for a specific token.

GET /api/v1/prices/:tokenAddress/history

curl http://localhost:3000/api/v1/prices/0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359/history

WebSocket

Connect to ws://localhost:3000/ws for real-time updates.

Subscribe to portfolio updates

{ "type": "subscribe", "channel": "portfolio", "params": { "walletId": "0731a3bd-..." } }

Server pushes:

{
  "type": "portfolio:update",
  "data": {
    "walletId": "0731a3bd-...",
    "totalValueUsd": 1250.50,
    "tokenCount": 3,
    "updatedAt": "2026-03-26T15:35:00.000Z"
  }
}

Subscribe to price updates

{ "type": "subscribe", "channel": "prices" }

Server pushes:

{
  "type": "price:update",
  "data": {
    "prices": {
      "0x3c499c...": { "priceUsd": 1.0, "change24h": 0.01 }
    }
  }
}

Subscribe to new transactions

{ "type": "subscribe", "channel": "transactions", "params": { "walletId": "0731a3bd-..." } }

Server pushes:

{
  "type": "transaction:new",
  "data": {
    "walletId": "0731a3bd-...",
    "txHash": "0xabc...",
    "chainId": 137,
    "blockNumber": 84709500,
    "fromAddress": "0xda7dc3...",
    "toAddress": "0x3c499c...",
    "value": "1000000",
    "timestamp": "2026-03-26T15:35:00.000Z"
  }
}

Unsubscribe

{ "type": "unsubscribe", "channel": "portfolio", "params": { "walletId": "0731a3bd-..." } }

Ping/Pong

{ "type": "ping" }

Response:

{ "type": "pong" }

Backfill

When you track a new wallet, the system automatically starts a backfill process to fetch historical transactions. The backfill:

  1. Uses eth_getLogs to find all ERC20 Transfer events involving the wallet
  2. Processes logs in chunks (10 blocks per request on Alchemy free tier)
  3. Enqueues relevant blocks for full transaction processing
  4. Runs asynchronously via BullMQ so the API responds immediately

The default backfill start block is configured per chain in packages/config/src/constants.ts:

export const BACKFILL_START_BLOCK: Record<number, number> = {
  137: 82522143, // Polygon - ~1 month of history
};

You can override this per wallet by passing startBlock in the POST or PATCH request.

Project Structure

genius-indexer/
├── apps/
│   ├── api/                    # Fastify REST + WebSocket server
│   │   └── src/
│   │       ├── routes/         # Wallet, portfolio, analytics, price, transaction routes
│   │       ├── ws/             # WebSocket handlers + subscription manager
│   │       └── plugins/        # Auth, error handling
│   │
│   ├── indexer/                # Block scanning + transaction indexing
│   │   └── src/
│   │       ├── services/       # Block scanner, TX parser, ERC20 decoder, receipt processor
│   │       ├── workers/        # Block processing + backfill workers
│   │       └── strategies/     # Chain-specific logic (Polygon, Ethereum)
│   │
│   ├── portfolio-worker/       # Balance + portfolio calculation
│   │   └── src/
│   │       ├── services/       # Balance calculator, portfolio valuator, token resolver
│   │       └── workers/        # Portfolio update, balance sync, snapshot workers
│   │
│   └── analytics-worker/       # P&L, volume, fee analytics
│       └── src/
│           ├── services/       # PnL calculator (FIFO), volume aggregator, fee tracker
│           └── workers/        # PnL, volume, fee workers
│
├── packages/
│   ├── database/               # Drizzle schema, migrations, seed, client
│   ├── types/                  # Shared TypeScript types (domain, API, events, jobs)
│   ├── queue/                  # BullMQ queue definitions, job creators, schedulers
│   ├── cache/                  # Redis cache layer (prices, balances, tracked wallets)
│   ├── config/                 # Environment validation, logger, constants
│   └── prices/                 # Price provider abstraction (CoinGecko, DefiLlama, fallback)
│
├── docker-compose.yml          # PostgreSQL (TimescaleDB) + Redis
├── turbo.json                  # Turborepo build pipeline
└── pnpm-workspace.yaml         # Workspace definitions

Design Decisions

Why Fastify over Express

Fastify delivers ~5x higher throughput than Express with built-in JSON Schema validation, first-class TypeScript generics for route definitions, and native WebSocket support via @fastify/websocket. The plugin system provides encapsulated, lifecycle-aware composition instead of a flat middleware chain.

Why Drizzle over Prisma

Drizzle provides a SQL-like API with full type inference from schema definitions -- no codegen step required. Critically, it allows raw SQL escape hatches needed for TimescaleDB hypertables and continuous aggregates, which Prisma cannot generate. The ~50KB bundle (vs Prisma's ~8MB engine binary) is a practical bonus.

Why viem over ethers.js

viem provides ABI-level type inference (our ERC20 Transfer decoder is compile-time checked), native batch transport that automatically combines RPC calls into single HTTP requests, and a built-in fallback() transport for automatic provider rotation. These features directly address the RPC cost optimization requirement.

Why PostgreSQL + TimescaleDB

TimescaleDB extends PostgreSQL with hypertables for time-series data (prices, portfolio snapshots, analytics) while keeping full SQL and JOIN capabilities with relational tables. This avoids the operational overhead of a separate analytics database (like ClickHouse) while providing automatic compression, partitioning, and continuous aggregates. At our scale (thousands of wallets, millions of transactions), it is more than sufficient.

Why BullMQ over Kafka

Our throughput is approximately one block every two seconds with roughly 100 relevant transactions -- Kafka is designed for 100,000+ messages per second. BullMQ piggybacks on Redis (which we already need for caching and Pub/Sub), provides built-in repeatable jobs (price fetching every 30s), rate limiting for RPC calls, exponential backoff retries, and priority queues. No additional infrastructure to deploy.

Why Hybrid Architecture (API + Workers)

A monolith would couple the indexer's CPU-bound block processing with the API's I/O-bound request handling. Full microservices would add unnecessary operational complexity for this scale. The hybrid approach gives us independent scaling (add more portfolio workers when wallet count grows) while sharing types, database schemas, and configuration through the monorepo. Each service is a separate process but shares the same codebase.

Build vs Buy (Price Data)

On-chain DEX price fetching is expensive (many RPC calls per token), limited to tokens with DEX liquidity, and complex to implement (routing, slippage). CoinGecko provides 14,000+ tokens via a simple REST API with a free tier. DefiLlama serves as a zero-cost fallback. We use a chain-of-responsibility pattern: CoinGecko first, DefiLlama on failure, with aggressive Redis caching (30s hot, 5min warm).

Consistency vs Performance

Balance updates use delta-based arithmetic with optimistic concurrency (WHERE last_updated_block < $current_block) for idempotent, order-safe writes without explicit locking. P&L calculations use PostgreSQL advisory locks per wallet-token pair to serialize FIFO lot processing. A periodic reconciliation worker (every 5 minutes) compares computed balances against on-chain state as a safety net for edge cases like chain reorgs.

RPC Cost Optimization

Six strategies minimize RPC costs: (1) a Redis Set of tracked wallets skips ~99.99% of irrelevant transactions, (2) eth_getLogs with topic filters for backfill instead of fetching every block, (3) viem's batch transport combines multiple calls into single HTTP requests, (4) batch receipt fetching (50 per request), (5) adaptive polling that backs off during quiet periods, (6) aggressive caching for token metadata, prices, and block data.

Full Architecture Document

For the comprehensive architecture design including database ER diagrams, data flow pipelines, TimescaleDB configuration, concurrency strategies, WebSocket protocol specification, and scalability analysis, see:

docs/ARCHITECTURE.md

Questions & Assumptions

See docs/QUESTIONS.md

About

Blockchain indexing and portfolio analytics platform for trading terminals

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages