News

Collect the web. Extract the story. Sound the alarm.

An always-on news collection, extraction, publication, and physical alerting service for a single trusted operator on a private network.

---

Overview

The system ingests article candidates from Google News RSS feeds, a secondary search service, and Telegram items stored in Redis. It validates each destination against a public-URL policy, scrapes accepted pages with a shared Camoufox browser context, cleans the text through a local ScraperService, and asks language models to extract structured stories. Durable records land in Prism (SQL); rebuildable publication views and operational state live in Redis. When the model accepts a new alert, the service fans out synchronously to Discord, a network siren, and a Kindle display—so the operator gets durable history plus immediate multi-channel notice.

Runtime is a long-lived Bun process scheduled with BullMQ against Redis. Extraction uses LangChain agents (OpenAI for the primary pipeline, Google for secondary structured extraction). Browser work is phase-scoped and concurrency-bounded; gRPC clients enforce deadlines; alert channels checkpoint independently so retries do not re-fire channels that already succeeded.

Features

Area	What the project provides
Primary discovery	Loads configurable Google News RSS feeds, resolves article links, applies the public URL policy, and scrapes with bounded page concurrency inside one run-scoped Camoufox context.
Secondary discovery	Queries SearchService with configured terms, scrapes unvisited public URLs, runs schema-validated one-result-per-article extraction, and stages qualifying items in a durable Redis batch list for the next primary run.
Structured extraction	Primary and secondary agents produce tool-validated story payloads; sources must map to the current run; unknown source identifiers fail closed.
Durable storage	Prism/SQL is the story of record. Saves send a deterministic idempotency_key. Redis holds publication views, staging, visited URLs, and delivery checkpoints—not the sole source of truth.
Unconditional alerts	Every newly accepted alert fans out to Discord, siren, and Kindle with no category, geography, genre, severity, source-count, or trusted-domain gate. One channel failing does not stop the others from being attempted.
Retry-safe delivery	Per-channel checkpoints under alert_delivery:<id> (seven-day TTL). Identical retries skip completed channels and re-attempt only incomplete ones.
Scraping safety	Credential-free HTTP(S) only; private, loopback, link-local, multicast, reserved, and metadata destinations rejected at resolve and redirect time; content length capped before downstream use.
Scheduling	Primary, secondary, nightly cleanup, and minute heartbeats on separate queues so a long scrape cannot starve liveness pings. Timezone defaults to Asia/Kolkata.
Lifecycle discipline	A URL becomes visited only after durable save or durable staging. Nightly cleanup clears published Redis views while preserving staging, queues, locks, visited URLs, and idempotency markers.
Operator monitoring	Uptime Kuma push URLs for process heartbeat (every minute) and successful pipeline jobs (separate signals).

Note

Completed: primary and secondary pipelines, public URL policy, Camoufox shared-context scraping, Prism ingest + Redis publication views, three-channel alert fan-out with checkpoints, BullMQ schedulers/workers, nightly cleanup, launchd service definition, and focused unit tests.

Partial / gated: automatic BullMQ retries for full primary/secondary jobs remain disabled until the independently deployed Prism server persists IngestStoryRequest.idempotency_key and enforces server-side upsert semantics. Redis publication is already idempotent; client-side markers cannot prove whether a timed-out remote write committed.

Operator-bound: defaults include private-network addresses (Kindle, siren, gRPC services) intended for a single trusted host—not multi-tenant or public SaaS deployment.

From input to result

flowchart LR
  RSS[RSS feeds] --> POLICY[Public URL policy]
  SEARCH[SearchService] --> SEC[Secondary pipeline]
  TG[Telegram in Redis] --> PRI[Primary pipeline]
  SEC --> STAGE[(secondary_news_batches)]
  STAGE --> PRI
  POLICY --> SCRAPE[Camoufox scrape]
  SCRAPE --> CLEAN[ScraperService]
  CLEAN --> PRI
  PRI --> LLM[Primary agent]
  LLM --> PRISM[(Prism / SQL)]
  LLM --> ALERT[Alert fan-out]
  PRISM --> REDIS[(Redis views)]
  ALERT --> OUT[Discord · Siren · Kindle]

Loading

Empty primary runs stop before invoking the model. Secondary items are acknowledged only after a successful primary run; Telegram claims restore to telegram_retry if the run fails mid-flight. Visited URLs and publication markers update only after durable success. Read-only gRPC calls get one bounded retry; writes do not retry blindly. Alert delivery starts all incomplete channels independently, waits for all three to settle, and returns an aggregate failure if any channel fails.

Product information architecture

flowchart TD
  STORY[Structured news story]
  STORY --> DISC[Discovery inputs]
  STORY --> STORE[Persistence]
  STORY --> PUB[Publication views]
  STORY --> ALERTS[Alert outputs]
  STORY --> OPS[Operational state]

  DISC --> D1[RSS candidates]
  DISC --> D2[Search terms]
  DISC --> D3[Telegram items]

  STORE --> S1[Prism durable history]
  STORE --> S2[Idempotency keys]

  PUB --> P1[news_collection_v2]
  PUB --> P2[Per-genre lists]
  PUB --> P3[alerts_send]

  ALERTS --> A1[Discord webhook]
  ALERTS --> A2[Network siren]
  ALERTS --> A3[Kindle display]

  OPS --> O1[visited_url]
  OPS --> O2[secondary_news_batches]
  OPS --> O3[alert_delivery checkpoints]
  OPS --> O4[scrape failure / auto-blacklist]

Loading

Published Redis collections are rebuildable and cleared nightly. Staging lists, visited sets, blacklists, locks, and seven-day checkpoints survive cleanup so recovery and deduplication keep working across days.

Architecture

flowchart TB
  subgraph App["Application · Bun process"]
    Q[newsQueueRoute<br/>BullMQ schedulers + workers]
    PRI[pipeline.ts]
    SEC[secondaryPipeline.ts]
    CLN[cleanupPipeline.ts]
    AGENTS[Primary / secondary agents]
    TOOLS[save-news · alert tools]
    POLICY[urlPolicy]
    SCRAPE[Shared Camoufox pool]
  end

  subgraph Services["Local gRPC services"]
    DISCORD[DiscordWebhook :50051]
    PRISM[PrismService :50052]
    SEARCH[SearchService :50053]
    SCRAPER[ScraperService :50057]
  end

  subgraph Data["Data and devices"]
    REDIS[(Redis)]
    SQL[(Prism SQL)]
    KINDLE[Kindle SSH]
    SIREN[Siren HTTP]
    KUMA[Uptime Kuma]
  end

  Q --> PRI
  Q --> SEC
  Q --> CLN
  Q --> KUMA
  PRI --> POLICY --> SCRAPE --> SCRAPER
  SEC --> SEARCH
  SEC --> POLICY
  PRI --> AGENTS --> TOOLS
  SEC --> AGENTS
  TOOLS --> PRISM
  TOOLS --> REDIS
  TOOLS --> DISCORD
  TOOLS --> SIREN
  TOOLS --> KINDLE
  PRI --> REDIS
  SEC --> REDIS
  PRISM --> SQL

Loading

Conventions that exist in this codebase:

• Config: src/config.ts freezes environment-derived settings and validates required keys at startup (validateStartupConfig for the queue process).
• Queues: Pipeline work (news-pipeline) and heartbeat work (news-heartbeat) use separate BullMQ workers. Jobs run as isolated Bun scripts via processRunner with a configurable timeout.
• Persistence boundary: Prism writes happen before atomic Redis publication. URLs are marked visited only after durable save or durable secondary staging.
• gRPC: Clients share deadline handling; addresses default to loopback ports. Read-only calls may retry once within budget.
• Model boundary: Article and Telegram content is labeled untrusted in prompts; tool inputs are Zod-validated; source indices must resolve to the current run map.
• Kindle delivery: Fixed executable invocation, POSIX-safe quoting, SSH deadlines, pinned host-key fingerprints, and a token-owned expiring Redis display lock with owner-checked release.
• Shutdown: SIGINT/SIGTERM stop workers, terminate active job processes, and close gRPC clients and Redis cleanly.
• Security posture: Designed for a private network and trusted operator. Autonomous model-triggered physical and network side effects remain intentional; prompt-injection risk cannot be eliminated while that design is retained.

Tech stack

Layer	Technology
Language / runtime	TypeScript, Bun
Scheduling	BullMQ on Redis
LLM orchestration	LangChain (@langchain/openai, @langchain/google, langchain)
Schema validation	Zod
Browser scraping	camoufox-js + playwright-core
RSS	rss-parser
RPC	@grpc/grpc-js + local .proto definitions
Persistence clients	Bun Redis client; Prism gRPC for durable SQL ingest/search
Device integration	ssh2 (Kindle), HTTP (siren)
URL / domain helpers	tldts
Testing / checks	bun test, tsc --noEmit, bun audit
Host packaging	macOS launchd (com.news.queue.plist)

Project structure

.
├── package.json                 # Scripts: queue, browser:install, test, typecheck, audit
├── com.news.queue.plist         # launchd unit (machine-specific paths)
├── scripts/
│   └── installCamoufox.ts       # Install/update Camoufox browser binary
├── src/
│   ├── config.ts                # Env-derived frozen config + startup validation
│   ├── newsQueueRoute.ts        # BullMQ schedulers, workers, heartbeats, shutdown
│   ├── ai/
│   │   ├── agents/              # Primary and secondary model orchestration
│   │   ├── tools/               # Run-scoped save-news and alert tools
│   │   ├── alertDelivery.ts     # All-channel fan-out + checkpoint interface
│   │   └── model.ts             # OpenAI primary + Google secondary models
│   ├── data/
│   │   ├── pipeline.ts          # Primary pipeline entry
│   │   ├── secondaryPipeline.ts # Secondary pipeline entry
│   │   ├── cleanupPipeline.ts   # Nightly published-view cleanup
│   │   ├── rss/                 # Feed discovery
│   │   ├── scrape/              # Camoufox pool, URL extraction, smoke test
│   │   ├── newsCollection/      # Redis publication views
│   │   ├── secondaryNews/       # Durable secondary staging + migration
│   │   ├── telegram/            # Recoverable Telegram claims
│   │   └── cleanScrapedContent.ts
│   ├── db/redis/                # Shared Bun Redis client
│   ├── grpc/                    # Deadline-aware clients + protos
│   ├── integrations/
│   │   ├── kindle/              # SSH display + lock
│   │   └── siren/               # HTTP siren trigger
│   ├── queue/                   # Isolated process runner, log truncation
│   ├── security/urlPolicy.ts    # SSRF / public-address policy
│   └── utils/                   # visited + blacklist helpers
└── LICENSE                      # AGPL-3.0-or-later

Requirements

• Runtime: Bun (project scripts assume Bun, not Node)
• Language tooling: TypeScript (via bunx tsc for typecheck)
• Redis: reachable at REDIS_URL (default redis://127.0.0.1:6379)
• Camoufox browser binary: installed via bun run browser:install (not committed)
• Local gRPC services: DiscordWebhook, Prism, Search, Scraper (default loopback ports 50051, 50052, 50053, 50057)
• API credentials: OPENAI_API_KEY, GOOGLE_API_KEY
• Monitoring: Uptime Kuma push URLs for process and job monitors
• Alert path (private network): Kindle SSH host + private key + host-key fingerprints; siren HTTP endpoint
• Network: outbound HTTPS for RSS/model providers; access to the private services above
• OS for production-style hosting: macOS with launchd when using com.news.queue.plist

There is no multi-tenant cloud mode. Pipeline scripts can be run directly for development; the queue process is the always-on operator path. Physical alert devices must be on the configured private network—they are not simulated in unit tests.

Getting started

1. Clone and enter the repository

git clone <repository-url>
cd news

2. Install dependencies

bun install

3. Configure environment

cp .env.example .env

Fill required values. Bun loads .env automatically.

Variable	Purpose
OPENAI_API_KEY	Primary model / tool-calling provider
GOOGLE_API_KEY	Secondary structured extraction provider
UPTIME_KUMA_PUSH_URL	Minute process-heartbeat push URL
UPTIME_KUMA_NEWS_JOB_PUSH_URL	Successful pipeline-job push URL
REDIS_URL	Redis connection (default redis://127.0.0.1:6379)
KINDLE_KEY_PATH	Absolute path to the Kindle SSH private key
KINDLE_HOST_FINGERPRINTS	Comma-separated SHA256:... fingerprints
DISCORD_GRPC_ADDR / PRISM_GRPC_ADDR / SEARCH_GRPC_ADDR / SCRAPER_GRPC_ADDR	gRPC host:port endpoints
SIREN_URL / ALERT_ENDPOINT_URL	Siren / alert HTTP endpoints

Useful optional knobs (defaults in src/config.ts): NEWS_TIMEZONE (Asia/Kolkata), SCRAPE_CONCURRENCY (4), SCRAPE_MAX_CONTENT_LENGTH (4000), RSS_SOURCES, SECONDARY_SEARCH_TERMS, queue retention and gRPC deadline settings.

4. Install and verify the browser

bun run browser:install
bun run browser:smoke

Set CAMOUFOX_INSTALL_DIR to a service-specific cache if needed. Do not share the cache path with Python Camoufox installs (incompatible layouts). Do not commit the browser binary.

5. Ensure Redis and gRPC services are up, then start the scheduler:

bun run queue

The queue process validates required configuration, registers/upserts schedulers, and enqueues one immediate primary job.

6. Optional: run pipelines directly during development

bun src/data/pipeline.ts
bun src/data/secondaryPipeline.ts

7. Optional: install as a launchd service (macOS)

Inspect machine-specific paths in com.news.queue.plist first, then:

cp com.news.queue.plist ~/Library/LaunchAgents/com.news.queue.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.news.queue.plist

Default launchd logs: /tmp/com.news.queue.out.log, /tmp/com.news.queue.err.log. The heartbeat worker truncates both in place at midnight in NEWS_TIMEZONE (launchd keeps file descriptors open for the service lifetime).

Important

.env.example and src/config.ts ship with private-network defaults (Kindle host, siren URLs, gRPC loopback ports, sample Kindle host-key fingerprints). Replace keys, fingerprints, hosts, and push URLs for your environment before any wider distribution. Keep API keys and the Kindle private key out of git, logs, and model prompts. SSH fingerprints are verification material and may be committed after a legitimate host-key change—do not disable host-key verification.

Running tests

Package manager / CLI:

bun test
bun run typecheck
bun run audit

Before committing, optionally verify patch cleanliness:

git diff --check

What the suite covers (focused unit tests):

• Public/private URL and DNS policy decisions (src/security/urlPolicy.test.ts)
• Secondary extraction ordering and sentinel filtering
• Cleanup key selection for published Redis views
• All-channel alert fan-out and partial retry/checkpoint behavior
• Kindle command-injection resistance and argument quoting
• Scraped-content cleaning
• Process runner and log-maintenance helpers

There is no separate IDE test target; use bun test from the repository root. Integration tests against live Prism, Discord, Kindle, or siren are not part of the default suite.

Roadmap

• Enable automatic BullMQ retries for full primary/secondary jobs once Prism server-side idempotent ingest (persist and enforce idempotency_key) is confirmed in the deployed service.
• Keep secondary-model provider wiring under review as LangChain/Google client APIs evolve (src/ai/model.ts).
• Expand coverage toward end-to-end pipeline runs against stubbed gRPC services where unit tests currently stop at boundaries.
• Revisit operator packaging for non-macOS hosts if the service is run outside launchd.

License

This project is free software under the GNU Affero General Public License, version 3 or any later version (AGPL-3.0-or-later). See LICENSE for the full text.

In plain terms: if you modify this program and let users interact with the modified version over a network, AGPL section 13 requires you to offer those users the Corresponding Source. Preserve copyright and license notices when redistributing. This README is an operational summary, not legal advice.

---

Bun, BullMQ, Camoufox, and local gRPC—news in, structured stories out, siren optional but loud.