Skip to content
View samwu4166's full-sized avatar
πŸ‡ΉπŸ‡Ό
Focusing on how to become a good data engineer
πŸ‡ΉπŸ‡Ό
Focusing on how to become a good data engineer
  • TAIPEI
  • 02:21 (UTC +08:00)

Block or report samwu4166

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
samwu4166/README.md

Hi πŸ‘‹, I'm Sam

Senior Data Engineer Β· building agent-native data systems


πŸ‡ΉπŸ‡Ό Based in Taipei, building production data platforms and increasingly agent-native engineering systems. 5+ years working with data pipelines β€” 4 of them full-time platform ownership β€” across ad-tech, fintech, and consumer-product data stacks.

πŸ“¦ Open source β€” claude-swarm-kit

Run several Claude Code agents on one machine, each reachable from its own Telegram chat, without them fighting over the same channel.

The problem it solves. Telegram's Bot API allows exactly one long-poll consumer per token β€” a second one just gets HTTP 409. So the obvious ways to run N agents are N forked binaries or a shared router service, and both add a component you then have to operate.

The trick. Run N stateless copies of the same MCP plugin, each pointed at its own TELEGRAM_STATE_DIR β€” its own token file, bot.pid lock, inbox, and allowlist. N agents β†’ N bots β†’ zero 409 conflicts, no extra services, no synchronization.

Around that core it layers what a fleet needs to be trustworthy rather than merely running: a tamper-evident hash-chain audit log, layered markdown memory with write-time linting and generated indexes, a deterministic fleet-invariant gate injected at session start, file-based worker spawning with budget/loop governors and kernel-level (Seatbelt) sandboxing, and self-contained HTML report publishing. Modules are optional β€” the isolation core stands alone.

It's the extracted, generalized form of the agent fleet I actually run every day.

πŸ”­ Currently building

  • πŸ€– A 5-agent Claude Code fleet, in daily use β€” five persistent agents on one machine (life/ops, market research, project delivery, opportunity scanning, career), each with its own isolated Telegram channel, coordinating stigmergically through shared files rather than direct messaging, under a deterministic invariant gate
  • πŸ›° An ops mission-control deck β€” a Next.js console over that fleet: agent health, log tails, published reports, memory search/graph, and the invariant gate's live verdict, with a strict read-only boundary between the web layer and anything that executes

πŸ— Recently shipped

  • πŸ“¦ Agent-native dbt PR review harness β€” a control plane around the model, not a wrapper around it: a multi-phase impact pipeline with deterministic cost/blast-radius gates, and a separate reviewer model from the writer model to blunt reward hacking. Built solo in ~4 weeks (AI-assisted); took AI review from unreliable to production-trusted, with misleading review reports driven to zero in the pre-rollout gate across 50+ real PR replays
  • πŸ›° Real-time ad-metrics pipeline (BigQuery β†’ Spark β†’ ClickHouse β†’ FastAPI) β€” query latency ~10 s β†’ ~100 ms (~100Γ—) at 4Γ— throughput, hardened with a cache-layer circuit breaker and a pre-flight guard that makes partition replacement fail fast instead of silently wiping data
  • 🀝 Two autonomous daily ops agents on an enterprise MCP platform β€” a warehouse cost/health monitor and an ETL-health narrator, both running unattended and deduping findings against the issue tracker so the daily brief surfaces only new problems

πŸ›  Stack

Data / Backend β€” Apache Airflow Β· dbt Β· ClickHouse Β· BigQuery Β· Spark Β· PostgreSQL Β· Python Β· FastAPI Infra β€” Kubernetes (GKE) Β· Pulumi (TypeScript IaC) Β· Docker Β· GitHub Actions Agentic AI β€” Claude Code Β· MCP Β· multi-agent orchestration patterns

✍️ Writing & speaking

Spoken at dbt and Airflow community meetups (150+ attendees), on dbt at scale and agent-native data engineering. Open to conference / meetup invites.

πŸ“« Connect


stats langs

Pinned Loading

  1. pagination-prediction pagination-prediction Public

    A Repo that contains ML-based Pagination Prediction and served with Fastapi

    Jupyter Notebook 1 1

  2. claude-swarm-kit claude-swarm-kit Public

    Run a fleet of Claude Code agents on one machine, each with an isolated Telegram channel + shared audit/memory/spawn/report infra

    Shell 2

  3. pc-build-audit-tw pc-build-audit-tw Public

    Claude Code Skill β€” audit Taiwan PC build menus for compat bugs, 2026 pricing, use-case fit (繁中)