🔱 Neptune AIS
Download, normalize, fuse, and analyze vessel tracking data from multiple open-source AIS archives.
One interface. Many sources. Clean output.
Installation • Quick Start • Data Sources • Features • CLI • Docs
Cinematic long-exposure AIS timelapses (D3 + Canvas 2D) — click either preview for the 60 fps MP4
Strait-crossing timelapse — vessels colored by type, dashed rings mark gate crossings by direction.
Generate your own for any strait: python scripts/generate_crossings_gif.py --bbox W,S,E,N --gate lon1,lat1,lon2,lat2 --date 2024-06-15 --days 2
Neptune is a Python library that gives you a single, unified interface to download, normalize, and analyze AIS (Automatic Identification System) vessel tracking data from multiple open-source archives.
AIS data powers maritime domain awareness — vessel tracking, trade analytics, environmental monitoring, fishing surveillance, and port operations. But working with it is painful: every provider uses a different format, schema, delivery mechanism, and quality profile. Neptune handles all of that so you can focus on analysis.
from neptune_ais import Neptune
n = Neptune("2024-06-15", sources=["noaa"])
n.download()
positions = n.positions() # Polars LazyFrame — normalized, QC'd
result = n.sql("SELECT mmsi, count(*) as n FROM positions GROUP BY mmsi ORDER BY n DESC LIMIT 5")Think of Neptune as Herbie for maritime data — a clean data-access layer that handles the messy plumbing of fetching, normalizing, and cataloging data from heterogeneous archives, so you get reproducible, analysis-ready output every time.
- Multi-source ingestion — Download from NOAA, DMA, AISHub through one API, ingest GFW fishing events and effort grids, and stream from AISStream and Finland (Digitraffic)
- Automatic normalization — Every source is normalized to a canonical schema with QC scoring and provenance tracking
- Multi-source fusion — Merge overlapping sources with configurable dedup strategies (
best,union,prefer:<source>) - Polars-native — Query positions, vessels, tracks, and events as lazy DataFrames with full predicate pushdown
- SQL via DuckDB — Run SQL queries directly over your cataloged data
- Event detection — Derive port calls, EEZ crossings, vessel encounters, and loitering from raw positions
- Real-time streaming — Connect to live AIS feeds with backpressure, checkpointing, and durable sinks
- Interactive maps — Visualize positions, tracks, and events with lonboard
- Plugin system — Add custom source adapters via Python entry points
- Animated vessel replay — Generate standalone HTML animations with deck.gl TripsLayer
- Timelapse corridor visualization — Cinematic long-exposure timelapses (D3 + Canvas 2D) that render shipping corridors as accumulating neon trails, plus the classic dot-accumulation renderer — colored by vessel type or direction
- World Port Index — Built-in dictionary of 3,800+ ports with search, nearby lookup, UNLOCODE resolution, AIS destination matching, and auto-derived port polygons from vessel behavior
- CLI included —
neptune download,neptune inventory,neptune sql,neptune ports, and more
Animated vessel replay with glowing trails, time controls, and dark basemap
Neptune's core is lightweight — only Polars, Pydantic, httpx, and Click are required. The neptune CLI works out of the box. Everything else is opt-in.
# Core (Polars + Pydantic + httpx + Click)
pip install neptune-ais
# With SQL support (DuckDB)
pip install neptune-ais[sql]
# With spatial & visualization (GeoDataFrames, lonboard, H3)
pip install neptune-ais[geo]
# With real-time streaming (WebSocket feeds)
pip install neptune-ais[stream]
# With richer CLI output (Rich)
pip install neptune-ais[cli]
# Everything
pip install neptune-ais[all]Requirements: Python 3.10+
Optional dependency groups explained
| Extra | Adds | Used by |
|---|---|---|
sql |
duckdb | Neptune.sql(), Neptune.duckdb(), DuckDBSink |
parquet |
pyarrow | Full Parquet write options (compression, statistics) |
geo |
shapely, geopandas, movingpandas, lonboard, h3, folium, mapclassify, matplotlib | Boundary lookups, GeoDataFrame bridges, maps |
gfw |
gfw-api-python-client | GFW source adapter (events, vessels, fishing effort) |
stream |
websockets, aiomqtt | NeptuneStream, live AIS feeds |
cli |
rich | Richer terminal formatting (click is in core) |
notebooks |
jupyter, ipykernel | Interactive notebook examples |
dev |
pytest, mypy, ruff, coverage, nbstripout | Development and testing |
all |
All of the above (except dev) | Full-featured install |
from neptune_ais import Neptune
# Download a day of NOAA AIS data
# Neptune handles: fetch → normalize → QC → partition → catalog
n = Neptune("2024-06-15", sources=["noaa"])
n.download()
# Query as a Polars LazyFrame
positions = n.positions()
df = positions.collect()
print(f"{len(df):,} position reports from {df['mmsi'].n_unique():,} vessels")
# SQL queries via DuckDB
top_vessels = n.sql("""
SELECT mmsi, count(*) as n
FROM positions
GROUP BY mmsi
ORDER BY n DESC
LIMIT 10
""")from neptune_ais.helpers import latest_positions, snapshot, vessel_history
# Most recent position per vessel
latest = latest_positions(positions)
# Point-in-time snapshot — where was every vessel at noon?
noon = snapshot(positions, when="2024-06-15T12:00:00")
# Full history for a single vessel
history = vessel_history(367000001, positions=positions)# Combine NOAA and DMA with automatic deduplication
n = Neptune(
("2024-06-15", "2024-06-16"),
sources=["noaa", "dma"],
merge="best", # "best" | "union" | "prefer:noaa"
)
n.download()
fused = n.positions() # Deduplicated across sources# Derive maritime events from position data
events = n.events(kind="port_call", min_confidence=0.7)
# Event types: port_call, eez_crossing, encounter, loitering
# Each event includes confidence scores and full provenanceimport asyncio
from neptune_ais.stream import NeptuneStream, StreamConfig
from neptune_ais.sinks import ParquetSink, promote_landing
config = StreamConfig(
source="aisstream",
api_key="YOUR_KEY",
bbox=(-74.5, 40.0, -73.5, 41.0), # New York harbor
)
async def ingest():
sink = ParquetSink("/tmp/neptune_landing", source="aisstream")
async with NeptuneStream(config=config) as stream:
await stream.run(sink, max_messages=10_000)
# Promote to canonical storage
promote_landing("/tmp/neptune_landing", store_root="~/.neptune", source="aisstream")
asyncio.run(ingest())Neptune includes adapters for six open AIS data providers, with a plugin system for adding more.
| Source | Provider | Coverage | Delivery | Auth | Backfill |
|---|---|---|---|---|---|
noaa |
NOAA AIS Archive | US waters, global ATON | Daily files | None | Yes |
dma |
Danish Maritime Authority | European waters | Daily files | None | Yes |
gfw |
Global Fishing Watch | Global | Events API + 4Wings | API token | Yes (2020+) |
finland |
Digitraffic Finland | Finnish waters | MQTT streaming | None | No (live only) |
aishub |
AISHub | Global (variable quality) | Multiple feeds | API key | Yes |
aisstream |
AISStream | Global (real-time) | WebSocket | API key | No (live only) |
from neptune_ais import sources
sources.load_all_adapters()
# List all sources
for s in sources.catalog():
print(f"{s.source_id:<12} {s.provider:<30} auth={s.auth_scheme or 'none'}")
# Find open-data sources with backfill
for s in sources.find_sources(backfill=True, auth=False):
print(s.source_id)External packages register adapters through Python entry points:
# In your plugin's pyproject.toml
[project.entry-points."neptune_ais.adapters"]
my_source = "my_package.adapter:MyAdapter"Neptune is organized around a canonical dataset family and a three-layer local store:
┌────────────────┐
│ Your Code │
│ Polars / SQL │
└───────┬────────┘
│
┌─────────▼─────────┐
│ Neptune API │
│ .positions() │
│ .tracks() │
│ .events() │
│ .sql() │
└──┬──────────────┬──┘
┌─────────────▼──┐ ┌─────▼──────────────┐
│ Archival Path │ │ Streaming Path │
│ fetch → norm │ │ NeptuneStream │
│ → QC → store │ │ → sink → promote │
└──┬─────────────┘ └─────┬──────────────┘
┌──────────────▼─────────────────────────▼────────────────┐
│ Three-Layer Store │
│ raw/ (source payloads) → canonical/ (normalized) │
│ → derived/ (cached products) │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ Catalog & Manifests │
│ partition tracking · schema versions · QC summaries │
│ staleness detection · atomic writes (stage → commit) │
└─────────────────────────────────────────────────────────┘
| Dataset | Description | Schema |
|---|---|---|
| positions | Timestamped AIS point observations (mmsi, lat, lon, sog, cog, ...) | positions/v1 |
| vessels | Vessel identity and reference data (slowly changing dimensions) | vessels/v1 |
| tracks | Derived trip/trajectory segments | tracks/v1 |
| events | Maritime events (port calls, EEZ crossings, encounters, loitering) | events/v1 |
Every ingested record passes through row-level and partition-level QC checks:
- Data type validation, range checks, sentinel detection
- Confidence scoring in three tiers: HIGH (>= 0.7), MEDIUM (0.3–0.7), LOW (< 0.3)
- Per-adapter QC rule injection for source-specific quirks
- Full provenance tracking from source through fusion
When querying across multiple sources, Neptune supports three merge strategies:
| Mode | Behavior |
|---|---|
best |
Deduplicate with configurable field-level precedence |
union |
Keep all records from all sources, tag provenance |
prefer:<source> |
Deterministic source preference (e.g., prefer:noaa) |
Neptune derives four maritime event families from position data using heuristic detectors:
| Event | Description |
|---|---|
| Port calls | Sustained low-speed presence within a port boundary |
| EEZ crossings | Transitions between exclusive economic zones |
| Encounters | Two vessels within 500m for a sustained duration |
| Loitering | Sustained low-speed movement in a small area |
Each event includes a deterministic event_id, confidence score, timestamps, and full provenance linking back to source positions. See HEURISTICS.md for detection assumptions and known limitations.
Neptune includes a full command-line interface (works with the base install — pip install neptune-ais[cli] adds richer formatting):
# Download data
neptune download --source noaa --date 2024-06-15
neptune download --source noaa --source dma --start 2024-06-01 --end 2024-06-07
# Inspect what you have
neptune inventory
neptune inventory --dataset positions
# Quality reports
neptune qc --source noaa --date 2024-06-15
# SQL queries from the terminal
neptune sql "SELECT count(*) FROM positions WHERE source = 'noaa'"
# Source catalog
neptune sources
neptune sources --compare noaa dma gfw
# Event queries
neptune events --kind port_call --date 2024-06-15
# World Port Index
neptune ports search "Rotterdam"
neptune ports near 51.9 4.5
neptune ports info NLRTM
neptune ports country NL
neptune ports derive --date 2024-06-15
neptune ports export -o ports.geojson
# Health and provenance
neptune health
neptune provenance --date 2024-06-15
# Promote streaming data to canonical store
neptune promote --landing-dir /tmp/neptune_landing --source aisstreamFull Sphinx documentation is planned. In the meantime:
| Resource | Description |
|---|---|
examples/ |
Eleven narrative examples covering the full workflow |
| HEURISTICS.md | Event detection assumptions, confidence limits, non-goals |
| RELEASING.md | Release procedures and checklist |
| RC_CHECKLIST.md | Release-candidate validation results |
| # | Example | Topics |
|---|---|---|
| 1 | Source Discovery (.py) | Inspect sources, capabilities, filters |
| 2 | Archival Ingest (.py) | Download, Polars queries, SQL, helpers |
| 3 | Multi-Source Fusion (.py) | Merge strategies, fusion config |
| 4 | Event Detection (.py) | Port calls, EEZ crossings, encounters |
| 5 | Streaming Pipeline (.py) | Live feeds, sinks, promotion |
| 6 | External Plugin | Custom adapter via entry point |
| 7 | Fishing Intelligence | GFW events, vessel identity, fishing effort grids |
| 8 | Spatial Visualization | Interactive maps, GeoDataFrames, MovingPandas trajectories |
| 9 | Intelligence Dashboard | Gate crossings, live counters, filters, standalone HTML dashboard |
| 10 | Timelapse Visualization | Cinematic corridor timelapse, vessel-type coloring, multi-panel |
| 11 | World Port Index | Port dictionary, zero-config port calls, enrichment, polygons, destination resolver |
Tip: Install notebook support with
pip install neptune-ais[notebooks]to run the interactive examples.
Contributions are welcome. To get started:
git clone https://github.com/yourorg/neptune-ais.git
cd neptune-ais
pip install -e ".[all,dev]"
pytestThe test suite includes 927 tests covering adapter certification, schema reproducibility, streaming soak tests, port intelligence, and packaging validation.