Skip to content

Commit 7e75e87

Browse files
norlandrhagennorlandrhagen
committed
docs: add design page, expand api and usage
Design page covers the plan/execute split, chunk and shard heuristics, the streaming memory model, kernel semantics, and tuning knobs; api gains engine entries; usage documents progress and memory tuning.
1 parent 7a9c88a commit 7e75e87

4 files changed

Lines changed: 108 additions & 2 deletions

File tree

‎docs/api.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,19 @@
11
# API Reference
22

3-
The public API is three objects: `create_pyramid` builds a write plan, `Pyramid` holds it, and `ZarrLayerVarConfig` carries optional visualization hints.
3+
The public API is three objects: `create_pyramid` builds a write plan, `Pyramid` holds it, and `ZarrLayerVarConfig` carries optional visualization hints. `CoarseningMethod` is the `Literal["mean", "max", "min", "sum"]` alias accepted by `create_pyramid(method=...)`.
44

55
::: topozarr.coarsen.create_pyramid
66

77
::: topozarr.pyramid.Pyramid
88

99
::: topozarr.metadata.ZarrLayerVarConfig
10+
11+
## Engine
12+
13+
Lower-level streaming drivers used by `Pyramid.write`; useful when writing custom pipelines on top of the kernel.
14+
15+
::: topozarr.engine.downsample_level
16+
17+
::: topozarr.engine.copy_array
18+
19+
::: topozarr.engine.default_max_workers

‎docs/design.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Design
2+
3+
How topozarr turns a georeferenced dataset into a multiscale Zarr store, and
4+
which knobs control memory and performance.
5+
6+
## Plan / execute split
7+
8+
[`create_pyramid`][topozarr.coarsen.create_pyramid] is pure planning — no data
9+
is read or computed. It produces a [`Pyramid`][topozarr.pyramid.Pyramid]
10+
holding:
11+
12+
- **level templates**: per-level `xr.Dataset`s with real (mean-coarsened)
13+
coordinates but zero-cost placeholder data variables,
14+
- **encoding**: chunk and shard sizes per variable per level,
15+
- **attrs**: root metadata following the zarr-conventions
16+
[multiscales](https://github.com/zarr-conventions/multiscales),
17+
[proj](https://github.com/zarr-conventions/geo-proj), and
18+
[spatial](https://github.com/zarr-conventions/spatial) specs.
19+
20+
`Pyramid.write` executes the plan: level 0 is streamed from the source
21+
dataset, then each level `N` is block-reduced from the already-written level
22+
`N - 1` through the Rust kernel (`topozarr_core.block_reduce`), so the source
23+
is read exactly once regardless of the number of levels.
24+
25+
## Chunk and shard heuristics
26+
27+
Spatial dimensions aim for square chunks of `target_chunk_bytes` (default
28+
~500 KB, sized for web visualization): the ideal chunk dim is
29+
`sqrt(target_chunk_bytes / itemsize)` with a floor of 128, then evened out so
30+
chunks divide the dimension as uniformly as possible. Non-spatial dimensions
31+
(time, band, ...) always get chunk size 1.
32+
33+
Shards group `chunks_per_shard` chunks per spatial dimension (default 4, i.e.
34+
4×4 = 16 chunks, ~8 MB). Shards are also the unit of work during generation:
35+
larger shards mean fewer, bigger reads/writes and more memory per worker.
36+
37+
When the source dataset is itself chunked (zarr/icechunk/dask), level-0 chunk
38+
sizes are *snapped* so the destination shard grid nests with the source chunk
39+
grid (the shard divides the source chunk or is a multiple of it), provided a
40+
candidate exists within a factor of 2 of the ideal chunk size. This lets each
41+
source chunk be decoded exactly once during the level-0 copy.
42+
43+
## Streaming memory model
44+
45+
The unit of work is a shard-aligned **region** of the destination array.
46+
Workers on a thread pool each read one region's input, reduce it, and write it
47+
out; nothing larger than `workers x region` is ever in memory.
48+
49+
- **Level 0**: regions are widened per axis to `lcm(shard, source_chunk)` so
50+
whole source chunks are read once, unless that exceeds `max_region_bytes`
51+
(default 256 MB), in which case the plain shard grid is used.
52+
- **Levels 1+**: the region is one output shard; the input block read from the
53+
previous level is the region scaled by the 2×2 stride (~4× larger).
54+
55+
Peak memory is roughly `max_workers * 5 * region_bytes` (source block,
56+
contiguous copy, reduced output, codec buffers). With `max_workers=None` the
57+
pool size is derived from that: `min(2 * cpu_count, mem_budget / (5 *
58+
region_bytes))`, where the budget is half the available RAM. Pass an explicit
59+
`max_workers` to override.
60+
61+
Levels are written sequentially — each one reads the previous — but all
62+
variables within a level stream through one shared pool.
63+
64+
## Kernel semantics
65+
66+
`topozarr_core.block_reduce` (Rust, rayon-parallel, GIL released):
67+
68+
- methods: `mean`, `max`, `min`, `sum`
69+
- dtypes: `u8`, `u16`, `i16`, `i32`, `i64`, `f32`, `f64`
70+
- 1–4 dimensional arrays
71+
- shape follows `xarray.coarsen(boundary="trim")`: trailing partial windows
72+
are dropped; an axis smaller than its stride still yields one window
73+
- `skipna=True` skips NaN and `_FillValue` elements; an all-missing window
74+
produces 0 for `sum` (matching `nansum`) and the fill value (or NaN) for
75+
`mean`/`max`/`min`
76+
77+
## Tuning knobs
78+
79+
| Knob | Where | Effect |
80+
|------|-------|--------|
81+
| `target_chunk_bytes` | `create_pyramid` | chunk size on disk |
82+
| `chunks_per_shard` | `create_pyramid` | shard size = work unit; `None` disables sharding |
83+
| `max_region_bytes` | `Pyramid.write` | cap on level-0 region widening |
84+
| `max_workers` | `Pyramid.write` | thread pool size; `None` = RAM/CPU-derived |
85+
| `progress` | `Pyramid.write` | tqdm bar over written regions |

‎docs/usage.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,17 @@ pyramid.write("pyramid.zarr")
2929

3030
`levels` is the total number of resolution levels including the original. Level `0` is the original (highest) resolution; each subsequent level is coarsened by 2× per spatial dimension.
3131

32-
`create_pyramid` returns a write plan. `pyramid.write(store)` does the work: level 0 is copied from the source dataset, then each level is block-reduced from the previously written one, streaming shard-sized regions through the Rust kernel on a thread pool. Tune parallelism with `max_workers`. Reduction semantics match `xarray.coarsen(boundary="trim")` exactly, including NaN / `_FillValue` handling.
32+
`create_pyramid` returns a write plan. `pyramid.write(store)` does the work: level 0 is copied from the source dataset, then each level is block-reduced from the previously written one, streaming shard-sized regions through the Rust kernel on a thread pool. Reduction semantics match `xarray.coarsen(boundary="trim")` exactly, including NaN / `_FillValue` handling.
33+
34+
## Progress and memory
35+
36+
Pass `progress=True` to show a [tqdm](https://tqdm.github.io/) bar over written regions (requires `tqdm` to be installed):
37+
38+
```python
39+
pyramid.write("pyramid.zarr", progress=True)
40+
```
41+
42+
By default the thread pool size is derived from the CPU count and available RAM; peak memory is roughly `max_workers * 5 * region_bytes`. Pass an explicit `max_workers` to override, and lower `max_region_bytes` (default 256 MB) to shrink level-0 read regions on chunked sources. For bounded memory on large stores, open the source lazily (e.g. `xr.open_zarr(store, chunks=None)`) so regions are materialized one at a time. See [Design](design.md) for the full memory model.
3343

3444
## Visualization hints
3545

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,5 +58,6 @@ plugins:
5858
nav:
5959
- Home: index.md
6060
- Usage: usage.md
61+
- Design: design.md
6162
- API Reference: api.md
6263
- Contributing: contributing.md

0 commit comments

Comments
 (0)