Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ DATA_DIR=./data

## 🚀 Quick Start

For the guided beginner path, start with the
For a guided introduction to both workflows, start with the
**[online documentation](https://spiffical.github.io/onc-hydrophone-data/)**.
An extended [tutorial notebook](notebooks/ONC_Data_Download_Tutorial.ipynb) is
also available for interactive exploration.
Expand Down Expand Up @@ -132,6 +132,7 @@ plot_availability_calendar(availability)
- **Parallel ONC Requests**: Submits many requests at once so ONC processes them in parallel, then downloads when ready (faster than sequential requests)
- **Resumable Audio Downloads**: Downloads FLAC/WAV files in parallel and skips files already present locally
- **Custom Spectrograms**: Generate spectrograms with configurable parameters
- **Event-Centred Spectrograms**: Retain a precise signal window while using automatic STFT context to prevent edge effects
- **Deployment Validation**: Ensures data exists for requested time periods
- **Deployment Availability Visuals**: Timeline/calendar views of data availability by device
- **Interactive Mode**: Guided CLI for easy setup
Expand Down
68 changes: 66 additions & 2 deletions docs/custom_spectrograms.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,22 @@ If you have not downloaded audio yet, start with
**[Download Audio](audio_downloads.md)** or the complete
**[audio-to-spectrogram walkthrough](quickstart.md)**.

## Choose the local workflow

| Workflow | What it does | Clip-boundary handling |
| --- | --- | --- |
| `process_single_file()` / `process_directory()` | Computes a spectrogram from a complete local audio file | Uses only complete STFT windows; it does not add artificial samples beyond the file |
| `clip_start` / `clip_end` or `--clip-start` / `--clip-end` | Computes a selected interval from local audio | Uses automatic half-window context by default, then removes it |
| `process_event()` | Computes around a known time in one local audio file | Automatically reads an extra half-window on each side, then removes that context |
| `--event-time` command-line mode | Command-line form of `process_event()` | Same automatic half-window context and trimming |
| `create_custom_spectrograms_from_json()` | Downloads ONC audio for timestamped events and computes spectrograms locally | Automatically downloads extra context, computes the STFT, and trims back to the requested interval |
| `download_requests_from_json()` | Downloads spectrogram products computed by ONC | Processing at product boundaries is controlled by ONC |

Use `create_custom_spectrograms_from_json()` when you want the FFT and plotting
settings in the JSON file to control newly computed local spectrograms. Use
`download_requests_from_json()` when you want ONC's existing or
server-generated spectrogram products.

## Process an audio directory

```python
Expand Down Expand Up @@ -110,7 +126,44 @@ python scripts/generate_spectrograms.py \

Run `python scripts/generate_spectrograms.py --help` for every option.

## Generate event clips from JSON
## Generate around a known signal time

Use event mode when a signal occurs at a known offset in an existing audio
file. The default retains five seconds before and after the event. It also reads
an extra half-window of audio on each side while computing the STFT, then keeps
only frames centred inside the requested ten-second interval. This ensures that
every retained time bin is calculated from a complete analysis window.

```python
result = generator.process_event(
audio_dir / "example.flac",
output_dir,
event_time_seconds=123.4,
pad_before_seconds=5,
pad_after_seconds=5,
edge_padding_seconds="auto",
save_plot=True,
save_mat=True,
)
```

`edge_padding_seconds="auto"` is the default and resolves to half the actual
STFT window after the audio sample rate and any `win_length` override are known.
The resolved value, event time, target interval, and retained padding are stored
in the output metadata.

The same mode is available from the command line:

```bash
python scripts/generate_spectrograms.py \
--input-file audio/example.flac \
--event-time 123.4 \
--event-pad-before 5 \
--event-pad-after 5 \
--output-dir spectrograms
```

## Generate local event spectrograms from JSON

For many labeled events, one workflow can download the needed audio context,
clip each event, and generate local spectrograms:
Expand All @@ -124,6 +177,7 @@ dl = HydrophoneDownloader(onc_token, data_dir)

results = dl.create_custom_spectrograms_from_json(
"custom_requests.json",
clip_pad_seconds="auto",
save_mat=True,
save_png=True,
)
Expand Down Expand Up @@ -154,7 +208,17 @@ results = dl.create_custom_spectrograms_from_json(
```

The workflow requests adjacent source files when padding crosses a five-minute
boundary and trims the generated result back to the target interval.
boundary. With `clip_pad_seconds="auto"`, it adds half of the configured
`win_dur` on both sides before computing the STFT, removes that context before
relative-dB normalization, and returns only time bins centred inside the
requested event interval. This prevents incomplete-window artifacts at the
requested clip boundaries. If `generator_options` sets a sample-based
`win_length` that differs from `win_dur`, set `clip_pad_seconds` explicitly in
seconds so the download context matches that window.

This edge handling applies to spectrograms computed locally by this package.
Spectrograms returned by `download_requests_from_json()` or the other ONC
spectrogram download methods are computed by ONC and follow ONC's processing.

## Understand the saved values

Expand Down
17 changes: 13 additions & 4 deletions docs/downloads.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Advanced and Batch Download Workflows

This page covers server-generated spectrograms, sampling, event batches, and
JSON/CSV request files. If this is your first ONC download, start with
**[Download Audio and Make a Spectrogram](quickstart.md)**. For ordinary audio
ranges, see **[Download Audio](audio_downloads.md)**.
JSON/CSV request files. The complete **[Download Audio and Make a
Spectrogram](quickstart.md)** example covers the common local-generation
workflow. For ordinary audio ranges, see **[Download Audio](audio_downloads.md)**.

For the differences between ONC's one-minute, plot-resolution, and
full-resolution MAT products—plus concatenation, source, channel, diversion,
Expand Down Expand Up @@ -100,7 +100,16 @@ result = dl.download_audio_for_range(
)
```

## JSON/CSV request files
## JSON/CSV requests for ONC products

This workflow downloads audio and/or spectrogram products from ONC. With its
default `download_spectrogram: true`, the spectrogram is computed by ONC; the
local `SpectrogramGenerator` and its edge-context settings are not used.

To use JSON timestamps to download source audio and compute your own
spectrograms with custom FFT settings and automatic clip-boundary context, use
[`create_custom_spectrograms_from_json()`](custom_spectrograms.md#generate-local-event-spectrograms-from-json)
instead.

```python
results = dl.download_requests_from_json("/path/to/requests.json")
Expand Down
27 changes: 15 additions & 12 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
# ONC Hydrophone Data

Download hydrophone audio from Ocean Networks Canada (ONC), then turn it into
spectrograms with parameters you control.
Download hydrophone audio and generate spectrograms locally, or retrieve
spectrogram products generated by Ocean Networks Canada (ONC).

!!! tip "New to ONC hydrophones?"
Follow the three **Start Here** pages in order. They take you from an ONC
account to your first locally generated spectrogram.
The three **Start Here** pages lead with the most common workflow: going
from an ONC account to downloaded audio and a locally generated
spectrogram. ONC-generated spectrogram products are introduced below and
covered fully in their own guide.

## The beginner path
## Start with the common audio workflow

1. **[Install and configure](setup.md)** — install the package, save your ONC
token safely, and choose a data directory.
Expand All @@ -22,24 +24,25 @@ spectrograms with parameters you control.
ICLISTENHF1205 at Folger Passage, 2012-08-01 12:24 UTC. Audio source and credit:
[Ocean Networks Canada Multimedia Manager](https://ibase.oceannetworks.ca/view-item?i=9860).*

## Audio first, server products second
## Two ways to work with spectrograms

Most users should download **FLAC/WAV audio** and create spectrograms locally.
That path preserves the source audio and lets you change window length,
A common workflow is to download **FLAC/WAV audio** and create spectrograms
locally. This preserves the source audio and lets you change window length,
frequency range, overlap, colour limits, and output format without requesting
the data again.

ONC also offers server-generated MAT, PNG, and PDF spectral products. Those are
useful when you need calibrated ONC products, compact long-term summaries, or a
quick visual scan. See **[Choose ONC Server Spectrograms](onc_spectrogram_options.md)**
when that is your goal.
The other workflow is to download ONC-generated MAT, PNG, and PDF spectral
products. These are useful when you need calibrated ONC products, compact
long-term summaries, or a quick visual scan. See **[Choose ONC Server
Spectrograms](onc_spectrogram_options.md)** for that workflow.

## Choose the guide for your task

| I want to… | Start with |
| --- | --- |
| Download a short audio range | [Download Audio](audio_downloads.md) |
| Generate PNG/MAT spectrograms from audio | [Generate Local Spectrograms](custom_spectrograms.md) |
| Generate an edge-safe spectrogram around a known signal time | [Generate Local Spectrograms](custom_spectrograms.md#generate-around-a-known-signal-time) |
| Check whether a device has data for my dates | [Find a Hydrophone](inventory.md) |
| Sample many windows or download events from JSON/CSV | [Advanced & Batch Downloads](downloads.md) |
| Understand ONC's one-minute, plot, or full-resolution products | [Choose ONC Server Spectrograms](onc_spectrogram_options.md) |
Expand Down
10 changes: 5 additions & 5 deletions docs/onc_spectrogram_options.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ its server-side options map to this package. ONC reports the options available
for a particular device and format through its discovery API, so a device may
offer only a subset of the values listed here.

!!! tip "Most beginners should start with audio"
If your goal is to create spectrograms with your own settings, follow
**[Download Audio and Make a Spectrogram](quickstart.md)** instead. Use this
page when you specifically need ONC's server-generated or calibrated
spectral products.
!!! tip "Choose the workflow that matches the data you need"
To retain the source audio and control the FFT and output settings, follow
**[Download Audio and Make a Spectrogram](quickstart.md)**. To retrieve
ONC-generated or calibrated spectral products, use the options on this
page.

## Quick chooser

Expand Down
6 changes: 5 additions & 1 deletion docs/quickstart.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
# 3. Download Audio and Make a Spectrogram

This walkthrough downloads a short ONC audio range and generates PNG and MAT
spectrograms locally. It is the recommended first workflow for new users.
spectrograms locally. This common workflow is presented first because many
users want the source audio and control over their spectrogram settings. ONC's
server-generated spectrogram products are introduced on the
**[ONC Spectrogram Products and Server Options](onc_spectrogram_options.md)**
page.

Before continuing, complete **[Install and Configure](setup.md)** and use
**[Find a Hydrophone](inventory.md)** to confirm that your device and dates are
Expand Down
Loading