Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 66 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Docs](https://img.shields.io/badge/docs-online-brightgreen.svg)](https://spiffical.github.io/onc-hydrophone-data/)

Tools for downloading and processing Ocean Networks Canada hydrophone data, including spectrograms, FLAC audio files, and custom spectrogram generation.
Python tools for downloading Ocean Networks Canada hydrophone audio and
server-generated spectral products, checking deployment availability, and
generating custom spectrograms locally.

## 📦 Installation

Expand Down Expand Up @@ -53,7 +55,7 @@ For a guided introduction to both workflows, start with the
An extended [tutorial notebook](notebooks/ONC_Data_Download_Tutorial.ipynb) is
also available for interactive exploration.

### Python API
### Common workflow: download audio and generate spectrograms locally

```python
from datetime import datetime, timezone
Expand All @@ -66,11 +68,15 @@ from onc_hydrophone_data.onc.common import load_config
onc_token, data_dir = load_config()
downloader = HydrophoneDownloader(onc_token, data_dir)

device = "ICLISTENHF6324"
start = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)
end = datetime(2024, 4, 1, 12, 10, tzinfo=timezone.utc)

# Download a short, verified ONC audio range.
downloader.download_audio_for_range(
device_code="ICLISTENHF6324",
start_dt=datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc),
end_dt=datetime(2024, 4, 1, 12, 10, tzinfo=timezone.utc),
device_code=device,
start_dt=start,
end_dt=end,
)

# Generate PNG and MAT spectrograms locally from the downloaded audio.
Expand All @@ -89,6 +95,48 @@ generator.process_directory(
)
```

### Download ONC-generated spectrogram products

Use the same downloader when you want ONC to provide the spectral product
instead of computing it locally:

```python
result = downloader.download_spectrograms_for_range(
device_code=device,
start_dt=start,
end_dt=end,
spectrograms_per_batch=2,
)
print(result)
```

See the [ONC spectrogram products and server options
guide](https://spiffical.github.io/onc-hydrophone-data/onc_spectrogram_options/)
for one-minute, plot-resolution, and full-resolution products.

### Generate around a known signal time

When a signal occurs at a known offset in an audio file, event mode retains the
requested signal window and automatically includes extra computation context
to prevent incomplete-window artifacts at its boundaries:

```python
audio_file = next(
path
for pattern in ("*.flac", "*.wav")
for path in audio_dir.glob(pattern)
)
event_result = generator.process_event(
audio_file,
audio_dir.parent / "event_spectrograms",
event_time_seconds=30.0,
pad_before_seconds=5.0,
pad_after_seconds=5.0,
edge_padding_seconds="auto",
)
print(event_result["png_file"])
```

### Command Line

```bash
Expand All @@ -106,6 +154,11 @@ python scripts/download_hydrophone_data.py --mode sampling \
# Generate custom spectrograms
python scripts/generate_spectrograms.py --input-dir data/DEVICE/audio/ --win-dur 2.0

# Generate around a known signal time with automatic edge context
python scripts/generate_spectrograms.py --input-file audio/example.flac \
--event-time 30 --event-pad-before 5 --event-pad-after 5 \
--output-dir event_spectrograms/

# Save only the frequency range needed by the dashboard (much smaller MAT files)
python scripts/generate_spectrograms.py --input-dir data/DEVICE/audio/ \
--freq-min 10 --freq-max 10000 --crop-freq-lims
Expand Down Expand Up @@ -133,6 +186,7 @@ plot_availability_calendar(availability)
- **Resumable Audio Downloads**: Downloads FLAC/WAV files in parallel and skips files already present locally
- **Custom Spectrograms**: Generate spectrograms with configurable parameters
- **Event-Centred Spectrograms**: Retain a precise signal window while using automatic STFT context to prevent edge effects
- **JSON Event Workflows**: Download ONC products or generate local event spectrograms with clearly separated request methods
- **Deployment Validation**: Ensures data exists for requested time periods
- **Deployment Availability Visuals**: Timeline/calendar views of data availability by device
- **Interactive Mode**: Guided CLI for easy setup
Expand All @@ -149,10 +203,10 @@ data/
│ ├── *.mat # Spectrogram data files
│ └── anomaly_report.txt # Any validation issues (if found)
├── audio/ # Downloaded audio files
│ └── *.flac
│ └── *.flac / *.wav
└── custom_spectrograms/ # Locally-generated spectrograms
├── mat/ # Custom MAT files
└── png/ # Custom PNG plots
├── *.mat # Spectrogram arrays + metadata
└── *.png # Spectrogram plots
```

## 🛠️ Troubleshooting
Expand All @@ -161,7 +215,8 @@ data/
|-------|----------|
| Invalid ONC Token | Verify token in `.env` file |
| No data found | Use `--check-deployments` to verify coverage |
| Memory errors | Reduce `--spectrograms-per-batch` |
| ONC request timeouts | Reduce `--spectrograms-per-batch` or request a shorter range |
| Local generation is slow or uses too much memory | Use `--max-workers 1` and `--crop-freq-lims` with a focused frequency range |

## 📚 Documentation

Expand All @@ -170,7 +225,8 @@ See the **[Tutorial Notebook](notebooks/ONC_Data_Download_Tutorial.ipynb)** for
- Different download modes (sampling, range, specific times)
- Parallel download optimization
- Custom spectrogram generation
- JSON timestamp requests
- Edge-safe generation around known signal times
- JSON requests for local generation versus ONC-generated products

## 📄 License

Expand Down
11 changes: 6 additions & 5 deletions docs/inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,9 @@ timeline_fig.savefig(
![Timeline generated from real ONC archive availability for ICLISTENHF6324](assets/figures/availability_timeline.webp){: width="100%" loading="lazy" }

The availability result distinguishes archived data, gaps during a deployment,
and dates when the device was not deployed. Coverage is calculated from the
fraction of expected files present in each daily bin.
and dates when the device was not deployed. Coverage is the duration for which
merged archived audio intervals overlap each daily bin, divided by the bin's
total duration.

### Calendar view

Expand All @@ -124,9 +125,9 @@ calendar_fig.savefig(

![Calendar generated from real ONC archive availability for ICLISTENHF6324](assets/figures/availability_calendar.webp){: width="100%" loading="lazy" }

Each calendar value is the fraction of expected files found for that day. A
zero value within a deployment is an archive gap; dates outside a deployment
are recorded separately rather than treated as missing data.
Each calendar value is the fraction of that day covered by archived audio time
intervals. A zero value within a deployment is an archive gap; dates outside a
deployment are recorded separately rather than treated as missing data.

!!! info "Real archive result"
Both plots above were generated on 2026-07-14 by running the documented
Expand Down
23 changes: 20 additions & 3 deletions onc_hydrophone_data/audio/spectrogram_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -694,6 +694,7 @@ def process_single_file(self, audio_path: Union[str, Path],

Raises:
FileNotFoundError: If the audio file does not exist.
ValueError: If ``output_stem`` is a path or includes an extension.

Example:
```python
Expand All @@ -709,6 +710,25 @@ def process_single_file(self, audio_path: Union[str, Path],
"""
audio_path = Path(audio_path)
save_dir = Path(save_dir)

if output_stem is None:
base_name = audio_path.stem
else:
base_name = str(output_stem)
if (
not base_name.strip()
or base_name in {'.', '..'}
or '/' in base_name
or '\\' in base_name
):
raise ValueError(
"output_stem must be a filename stem, not a path"
)
if Path(base_name).suffix or base_name.endswith('.'):
raise ValueError(
"output_stem must not include a file extension"
)

save_dir.mkdir(parents=True, exist_ok=True)

# Load audio
Expand All @@ -727,9 +747,6 @@ def process_single_file(self, audio_path: Union[str, Path],
)

# Create output filenames
base_name = Path(output_stem).name if output_stem else audio_path.stem
if not base_name or base_name in {'.', '..'}:
raise ValueError("output_stem must contain a valid filename stem")
mat_path = save_dir / f"{base_name}.mat"
png_path = save_dir / f"{base_name}.png"
npy_path = save_dir / f"{base_name}.npy"
Expand Down
30 changes: 24 additions & 6 deletions scripts/generate_spectrograms.py
Original file line number Diff line number Diff line change
Expand Up @@ -653,6 +653,29 @@ def main():
help='Disable logging module output')

args = parser.parse_args()

if args.event_time is not None:
Comment on lines 655 to +657
if not args.input_file:
parser.error("--event-time requires --input-file")
if args.clip_start is not None or args.clip_end is not None:
parser.error(
"--event-time cannot be combined with --clip-start/--clip-end"
)
if args.event_time < 0:
parser.error("--event-time must be non-negative")
if args.event_pad_before < 0:
parser.error("--event-pad-before must be non-negative")
event_pad_after = (
args.event_pad_before
if args.event_pad_after is None
else args.event_pad_after
)
if event_pad_after < 0:
parser.error("--event-pad-after must be non-negative")
if args.event_pad_before + event_pad_after <= 0:
parser.error("event padding must retain more than zero seconds")
if args.clip_pad_seconds is not None and args.clip_pad_seconds < 0:
parser.error("--edge-pad-seconds must be non-negative")

try:
print_header("CUSTOM SPECTROGRAM GENERATOR")
Expand All @@ -679,12 +702,7 @@ def main():
if not input_path_obj.exists():
print_status(f"Input path not found: {input_path}", "ERROR")
return
if args.event_time is not None:
if is_directory:
raise ValueError("--event-time requires --input-file")
if args.clip_start is not None or args.clip_end is not None:
raise ValueError("--event-time cannot be combined with --clip-start/--clip-end")


# Determine output directory
output_dir = determine_output_directory(input_path, is_directory, args.output_dir)
print_status(f"Output directory: {output_dir}", "INFO")
Expand Down
58 changes: 58 additions & 0 deletions tests/test_generate_spectrograms_cli.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
import subprocess
import sys
from pathlib import Path

import pytest


REPO_ROOT = Path(__file__).resolve().parents[1]
SCRIPT = REPO_ROOT / "scripts" / "generate_spectrograms.py"


@pytest.mark.parametrize(
("input_flag", "extra_args", "message"),
[
("--input-dir", [], "--event-time requires --input-file"),
(
"--input-file",
["--clip-start", "0"],
"--event-time cannot be combined with --clip-start/--clip-end",
),
(
"--input-file",
["--event-pad-before", "-1"],
"--event-pad-before must be non-negative",
),
],
)
def test_event_argument_errors_use_argparse_without_traceback(
tmp_path: Path,
input_flag: str,
extra_args: list[str],
message: str,
):
input_path = tmp_path
if input_flag == "--input-file":
input_path = tmp_path / "audio.wav"
input_path.touch()

completed = subprocess.run(
[
sys.executable,
str(SCRIPT),
input_flag,
str(input_path),
"--event-time",
"1",
*extra_args,
],
cwd=REPO_ROOT,
text=True,
capture_output=True,
check=False,
)

assert completed.returncode == 2
assert f"error: {message}" in completed.stderr
assert "Traceback" not in completed.stderr
assert "Unexpected error" not in completed.stdout + completed.stderr
27 changes: 27 additions & 0 deletions tests/test_spectrogram_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,33 @@ def test_hashable_windows_are_cached():
assert first is second


@pytest.mark.parametrize(
("output_stem", "message"),
[
("event.mat", "must not include a file extension"),
("event.", "must not include a file extension"),
("nested/event", "must be a filename stem"),
(r"nested\\event", "must be a filename stem"),
],
)
def test_process_single_file_rejects_invalid_output_stems_before_loading(
tmp_path: Path,
output_stem: str,
message: str,
):
generator = SpectrogramGenerator(quiet=True)
output_dir = tmp_path / "spectrograms"

with pytest.raises(ValueError, match=message):
generator.process_single_file(
tmp_path / "missing.wav",
output_dir,
output_stem=output_stem,
)

assert not output_dir.exists()


def test_process_event_uses_complete_windows_and_trims_context(tmp_path: Path):
sample_rate = 1_000
duration_seconds = 20
Expand Down