diff --git a/README.md b/README.md index e9454ae..429c2ec 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ DATA_DIR=./data ## 🚀 Quick Start -For the guided beginner path, start with the +For a guided introduction to both workflows, start with the **[online documentation](https://spiffical.github.io/onc-hydrophone-data/)**. An extended [tutorial notebook](notebooks/ONC_Data_Download_Tutorial.ipynb) is also available for interactive exploration. @@ -132,6 +132,7 @@ plot_availability_calendar(availability) - **Parallel ONC Requests**: Submits many requests at once so ONC processes them in parallel, then downloads when ready (faster than sequential requests) - **Resumable Audio Downloads**: Downloads FLAC/WAV files in parallel and skips files already present locally - **Custom Spectrograms**: Generate spectrograms with configurable parameters +- **Event-Centred Spectrograms**: Retain a precise signal window while using automatic STFT context to prevent edge effects - **Deployment Validation**: Ensures data exists for requested time periods - **Deployment Availability Visuals**: Timeline/calendar views of data availability by device - **Interactive Mode**: Guided CLI for easy setup diff --git a/docs/custom_spectrograms.md b/docs/custom_spectrograms.md index 9c78728..797e998 100644 --- a/docs/custom_spectrograms.md +++ b/docs/custom_spectrograms.md @@ -7,6 +7,22 @@ If you have not downloaded audio yet, start with **[Download Audio](audio_downloads.md)** or the complete **[audio-to-spectrogram walkthrough](quickstart.md)**. +## Choose the local workflow + +| Workflow | What it does | Clip-boundary handling | +| --- | --- | --- | +| `process_single_file()` / `process_directory()` | Computes a spectrogram from a complete local audio file | Uses only complete STFT windows; it does not add artificial samples beyond the file | +| `clip_start` / `clip_end` or `--clip-start` / `--clip-end` | Computes a selected interval from local audio | Uses automatic half-window context by default, then removes it | +| `process_event()` | Computes around a known time in one local audio file | Automatically reads an extra half-window on each side, then removes that context | +| `--event-time` command-line mode | Command-line form of `process_event()` | Same automatic half-window context and trimming | +| `create_custom_spectrograms_from_json()` | Downloads ONC audio for timestamped events and computes spectrograms locally | Automatically downloads extra context, computes the STFT, and trims back to the requested interval | +| `download_requests_from_json()` | Downloads spectrogram products computed by ONC | Processing at product boundaries is controlled by ONC | + +Use `create_custom_spectrograms_from_json()` when you want the FFT and plotting +settings in the JSON file to control newly computed local spectrograms. Use +`download_requests_from_json()` when you want ONC's existing or +server-generated spectrogram products. + ## Process an audio directory ```python @@ -110,7 +126,44 @@ python scripts/generate_spectrograms.py \ Run `python scripts/generate_spectrograms.py --help` for every option. -## Generate event clips from JSON +## Generate around a known signal time + +Use event mode when a signal occurs at a known offset in an existing audio +file. The default retains five seconds before and after the event. It also reads +an extra half-window of audio on each side while computing the STFT, then keeps +only frames centred inside the requested ten-second interval. This ensures that +every retained time bin is calculated from a complete analysis window. + +```python +result = generator.process_event( + audio_dir / "example.flac", + output_dir, + event_time_seconds=123.4, + pad_before_seconds=5, + pad_after_seconds=5, + edge_padding_seconds="auto", + save_plot=True, + save_mat=True, +) +``` + +`edge_padding_seconds="auto"` is the default and resolves to half the actual +STFT window after the audio sample rate and any `win_length` override are known. +The resolved value, event time, target interval, and retained padding are stored +in the output metadata. + +The same mode is available from the command line: + +```bash +python scripts/generate_spectrograms.py \ + --input-file audio/example.flac \ + --event-time 123.4 \ + --event-pad-before 5 \ + --event-pad-after 5 \ + --output-dir spectrograms +``` + +## Generate local event spectrograms from JSON For many labeled events, one workflow can download the needed audio context, clip each event, and generate local spectrograms: @@ -124,6 +177,7 @@ dl = HydrophoneDownloader(onc_token, data_dir) results = dl.create_custom_spectrograms_from_json( "custom_requests.json", + clip_pad_seconds="auto", save_mat=True, save_png=True, ) @@ -154,7 +208,17 @@ results = dl.create_custom_spectrograms_from_json( ``` The workflow requests adjacent source files when padding crosses a five-minute -boundary and trims the generated result back to the target interval. +boundary. With `clip_pad_seconds="auto"`, it adds half of the configured +`win_dur` on both sides before computing the STFT, removes that context before +relative-dB normalization, and returns only time bins centred inside the +requested event interval. This prevents incomplete-window artifacts at the +requested clip boundaries. If `generator_options` sets a sample-based +`win_length` that differs from `win_dur`, set `clip_pad_seconds` explicitly in +seconds so the download context matches that window. + +This edge handling applies to spectrograms computed locally by this package. +Spectrograms returned by `download_requests_from_json()` or the other ONC +spectrogram download methods are computed by ONC and follow ONC's processing. ## Understand the saved values diff --git a/docs/downloads.md b/docs/downloads.md index 7f7eeb2..26c68d2 100644 --- a/docs/downloads.md +++ b/docs/downloads.md @@ -1,9 +1,9 @@ # Advanced and Batch Download Workflows This page covers server-generated spectrograms, sampling, event batches, and -JSON/CSV request files. If this is your first ONC download, start with -**[Download Audio and Make a Spectrogram](quickstart.md)**. For ordinary audio -ranges, see **[Download Audio](audio_downloads.md)**. +JSON/CSV request files. The complete **[Download Audio and Make a +Spectrogram](quickstart.md)** example covers the common local-generation +workflow. For ordinary audio ranges, see **[Download Audio](audio_downloads.md)**. For the differences between ONC's one-minute, plot-resolution, and full-resolution MAT products—plus concatenation, source, channel, diversion, @@ -100,7 +100,16 @@ result = dl.download_audio_for_range( ) ``` -## JSON/CSV request files +## JSON/CSV requests for ONC products + +This workflow downloads audio and/or spectrogram products from ONC. With its +default `download_spectrogram: true`, the spectrogram is computed by ONC; the +local `SpectrogramGenerator` and its edge-context settings are not used. + +To use JSON timestamps to download source audio and compute your own +spectrograms with custom FFT settings and automatic clip-boundary context, use +[`create_custom_spectrograms_from_json()`](custom_spectrograms.md#generate-local-event-spectrograms-from-json) +instead. ```python results = dl.download_requests_from_json("/path/to/requests.json") diff --git a/docs/index.md b/docs/index.md index 0a07ccc..dbc537d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,13 +1,15 @@ # ONC Hydrophone Data -Download hydrophone audio from Ocean Networks Canada (ONC), then turn it into -spectrograms with parameters you control. +Download hydrophone audio and generate spectrograms locally, or retrieve +spectrogram products generated by Ocean Networks Canada (ONC). !!! tip "New to ONC hydrophones?" - Follow the three **Start Here** pages in order. They take you from an ONC - account to your first locally generated spectrogram. + The three **Start Here** pages lead with the most common workflow: going + from an ONC account to downloaded audio and a locally generated + spectrogram. ONC-generated spectrogram products are introduced below and + covered fully in their own guide. -## The beginner path +## Start with the common audio workflow 1. **[Install and configure](setup.md)** — install the package, save your ONC token safely, and choose a data directory. @@ -22,17 +24,17 @@ spectrograms with parameters you control. ICLISTENHF1205 at Folger Passage, 2012-08-01 12:24 UTC. Audio source and credit: [Ocean Networks Canada Multimedia Manager](https://ibase.oceannetworks.ca/view-item?i=9860).* -## Audio first, server products second +## Two ways to work with spectrograms -Most users should download **FLAC/WAV audio** and create spectrograms locally. -That path preserves the source audio and lets you change window length, +A common workflow is to download **FLAC/WAV audio** and create spectrograms +locally. This preserves the source audio and lets you change window length, frequency range, overlap, colour limits, and output format without requesting the data again. -ONC also offers server-generated MAT, PNG, and PDF spectral products. Those are -useful when you need calibrated ONC products, compact long-term summaries, or a -quick visual scan. See **[Choose ONC Server Spectrograms](onc_spectrogram_options.md)** -when that is your goal. +The other workflow is to download ONC-generated MAT, PNG, and PDF spectral +products. These are useful when you need calibrated ONC products, compact +long-term summaries, or a quick visual scan. See **[Choose ONC Server +Spectrograms](onc_spectrogram_options.md)** for that workflow. ## Choose the guide for your task @@ -40,6 +42,7 @@ when that is your goal. | --- | --- | | Download a short audio range | [Download Audio](audio_downloads.md) | | Generate PNG/MAT spectrograms from audio | [Generate Local Spectrograms](custom_spectrograms.md) | +| Generate an edge-safe spectrogram around a known signal time | [Generate Local Spectrograms](custom_spectrograms.md#generate-around-a-known-signal-time) | | Check whether a device has data for my dates | [Find a Hydrophone](inventory.md) | | Sample many windows or download events from JSON/CSV | [Advanced & Batch Downloads](downloads.md) | | Understand ONC's one-minute, plot, or full-resolution products | [Choose ONC Server Spectrograms](onc_spectrogram_options.md) | diff --git a/docs/onc_spectrogram_options.md b/docs/onc_spectrogram_options.md index 9029978..8c3e103 100644 --- a/docs/onc_spectrogram_options.md +++ b/docs/onc_spectrogram_options.md @@ -11,11 +11,11 @@ its server-side options map to this package. ONC reports the options available for a particular device and format through its discovery API, so a device may offer only a subset of the values listed here. -!!! tip "Most beginners should start with audio" - If your goal is to create spectrograms with your own settings, follow - **[Download Audio and Make a Spectrogram](quickstart.md)** instead. Use this - page when you specifically need ONC's server-generated or calibrated - spectral products. +!!! tip "Choose the workflow that matches the data you need" + To retain the source audio and control the FFT and output settings, follow + **[Download Audio and Make a Spectrogram](quickstart.md)**. To retrieve + ONC-generated or calibrated spectral products, use the options on this + page. ## Quick chooser diff --git a/docs/quickstart.md b/docs/quickstart.md index a5104c0..46ed262 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -1,7 +1,11 @@ # 3. Download Audio and Make a Spectrogram This walkthrough downloads a short ONC audio range and generates PNG and MAT -spectrograms locally. It is the recommended first workflow for new users. +spectrograms locally. This common workflow is presented first because many +users want the source audio and control over their spectrogram settings. ONC's +server-generated spectrogram products are introduced on the +**[ONC Spectrogram Products and Server Options](onc_spectrogram_options.md)** +page. Before continuing, complete **[Install and Configure](setup.md)** and use **[Find a Hydrophone](inventory.md)** to confirm that your device and dates are diff --git a/notebooks/ONC_Data_Download_Tutorial.ipynb b/notebooks/ONC_Data_Download_Tutorial.ipynb index a14a114..ab88043 100644 --- a/notebooks/ONC_Data_Download_Tutorial.ipynb +++ b/notebooks/ONC_Data_Download_Tutorial.ipynb @@ -1,2845 +1,1340 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Table of Contents\n", - "\n", - "- [1. Introduction & Setup](#1-introduction-and-setup)\n", - " - [What This Notebook Covers](#what-this-notebook-covers)\n", - " - [ONC Data Products Overview](#onc-data-products-overview)\n", - " - [Prerequisites](#prerequisites)\n", - " - [1.1 Hydrophone Deployments & Inventory](#11-hydrophone-deployments-and-inventory)\n", - " - [1.1a Hydrophone Inventory (Current + History)](#11a-hydrophone-inventory-current-history)\n", - " - [1.1b Interactive Availability Widget (Plotly + ipywidgets)](#11b-interactive-availability-widget-plotly-ipywidgets)\n", - "- [2. Download Workflows (Spectrograms / Audio / Both)](#2-download-workflows-spectrograms-audio-both)\n", - " - [2.1 Basic Spectrogram Download (2 Spectrograms / 10 Minutes)](#21-basic-spectrogram-download-2-spectrograms-10-minutes)\n", - " - [2.2 Range Downloads (Between Two Dates)](#22-range-downloads-between-two-dates)\n", - " - [Spectrograms (optional audio)](#spectrograms-optional-audio)\n", - " - [Audio only](#audio-only)\n", - " - [2.3 Sampling Mode (Uniform Samples Across Range)](#23-sampling-mode-uniform-samples-across-range)\n", - " - [Spectrograms (optional audio)](#spectrograms-optional-audio-1)\n", - " - [Audio only](#audio-only-1)\n", - " - [2.4 Event-Based Downloads (Simple)](#24-event-based-downloads-simple)\n", - " - [Spectrograms (optional audio)](#spectrograms-optional-audio-2)\n", - " - [Audio only](#audio-only-2)\n", - " - [2.5 Centered Audio Clip (Custom Duration)](#25-centered-audio-clip-custom-duration)\n", - "- [3. Custom Spectrogram Generation (Local)](#3-custom-spectrogram-generation-local)\n", - " - [3.1 SpectrogramGenerator Basics](#31-spectrogramgenerator-basics)\n", - " - [3.2 Custom Parameters](#32-custom-parameters)\n", - " - [3.3 Batch Processing Audio Directory](#33-batch-processing-audio-directory)\n", - "- [4. Event Requests & Request Files](#4-event-requests-and-request-files)\n", - " - [4.1 Direct Timestamps (Python Lists / Datetime Objects)](#41-direct-timestamps-python-lists-datetime-objects)\n", - " - [4.2 Request Files (JSON + CSV)](#42-request-files-json-csv)\n", - " - [4.2a JSON Example + Execution](#42a-json-example-execution)\n", - " - [4.2b CSV Example + Execution](#42b-csv-example-execution)\n", - " - [4.3 Supported Date/Time Formats](#43-supported-datetime-formats)\n", - "- [5. End-to-End Pipelines](#5-end-to-end-pipelines)\n", - " - [5.1 Request-Driven Audio Downloads + Local Spectrograms (JSON/CSV)](#51-request-driven-audio-downloads-local-spectrograms-jsoncsv)\n", - " - [5.2 Batch Pipeline: Download Audio → Local Spectrograms](#52-batch-pipeline-download-audio-local-spectrograms)\n", - " - [5.3 Multi-Device Downloads](#53-multi-device-downloads)\n", - "- [6. Output Folder Structure](#6-output-folder-structure)\n", - "- [7. Troubleshooting & Tips](#7-troubleshooting-and-tips)\n", - " - [Common Issues](#common-issues)\n", - " - [Performance Tips](#performance-tips)\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 1. Introduction & Setup\n", - "This section covers setup, prerequisites, and the main data products you'll use throughout the notebook.\n", - "\n", - "## What This Notebook Covers\n", - "\n", - "This notebook demonstrates how to:\n", - "- Download **ONC-generated spectrograms** (MAT/PNG files)\n", - "- Download **raw audio files** (FLAC/WAV)\n", - "- Create **custom spectrograms** from audio with your own parameters\n", - "- Handle various **input formats** (JSON, CSV, Python lists)\n", - "- Work with **specific timestamps** or **date ranges**\n", - "\n", - "## ONC Data Products Overview\n", - "\n", - "| Product Code | Description | Use Case |\n", - "| --- | --- | --- |\n", - "| `HSD` | Hydrophone Spectrogram Data | Spectrogram plots + spectral MAT; 1-min MAT pre-generated, higher-res on request |\n", - "| `HAF` | Hydrophone Audio Files | Raw audio (FLAC/WAV) |\n", - "\n", - "## Prerequisites\n", - "\n", - "- `.env` with `ONC_TOKEN=...` in the repo root (optional: `DATA_DIR=/path/to/data`; default is `data/`)\n", - "- Package installed: `pip install onc-hydrophone-data`\n", - "- All timestamps are converted to UTC for requests; provide tz-aware datetimes or a `timezone` field in JSON/CSV.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "# Standard imports\n", - "import os\n", - "import sys\n", - "import json\n", - "import numpy as np\n", - "from pathlib import Path\n", - "from datetime import datetime, timedelta, timezone\n", - "\n", - "# Ensure repo is in path\n", - "REPO_ROOT = Path(\"..\").resolve()\n", - "if str(REPO_ROOT) not in sys.path:\n", - " sys.path.append(str(REPO_ROOT))\n", - "\n", - "# Core imports\n", - "from onc_hydrophone_data.onc.common import load_config, print_status\n", - "from onc_hydrophone_data.data.hydrophone_downloader import HydrophoneDownloader\n", - "\n", - "from onc_hydrophone_data.utils.plotting import (\n", - " find_first_file,\n", - " plot_first_spectrogram,\n", - " plot_first_audio,\n", - " plot_onc_mat_spectrogram,\n", - " plot_audio_waveform,\n", - " plot_clip_pair,\n", - " plot_spectrogram_clip,\n", - " plot_request_results,\n", - ")\n" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "✅ Data directory: /home/sbialek/ONC/onc-hydrophone-data/data\n" - ] - } - ], - "source": [ - "# Load configuration\n", - "ONC_TOKEN, DATA_DIR = load_config()\n", - "dl = HydrophoneDownloader(ONC_TOKEN, DATA_DIR)\n", - "print(f\"✅ Data directory: {DATA_DIR}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## 1.1 Hydrophone Deployments & Inventory\n", - "Use deployment dates to pick time ranges that actually contain data before making requests.\n", - "\n", - "In this section we:\n", - "- Pull a full hydrophone inventory (current + history)\n", - "- Select devices and set an example date for the rest of the notebook\n", - "\n", - "### 1.1a Hydrophone Inventory (Current + History)\n", - "Collect all hydrophones, their current deployments, and a history view with location metadata.\n", - "---\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Hydrophone Inventory**\n", - "Pulls deployment metadata for all hydrophones and builds two views: current deployments and full history.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Fetching deployments: 95/95\n" - ] - } - ], - "source": [ - "from onc_hydrophone_data.data.deployment_checker import HydrophoneDeploymentChecker\n", - "\n", - "checker = HydrophoneDeploymentChecker(ONC_TOKEN)\n", - "inventory = checker.collect_hydrophone_inventory()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Table 1: Current Deployments (Active Devices)**\n", - "One row per active device with `device_id`, location metadata, depth/coords, and mapping labels.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
device_codedevice_idlocation_codelocation_namemapped_location_namesmapped_systemsbegin_dateend_datedepth_mlatitudelongitudedeployment_counthistory_starthistory_end
0ICLISTENAF254824348HBIPHartley Bay Underwater NetworkHartley Bay2025-09-11 19:0666.053.4225-129.247522020-03-05 16:39
1ICLISTENAF253429499BACUSUpper Slope SouthBarkley CanyonNC-DDS2024-06-26 06:44393.048.4267-126.174442019-11-26 18:25
2ICLISTENHF125223155DIIPDigby Island Underwater NetworkDigby Island2025-09-12 21:3829.054.2586-130.431172013-09-03 03:18
3ICLISTENHF126623235FGPDFolger DeepFolger PassNC-DDS2020-03-08 14:5395.048.8139-125.274662014-05-07 03:39
4ICLISTENHF133223478PVIPH.H1Saanich Inlet VENUS Instrument PlatformSaanich InletSaanich DDS2024-03-12 20:1292.048.6517-123.487132020-05-27 09:00
5ICLISTENHF135323484PSGCH.H3Strait of Georgia VENUS Instrument PlatformSoG CentralSoG DDS2023-03-17 15:34298.049.0396-123.425322016-06-23 20:45
6ICLISTENHF182226239CBCH.H4ODP 1027CCascadia Basin (ODP 1027)NC-DDS2023-07-14 01:562660.047.7571-127.731142019-09-12 03:15
7ICLISTENHF156124347CCIPChina Creek Underwater NetworkChina Creek2025-06-05 17:25113.049.1539-124.802552018-06-29 20:01
8ICLISTENHF182326259CBCH.H3ODP 1027CCascadia Basin (ODP 1027)NC-DDS2023-07-14 01:562660.047.7571-127.731122020-06-13 16:23
9ICLISTENHF135423485BIIPBurrard Inlet Underwater NetworkBurrard Inlet2025-01-25 20:4124.049.3010-123.112592015-08-30 04:59
10ICLISTENHF156024346PVIPH.H3Saanich Inlet VENUS Instrument PlatformSaanich InletSaanich DDS2024-03-12 20:1292.048.6517-123.487132018-10-01 17:11
11ICLISTENHF195147240CBYIPCambridge Bay Underwater NetworkCambridge Bay2024-08-29 22:3213.069.1125-105.063232020-10-22 21:45
12ICLISTENHF195247241CBCH.H2ODP 1027CCascadia Basin (ODP 1027)NC-DDS2023-07-14 01:562660.047.7571-127.731112023-07-14 01:56
13ICLISTENHF195347260CBCH.H1ODP 1027CCascadia Basin (ODP 1027)NC-DDS2023-07-14 01:562660.047.7571-127.731112023-07-14 01:56
14ICLISTENHF601350260CQSH.H4ODP 1364AClayoquot Slope, ODP 1364A, ODP 889NC-DDS2020-09-14 21:051314.048.6992-126.872412020-09-14 21:05
15ICLISTENHF601650320CQSH.H1ODP 1364AClayoquot Slope, ODP 1364A, ODP 889NC-DDS2020-09-14 21:051314.048.6992-126.872412020-09-14 21:05
16ICLISTENHF601750340CRIPCampbell River Underwater Network2025-08-01 18:498.050.0208-125.235422020-10-01 03:34
17ICLISTENHF601450280CQSH.H3ODP 1364AClayoquot Slope, ODP 1364A, ODP 889NC-DDS2020-09-14 21:051314.048.6992-126.872412020-09-14 21:05
18ICLISTENHF601550300CQSH.H2ODP 1364AClayoquot Slope, ODP 1364A, ODP 889NC-DDS2020-09-14 21:051314.048.6992-126.872412020-09-14 21:05
19ICLISTENHF602150400KVIPKitamaat Village Underwater NetworkKitamaat Village2025-09-10 19:0643.053.9751-128.657122020-10-01 03:34
20ICLISTENHF609250361BACNH.H4Barkley NodeBarkley CanyonNC-DDS2021-02-10 21:11641.048.3452-126.157322020-09-11 21:16
21ICLISTENHF609450440BACNH.H2Barkley NodeBarkley CanyonNC-DDS2021-02-10 21:11641.048.3452-126.157322020-09-11 21:16
22ICLISTENHF609350420BACNH.H3Barkley NodeBarkley CanyonNC-DDS2021-02-10 21:11641.048.3452-126.157322020-09-11 21:16
23ICLISTENHF609550381BACNH.H1Barkley NodeBarkley CanyonNC-DDS2021-02-10 21:11641.048.3452-126.157322020-09-11 21:16
24ICLISTENHF632474140KEMFH.H1Main Endeavour FieldEndeavourNC-DDS2023-09-08 22:562195.047.9493-129.098212023-09-08 22:56
25ICLISTENHF632674180PSGCH.H1Strait of Georgia VENUS Instrument PlatformSoG CentralSoG DDS2023-03-17 15:34298.049.0396-123.425312023-03-17 15:34
26ICLISTENHF632774200KEMFH.H2Main Endeavour FieldEndeavourNC-DDS2023-09-08 22:562195.047.9493-129.098212023-09-08 22:56
27ICLISTENHF632974240KEMFH.H4Main Endeavour FieldEndeavourNC-DDS2023-09-08 22:562195.047.9493-129.098212023-09-08 22:56
28ICLISTENHF632874220KEMFH.H3Main Endeavour FieldEndeavourNC-DDS2023-09-08 22:562195.047.9493-129.098212023-09-08 22:56
29ICLISTENHF707991622HRBIPHolyrood Bay Underwater NetworkHolyrood Bay2025-11-04 14:0085.047.4255-53.121112025-11-04 14:00
30JASCOAMARHYDROPHONED00102243240ECHO3.H2Strait of Georgia EastSoG EastSoG DDS2020-03-04 01:01164.049.0433-123.316112020-03-04 01:01
31JASCOAMARHYDROPHONEE00002943242ECHO3.H4Strait of Georgia EastSoG EastSoG DDS2020-03-04 01:01164.049.0433-123.316112020-03-04 01:01
32JASCOAMARHYDROPHONED00102543241ECHO3.H3Strait of Georgia EastSoG EastSoG DDS2020-03-04 01:01164.049.0433-123.316112020-03-04 01:01
33JASCOAMARHYDROPHONEE00018643260ECHO3.H1Strait of Georgia EastSoG EastSoG DDS2020-03-04 01:01164.049.0433-123.316112020-03-04 01:01
\n", - "
" - ], - "text/plain": [ - " device_code device_id location_code \\\n", - "0 ICLISTENAF2548 24348 HBIP \n", - "1 ICLISTENAF2534 29499 BACUS \n", - "2 ICLISTENHF1252 23155 DIIP \n", - "3 ICLISTENHF1266 23235 FGPD \n", - "4 ICLISTENHF1332 23478 PVIPH.H1 \n", - "5 ICLISTENHF1353 23484 PSGCH.H3 \n", - "6 ICLISTENHF1822 26239 CBCH.H4 \n", - "7 ICLISTENHF1561 24347 CCIP \n", - "8 ICLISTENHF1823 26259 CBCH.H3 \n", - "9 ICLISTENHF1354 23485 BIIP \n", - "10 ICLISTENHF1560 24346 PVIPH.H3 \n", - "11 ICLISTENHF1951 47240 CBYIP \n", - "12 ICLISTENHF1952 47241 CBCH.H2 \n", - "13 ICLISTENHF1953 47260 CBCH.H1 \n", - "14 ICLISTENHF6013 50260 CQSH.H4 \n", - "15 ICLISTENHF6016 50320 CQSH.H1 \n", - "16 ICLISTENHF6017 50340 CRIP \n", - "17 ICLISTENHF6014 50280 CQSH.H3 \n", - "18 ICLISTENHF6015 50300 CQSH.H2 \n", - "19 ICLISTENHF6021 50400 KVIP \n", - "20 ICLISTENHF6092 50361 BACNH.H4 \n", - "21 ICLISTENHF6094 50440 BACNH.H2 \n", - "22 ICLISTENHF6093 50420 BACNH.H3 \n", - "23 ICLISTENHF6095 50381 BACNH.H1 \n", - "24 ICLISTENHF6324 74140 KEMFH.H1 \n", - "25 ICLISTENHF6326 74180 PSGCH.H1 \n", - "26 ICLISTENHF6327 74200 KEMFH.H2 \n", - "27 ICLISTENHF6329 74240 KEMFH.H4 \n", - "28 ICLISTENHF6328 74220 KEMFH.H3 \n", - "29 ICLISTENHF7079 91622 HRBIP \n", - "30 JASCOAMARHYDROPHONED001022 43240 ECHO3.H2 \n", - "31 JASCOAMARHYDROPHONEE000029 43242 ECHO3.H4 \n", - "32 JASCOAMARHYDROPHONED001025 43241 ECHO3.H3 \n", - "33 JASCOAMARHYDROPHONEE000186 43260 ECHO3.H1 \n", - "\n", - " location_name \\\n", - "0 Hartley Bay Underwater Network \n", - "1 Upper Slope South \n", - "2 Digby Island Underwater Network \n", - "3 Folger Deep \n", - "4 Saanich Inlet VENUS Instrument Platform \n", - "5 Strait of Georgia VENUS Instrument Platform \n", - "6 ODP 1027C \n", - "7 China Creek Underwater Network \n", - "8 ODP 1027C \n", - "9 Burrard Inlet Underwater Network \n", - "10 Saanich Inlet VENUS Instrument Platform \n", - "11 Cambridge Bay Underwater Network \n", - "12 ODP 1027C \n", - "13 ODP 1027C \n", - "14 ODP 1364A \n", - "15 ODP 1364A \n", - "16 Campbell River Underwater Network \n", - "17 ODP 1364A \n", - "18 ODP 1364A \n", - "19 Kitamaat Village Underwater Network \n", - "20 Barkley Node \n", - "21 Barkley Node \n", - "22 Barkley Node \n", - "23 Barkley Node \n", - "24 Main Endeavour Field \n", - "25 Strait of Georgia VENUS Instrument Platform \n", - "26 Main Endeavour Field \n", - "27 Main Endeavour Field \n", - "28 Main Endeavour Field \n", - "29 Holyrood Bay Underwater Network \n", - "30 Strait of Georgia East \n", - "31 Strait of Georgia East \n", - "32 Strait of Georgia East \n", - "33 Strait of Georgia East \n", - "\n", - " mapped_location_names mapped_systems begin_date \\\n", - "0 Hartley Bay 2025-09-11 19:06 \n", - "1 Barkley Canyon NC-DDS 2024-06-26 06:44 \n", - "2 Digby Island 2025-09-12 21:38 \n", - "3 Folger Pass NC-DDS 2020-03-08 14:53 \n", - "4 Saanich Inlet Saanich DDS 2024-03-12 20:12 \n", - "5 SoG Central SoG DDS 2023-03-17 15:34 \n", - "6 Cascadia Basin (ODP 1027) NC-DDS 2023-07-14 01:56 \n", - "7 China Creek 2025-06-05 17:25 \n", - "8 Cascadia Basin (ODP 1027) NC-DDS 2023-07-14 01:56 \n", - "9 Burrard Inlet 2025-01-25 20:41 \n", - "10 Saanich Inlet Saanich DDS 2024-03-12 20:12 \n", - "11 Cambridge Bay 2024-08-29 22:32 \n", - "12 Cascadia Basin (ODP 1027) NC-DDS 2023-07-14 01:56 \n", - "13 Cascadia Basin (ODP 1027) NC-DDS 2023-07-14 01:56 \n", - "14 Clayoquot Slope, ODP 1364A, ODP 889 NC-DDS 2020-09-14 21:05 \n", - "15 Clayoquot Slope, ODP 1364A, ODP 889 NC-DDS 2020-09-14 21:05 \n", - "16 2025-08-01 18:49 \n", - "17 Clayoquot Slope, ODP 1364A, ODP 889 NC-DDS 2020-09-14 21:05 \n", - "18 Clayoquot Slope, ODP 1364A, ODP 889 NC-DDS 2020-09-14 21:05 \n", - "19 Kitamaat Village 2025-09-10 19:06 \n", - "20 Barkley Canyon NC-DDS 2021-02-10 21:11 \n", - "21 Barkley Canyon NC-DDS 2021-02-10 21:11 \n", - "22 Barkley Canyon NC-DDS 2021-02-10 21:11 \n", - "23 Barkley Canyon NC-DDS 2021-02-10 21:11 \n", - "24 Endeavour NC-DDS 2023-09-08 22:56 \n", - "25 SoG Central SoG DDS 2023-03-17 15:34 \n", - "26 Endeavour NC-DDS 2023-09-08 22:56 \n", - "27 Endeavour NC-DDS 2023-09-08 22:56 \n", - "28 Endeavour NC-DDS 2023-09-08 22:56 \n", - "29 Holyrood Bay 2025-11-04 14:00 \n", - "30 SoG East SoG DDS 2020-03-04 01:01 \n", - "31 SoG East SoG DDS 2020-03-04 01:01 \n", - "32 SoG East SoG DDS 2020-03-04 01:01 \n", - "33 SoG East SoG DDS 2020-03-04 01:01 \n", - "\n", - " end_date depth_m latitude longitude deployment_count history_start \\\n", - "0 66.0 53.4225 -129.2475 2 2020-03-05 16:39 \n", - "1 393.0 48.4267 -126.1744 4 2019-11-26 18:25 \n", - "2 29.0 54.2586 -130.4311 7 2013-09-03 03:18 \n", - "3 95.0 48.8139 -125.2746 6 2014-05-07 03:39 \n", - "4 92.0 48.6517 -123.4871 3 2020-05-27 09:00 \n", - "5 298.0 49.0396 -123.4253 2 2016-06-23 20:45 \n", - "6 2660.0 47.7571 -127.7311 4 2019-09-12 03:15 \n", - "7 113.0 49.1539 -124.8025 5 2018-06-29 20:01 \n", - "8 2660.0 47.7571 -127.7311 2 2020-06-13 16:23 \n", - "9 24.0 49.3010 -123.1125 9 2015-08-30 04:59 \n", - "10 92.0 48.6517 -123.4871 3 2018-10-01 17:11 \n", - "11 13.0 69.1125 -105.0632 3 2020-10-22 21:45 \n", - "12 2660.0 47.7571 -127.7311 1 2023-07-14 01:56 \n", - "13 2660.0 47.7571 -127.7311 1 2023-07-14 01:56 \n", - "14 1314.0 48.6992 -126.8724 1 2020-09-14 21:05 \n", - "15 1314.0 48.6992 -126.8724 1 2020-09-14 21:05 \n", - "16 8.0 50.0208 -125.2354 2 2020-10-01 03:34 \n", - "17 1314.0 48.6992 -126.8724 1 2020-09-14 21:05 \n", - "18 1314.0 48.6992 -126.8724 1 2020-09-14 21:05 \n", - "19 43.0 53.9751 -128.6571 2 2020-10-01 03:34 \n", - "20 641.0 48.3452 -126.1573 2 2020-09-11 21:16 \n", - "21 641.0 48.3452 -126.1573 2 2020-09-11 21:16 \n", - "22 641.0 48.3452 -126.1573 2 2020-09-11 21:16 \n", - "23 641.0 48.3452 -126.1573 2 2020-09-11 21:16 \n", - "24 2195.0 47.9493 -129.0982 1 2023-09-08 22:56 \n", - "25 298.0 49.0396 -123.4253 1 2023-03-17 15:34 \n", - "26 2195.0 47.9493 -129.0982 1 2023-09-08 22:56 \n", - "27 2195.0 47.9493 -129.0982 1 2023-09-08 22:56 \n", - "28 2195.0 47.9493 -129.0982 1 2023-09-08 22:56 \n", - "29 85.0 47.4255 -53.1211 1 2025-11-04 14:00 \n", - "30 164.0 49.0433 -123.3161 1 2020-03-04 01:01 \n", - "31 164.0 49.0433 -123.3161 1 2020-03-04 01:01 \n", - "32 164.0 49.0433 -123.3161 1 2020-03-04 01:01 \n", - "33 164.0 49.0433 -123.3161 1 2020-03-04 01:01 \n", - "\n", - " history_end \n", - "0 \n", - "1 \n", - "2 \n", - "3 \n", - "4 \n", - "5 \n", - "6 \n", - "7 \n", - "8 \n", - "9 \n", - "10 \n", - "11 \n", - "12 \n", - "13 \n", - "14 \n", - "15 \n", - "16 \n", - "17 \n", - "18 \n", - "19 \n", - "20 \n", - "21 \n", - "22 \n", - "23 \n", - "24 \n", - "25 \n", - "26 \n", - "27 \n", - "28 \n", - "29 \n", - "30 \n", - "31 \n", - "32 \n", - "33 " - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "_ = checker.show_hydrophone_inventory_table(inventory, view='current')\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Table 2: Deployment History (All Deployments)**\n", - "One row per deployment (includes `device_id`). Increase `max_rows` or set it to `None` to show everything.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
device_codedevice_idlocation_codelocation_namemapped_location_namesmapped_systemsbegin_dateend_datedepth_mlatitudelongitudeposition_namelocation_path
0ICHYDROPHONE2031230SGE.H2Strait of Georgia2011-12-11 21:002012-08-09 21:00170.049.0424-123.3171Hydrophone Low FrequencyOcean Networks Canada > Pacific > Salish Sea >...
1ICHYDROPHONE2031230LSHA.H5Ocean Sonics Hydrophone Low Frequency2013-05-03 17:002013-10-22 01:06146.049.0809-123.3405Ocean Networks Canada > Pacific > Salish Sea >...
2ICHYDROPHONE2031230LSHA.H5Ocean Sonics Hydrophone Low Frequency2014-03-08 22:102015-08-28 16:50144.049.0810-123.3403Ocean Networks Canada > Pacific > Salish Sea >...
3ICLISTENAF252323818LSBBLBottom Boundary Layer2016-05-02 15:242017-06-24 08:33141.049.0809-123.3387Ocean Networks Canada > Pacific > Salish Sea >...
4ICLISTENAF252323818FAEFish Acoustics Experiment2017-06-24 09:172017-11-03 18:59147.049.0808-123.3392Ocean Networks Canada > Pacific > Salish Sea >...
5ICLISTENAF252323818FAEFish Acoustics Experiment2017-11-04 18:332018-10-04 23:32144.049.0808-123.3391Ocean Networks Canada > Pacific > Salish Sea >...
6ICLISTENAF252323818CRIPCampbell River Underwater Network2020-07-15 17:102022-01-28 16:007.050.0208-125.2353Ocean Networks Canada > Pacific > Salish Sea >...
7ICLISTENAF252323818USDDLDelta Dynamics Laboratory2023-03-16 14:562024-03-15 17:14101.049.0847-123.3282Ocean Networks Canada > Pacific > Salish Sea >...
8ICHYDROPHONE21623168SEHA.H5Ocean Sonics Hydrophone Low Frequency2013-05-05 17:002013-10-21 22:21170.049.0429-123.3180Ocean Networks Canada > Pacific > Salish Sea >...
9ICLISTENAF250423379NC89.H2Clayoquot Slope2015-09-15 01:092016-05-17 07:221254.048.6714-126.8470Hydrophone Low FrequencyOcean Networks Canada > Pacific > Northeast Pa...
10ICLISTENAF250423379NC27.H4Cascadia BasinODP 10262017-06-19 07:122023-07-12 09:022670.047.7631-127.7585Hydrophone Audio Frequency 2.8 mabOcean Networks Canada > Pacific > Northeast Pa...
11ICLISTENAF254524206HBIPHartley Bay Underwater NetworkHartley Bay2020-10-17 18:152021-10-16 20:2096.053.4221-129.2459Ocean Networks Canada > Pacific > British Colu...
12ICLISTENAF254524206HBIPHartley Bay Underwater NetworkHartley Bay2021-10-16 20:202022-07-14 16:2198.053.4220-129.2462Ocean Networks Canada > Pacific > British Colu...
13ICLISTENAF254524206HBIPHartley Bay Underwater NetworkHartley Bay2023-09-15 23:432025-09-11 19:0365.053.4225-129.2475Ocean Networks Canada > Pacific > British Colu...
14ICLISTENHF120513203FGPDFolger DeepFolger PassNC-DDS2012-06-12 11:132013-05-17 19:1495.048.8138-125.2801Ocean Networks Canada > Pacific > Northeast Pa...
15ICLISTENAF254824348SCVIPStrait of Georgia VENUS Instrument Platform2020-03-05 16:392023-03-17 20:53297.049.0395-123.4254Ocean Networks Canada > Pacific > Salish Sea >...
16ICLISTENAF254824348HBIPHartley Bay Underwater NetworkHartley Bay2025-09-11 19:0666.053.4225-129.2475Ocean Networks Canada > Pacific > British Colu...
17ICLISTENAF252223817NC27.H3Cascadia BasinODP 10262017-06-19 07:122023-07-12 09:022670.047.7631-127.7585Hydrophone Audio Frequency 1.3 mabOcean Networks Canada > Pacific > Northeast Pa...
18ICLISTENAF253429499CCIPChina Creek Underwater NetworkChina Creek2019-11-26 18:252020-03-08 23:50109.049.1543-124.8031Ocean Networks Canada > Pacific > Vancouver Is...
19ICLISTENAF253429499CCIPChina Creek Underwater NetworkChina Creek2020-03-09 01:142021-07-12 20:30107.049.1536-124.8020Ocean Networks Canada > Pacific > Vancouver Is...
\n", - "
" - ], - "text/plain": [ - " device_code device_id location_code \\\n", - "0 ICHYDROPHONE203 1230 SGE.H2 \n", - "1 ICHYDROPHONE203 1230 LSHA.H5 \n", - "2 ICHYDROPHONE203 1230 LSHA.H5 \n", - "3 ICLISTENAF2523 23818 LSBBL \n", - "4 ICLISTENAF2523 23818 FAE \n", - "5 ICLISTENAF2523 23818 FAE \n", - "6 ICLISTENAF2523 23818 CRIP \n", - "7 ICLISTENAF2523 23818 USDDL \n", - "8 ICHYDROPHONE216 23168 SEHA.H5 \n", - "9 ICLISTENAF2504 23379 NC89.H2 \n", - "10 ICLISTENAF2504 23379 NC27.H4 \n", - "11 ICLISTENAF2545 24206 HBIP \n", - "12 ICLISTENAF2545 24206 HBIP \n", - "13 ICLISTENAF2545 24206 HBIP \n", - "14 ICLISTENHF1205 13203 FGPD \n", - "15 ICLISTENAF2548 24348 SCVIP \n", - "16 ICLISTENAF2548 24348 HBIP \n", - "17 ICLISTENAF2522 23817 NC27.H3 \n", - "18 ICLISTENAF2534 29499 CCIP \n", - "19 ICLISTENAF2534 29499 CCIP \n", - "\n", - " location_name mapped_location_names \\\n", - "0 Strait of Georgia \n", - "1 Ocean Sonics Hydrophone Low Frequency \n", - "2 Ocean Sonics Hydrophone Low Frequency \n", - "3 Bottom Boundary Layer \n", - "4 Fish Acoustics Experiment \n", - "5 Fish Acoustics Experiment \n", - "6 Campbell River Underwater Network \n", - "7 Delta Dynamics Laboratory \n", - "8 Ocean Sonics Hydrophone Low Frequency \n", - "9 Clayoquot Slope \n", - "10 Cascadia Basin ODP 1026 \n", - "11 Hartley Bay Underwater Network Hartley Bay \n", - "12 Hartley Bay Underwater Network Hartley Bay \n", - "13 Hartley Bay Underwater Network Hartley Bay \n", - "14 Folger Deep Folger Pass \n", - "15 Strait of Georgia VENUS Instrument Platform \n", - "16 Hartley Bay Underwater Network Hartley Bay \n", - "17 Cascadia Basin ODP 1026 \n", - "18 China Creek Underwater Network China Creek \n", - "19 China Creek Underwater Network China Creek \n", - "\n", - " mapped_systems begin_date end_date depth_m latitude \\\n", - "0 2011-12-11 21:00 2012-08-09 21:00 170.0 49.0424 \n", - "1 2013-05-03 17:00 2013-10-22 01:06 146.0 49.0809 \n", - "2 2014-03-08 22:10 2015-08-28 16:50 144.0 49.0810 \n", - "3 2016-05-02 15:24 2017-06-24 08:33 141.0 49.0809 \n", - "4 2017-06-24 09:17 2017-11-03 18:59 147.0 49.0808 \n", - "5 2017-11-04 18:33 2018-10-04 23:32 144.0 49.0808 \n", - "6 2020-07-15 17:10 2022-01-28 16:00 7.0 50.0208 \n", - "7 2023-03-16 14:56 2024-03-15 17:14 101.0 49.0847 \n", - "8 2013-05-05 17:00 2013-10-21 22:21 170.0 49.0429 \n", - "9 2015-09-15 01:09 2016-05-17 07:22 1254.0 48.6714 \n", - "10 2017-06-19 07:12 2023-07-12 09:02 2670.0 47.7631 \n", - "11 2020-10-17 18:15 2021-10-16 20:20 96.0 53.4221 \n", - "12 2021-10-16 20:20 2022-07-14 16:21 98.0 53.4220 \n", - "13 2023-09-15 23:43 2025-09-11 19:03 65.0 53.4225 \n", - "14 NC-DDS 2012-06-12 11:13 2013-05-17 19:14 95.0 48.8138 \n", - "15 2020-03-05 16:39 2023-03-17 20:53 297.0 49.0395 \n", - "16 2025-09-11 19:06 66.0 53.4225 \n", - "17 2017-06-19 07:12 2023-07-12 09:02 2670.0 47.7631 \n", - "18 2019-11-26 18:25 2020-03-08 23:50 109.0 49.1543 \n", - "19 2020-03-09 01:14 2021-07-12 20:30 107.0 49.1536 \n", - "\n", - " longitude position_name \\\n", - "0 -123.3171 Hydrophone Low Frequency \n", - "1 -123.3405 \n", - "2 -123.3403 \n", - "3 -123.3387 \n", - "4 -123.3392 \n", - "5 -123.3391 \n", - "6 -125.2353 \n", - "7 -123.3282 \n", - "8 -123.3180 \n", - "9 -126.8470 Hydrophone Low Frequency \n", - "10 -127.7585 Hydrophone Audio Frequency 2.8 mab \n", - "11 -129.2459 \n", - "12 -129.2462 \n", - "13 -129.2475 \n", - "14 -125.2801 \n", - "15 -123.4254 \n", - "16 -129.2475 \n", - "17 -127.7585 Hydrophone Audio Frequency 1.3 mab \n", - "18 -124.8031 \n", - "19 -124.8020 \n", - "\n", - " location_path \n", - "0 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "1 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "2 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "3 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "4 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "5 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "6 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "7 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "8 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "9 Ocean Networks Canada > Pacific > Northeast Pa... \n", - "10 Ocean Networks Canada > Pacific > Northeast Pa... \n", - "11 Ocean Networks Canada > Pacific > British Colu... \n", - "12 Ocean Networks Canada > Pacific > British Colu... \n", - "13 Ocean Networks Canada > Pacific > British Colu... \n", - "14 Ocean Networks Canada > Pacific > Northeast Pa... \n", - "15 Ocean Networks Canada > Pacific > Salish Sea >... \n", - "16 Ocean Networks Canada > Pacific > British Colu... \n", - "17 Ocean Networks Canada > Pacific > Northeast Pa... \n", - "18 Ocean Networks Canada > Pacific > Vancouver Is... \n", - "19 Ocean Networks Canada > Pacific > Vancouver Is... " - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "_ = checker.show_hydrophone_inventory_table(inventory, view='history', max_rows=20)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Select target devices**\n", - "Choose device codes or device IDs after reviewing the inventory tables above.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "# Global settings\n", - "# Default device for examples (update to your target device code)\n", - "DEVICE = 'ICLISTENHF6324'\n", - "# Optional second device for multi-device request examples\n", - "# (set to another device code you have access to)\n", - "DEVICE_2 = 'ICLISTENHF1332'" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Table 3: Deployments for Selected Devices**\n", - "Shows full deployment history for the devices you selected (code or ID).\n" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
device_codedevice_idlocation_codelocation_namemapped_location_namesmapped_systemsbegin_dateend_datedepth_mlatitudelongitudeposition_namelocation_path
0ICLISTENHF133223478KVIPKitamaat Village Underwater NetworkKitamaat Village2020-05-27 09:002021-10-15 23:2565.053.9734-128.6572Ocean Networks Canada > Pacific > British Colu...
1ICLISTENHF133223478PVIPH.H1Saanich Inlet VENUS Instrument PlatformSaanich InletSaanich DDS2023-03-14 03:342024-03-12 20:0595.048.6519-123.4869Hydrophone AOcean Networks Canada > Pacific > Salish Sea >...
2ICLISTENHF133223478PVIPH.H1Saanich Inlet VENUS Instrument PlatformSaanich InletSaanich DDS2024-03-12 20:1292.048.6517-123.4871Hydrophone AOcean Networks Canada > Pacific > Salish Sea >...
3ICLISTENHF632474140KEMFH.H1Main Endeavour FieldEndeavourNC-DDS2023-09-08 22:562195.047.9493-129.0982Hydrophone AOcean Networks Canada > Pacific > Northeast Pa...
\n", - "
" - ], - "text/plain": [ - " device_code device_id location_code \\\n", - "0 ICLISTENHF1332 23478 KVIP \n", - "1 ICLISTENHF1332 23478 PVIPH.H1 \n", - "2 ICLISTENHF1332 23478 PVIPH.H1 \n", - "3 ICLISTENHF6324 74140 KEMFH.H1 \n", - "\n", - " location_name mapped_location_names \\\n", - "0 Kitamaat Village Underwater Network Kitamaat Village \n", - "1 Saanich Inlet VENUS Instrument Platform Saanich Inlet \n", - "2 Saanich Inlet VENUS Instrument Platform Saanich Inlet \n", - "3 Main Endeavour Field Endeavour \n", - "\n", - " mapped_systems begin_date end_date depth_m latitude \\\n", - "0 2020-05-27 09:00 2021-10-15 23:25 65.0 53.9734 \n", - "1 Saanich DDS 2023-03-14 03:34 2024-03-12 20:05 95.0 48.6519 \n", - "2 Saanich DDS 2024-03-12 20:12 92.0 48.6517 \n", - "3 NC-DDS 2023-09-08 22:56 2195.0 47.9493 \n", - "\n", - " longitude position_name location_path \n", - "0 -128.6572 Ocean Networks Canada > Pacific > British Colu... \n", - "1 -123.4869 Hydrophone A Ocean Networks Canada > Pacific > Salish Sea >... \n", - "2 -123.4871 Hydrophone A Ocean Networks Canada > Pacific > Salish Sea >... \n", - "3 -129.0982 Hydrophone A Ocean Networks Canada > Pacific > Northeast Pa... " - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "_ = checker.show_device_deployments(device_codes=[DEVICE, DEVICE_2], inventory=inventory)\n", - "# Or filter by numeric device IDs if you have them:\n", - "# _ = checker.show_device_deployments(device_ids=[12345, 67890], inventory=inventory)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 1.1b Interactive Availability Widget\n", - "Explore deployment availability with a visual, zoomable widget. \n", - "\n", - "Tip: leave the dates empty to use the full deployment history, or set a tighter window to speed things up.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "fa57dbd045df4494ac269d9b42e86e86", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "VBox(children=(VBox(children=(HBox(children=(Dropdown(description='Device', index=45, layout=Layout(width='300…" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "42aa6518da774cc6a12a7e35e5484900", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Querying archive: 0it [00:00, ?it/s]" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "4df85a522c604039bf6f3266578b74e8", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Querying archive: 0it [00:00, ?it/s]" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "from onc_hydrophone_data.utils import availability_widget\n", - "\n", - "device_codes = sorted({row['device_code'] for row in inventory['history']})\n", - "availability_widget(\n", - " checker,\n", - " device_codes=device_codes,\n", - " default_device=DEVICE,\n", - " start_date=datetime(2024, 1, 1, tzinfo=timezone.utc),\n", - " end_date=datetime(2024, 3, 1, tzinfo=timezone.utc),\n", - " auto_run=False, # You will need to click the \"Update\" button to query ONC servers andsee the results\n", - ")\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example date used throughout the notebook\n", - "# Choose a time within the deployment ranges shown above\n", - "EXAMPLE_DATE = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "# 2. Download Workflows (Spectrograms / Audio / Both)\n", - "Choose the simplest pattern that matches your goal. Each helper builds the 5-minute request windows and handles batching/parallelism for you.\n", - "\n", - "- Spectrogram downloads pull ONC HSD files (MAT/PNG).\n", - "- To also download matching audio, add `download_audio=True` to any spectrogram call.\n", - "- Audio-only workflows use the `download_audio_*` helpers and fetch FLAC/WAV files.\n", - "\n", - "ONC provides HSD spectrograms in 5-minute windows. For MAT data, you can choose the spectral resolution with `dpo_spectralDataDownsample` in `HSD_OPTIONS`:\n", - "- `1`: one-minute averaged MAT (pre-generated, fast)\n", - "- `2`: spectrogram resolution MAT (on demand, file name includes `_plotRes`)\n", - "- `0`: full resolution MAT (on demand, file name includes `_fullRes`)\n", - "\n", - "The ONC API also exposes data product options you can pass via `data_product_options={...}`:\n", - "\n", - "| Option | DPO key | Values | Notes |\n", - "| --- | --- | --- | --- |\n", - "| Spectral downsample | `dpo_spectralDataDownsample` | `1`, `2`, `0` | 1=pre-generated; 2/0 are on-demand MAT |\n", - "| Diversion mode | `dpo_hydrophoneDataDiversionMode` | `OD`, `LPF`, `HPF`, `All` | Filter by diversion/filters |\n", - "| Acquisition mode | `dpo_hydrophoneAcquisitionMode` | `LF`, `HF`, `All` | Duty-cycle sample rate mode |\n", - "| Spectrogram source | `dpo_spectrogramSource` | `MIX`, `WAV`, `FFT` | PNG/PDF plots only |\n", - "| Concatenation | `dpo_spectrogramConcatenation` | `None`, `Adjacent`, `Daily`, `Weekly`, `Concatenate` | Default: None; non-default downsample disables concat |\n", - "| Colour palette | `dpo_spectrogramColourPalette` | `0`-`5` | PNG/PDF plots only |\n", - "| Upper colour limit | `dpo_upperColourLimit` | `-1000` or `0`-`140` | PNG/PDF plots only |\n", - "| Lower colour limit | `dpo_lowerColourLimit` | `-1000` or `-160`-`140` | PNG/PDF plots only |\n", - "| Upper frequency (preset) | `dpo_spectrogramFrequencyUpperLimit` | `-1`, `1000`, `10000` | PNG/PDF plots only |\n", - "| Upper frequency (explicit) | `dpo_spectrogramUpperFrequencyLimit` | `100`-`500000` | PNG/PDF plots only |\n", - "\n", - "FFT window/overlap are fixed on the ONC side; for custom FFT settings, use the custom spectrogram generation section later in the notebook.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Optional ONC data product options (override defaults).\n", - "# Leave empty to keep defaults; uncomment to customize.\n", - "HSD_OPTIONS = {\n", - " # \"dpo_spectralDataDownsample\": 1, # 1=min avg, 2=plotRes, 0=fullRes\n", - " # \"dpo_hydrophoneDataDiversionMode\": \"OD\", # OD, LPF, HPF, All\n", - " # \"dpo_hydrophoneAcquisitionMode\": \"All\", # LF, HF, All\n", - " # \"dpo_spectrogramSource\": \"MIX\", # PNG/PDF only: MIX, WAV, FFT\n", - " # \"dpo_spectrogramConcatenation\": \"None\", # MAT/PNG/PDF (default)\n", - " # \"dpo_spectrogramColourPalette\": 0, # PNG/PDF only: 0-5\n", - " # \"dpo_upperColourLimit\": -1000, # PNG/PDF only: -1000 or 0-140\n", - " # \"dpo_lowerColourLimit\": -1000, # PNG/PDF only: -1000 or -160-140\n", - " # \"dpo_spectrogramFrequencyUpperLimit\": -1, # PNG/PDF only: -1, 1000, 10000\n", - " # \"dpo_spectrogramUpperFrequencyLimit\": 10000, # PNG/PDF only: 100-500000\n", - "}\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2.1 Basic Spectrogram Download (2 Spectrograms / 10 Minutes)\n", - "Download a short window to validate your setup and directory paths.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download 10 minutes of spectrograms (2 x 5-min windows)\n", - "start = EXAMPLE_DATE\n", - "end = start + timedelta(minutes=10)\n", - "spectrograms_per_batch = 2\n", - "\n", - "info = dl.download_spectrograms_for_range(\n", - " DEVICE,\n", - " start,\n", - " end,\n", - " spectrograms_per_batch,\n", - " tag='basic_download',\n", - " # download_audio=True, # also download matching audio\n", - " # data_product_options=HSD_OPTIONS,\n", - ")\n", - "print(json.dumps(info, indent=2))\n", - "\n", - "plot_first_spectrogram(dl, title=\"Basic download spectrogram\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2.2 Range Downloads (Between Two Dates)\n", - "Download every 5-minute file between two dates. The helper builds the request windows and batches them for you.\n", - "\n", - "### Spectrograms (optional audio)\n", - "Set `spectrograms_per_batch` to control request size.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download ALL spectrograms between two dates, batched by spectrograms_per_batch\n", - "range_start = datetime(2024, 4, 1, 0, 0, tzinfo=timezone.utc)\n", - "range_end = range_start + timedelta(minutes=30) # keep short for tutorial\n", - "spectrograms_per_batch = 3 # number of 5-min spectrograms per request\n", - "\n", - "print(f\"Date range: {range_start} to {range_end}\")\n", - "\n", - "result = dl.download_spectrograms_for_range(\n", - " DEVICE,\n", - " range_start,\n", - " range_end,\n", - " spectrograms_per_batch,\n", - " # download_audio=True,\n", - " # data_product_options=HSD_OPTIONS,\n", - ")\n", - "print(json.dumps(result, indent=2))\n", - "\n", - "plot_first_spectrogram(dl, title=\"Date range spectrogram\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Audio only\n", - "Download all 5-minute audio files that overlap the range (FLAC, with WAV fallback).\n", - "\n", - "This is the audio-only equivalent of the spectrogram range download above.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download audio for a time range\n", - "audio_start = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)\n", - "audio_end = audio_start + timedelta(minutes=10) # 2 files\n", - "\n", - "print(f\"Audio range: {audio_start} to {audio_end}\")\n", - "\n", - "dl.download_audio_for_range(\n", - " DEVICE,\n", - " audio_start,\n", - " audio_end,\n", - ")\n", - "print(f\"Audio saved to: {dl.audio_path}\")\n", - "plot_first_audio(dl, max_seconds=10.0)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2.3 Sampling Mode (Uniform Samples Across Range)\n", - "Sampling selects evenly spaced 5-minute windows across the full date range. This gives a fast, representative overview without downloading everything.\n", - "\n", - "You control:\n", - "- start/end date\n", - "- total samples (number of 5-minute windows)\n", - "- per-request batch size (how many windows per request)\n", - "\n", - "### Spectrograms (optional audio)\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Sample N spectrograms evenly across a date range\n", - "sampling_start = datetime(2024, 4, 1, 0, 0, tzinfo=timezone.utc)\n", - "sampling_end = datetime(2024, 4, 1, 2, 0, tzinfo=timezone.utc) # 2 hours\n", - "total_samples = 4\n", - "spectrograms_per_request = 2\n", - "\n", - "print(f\"Sampling {total_samples} spectrograms from {sampling_start} to {sampling_end}\")\n", - "\n", - "info = dl.download_sampled_spectrograms(\n", - " DEVICE,\n", - " sampling_start,\n", - " sampling_end,\n", - " total_samples,\n", - " spectrograms_per_request,\n", - " # download_audio=True,\n", - " # data_product_options=HSD_OPTIONS,\n", - ")\n", - "print(json.dumps(info, indent=2))\n", - "\n", - "plot_first_spectrogram(dl, title=\"Sampled spectrogram\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Audio only\n", - "Sample evenly spaced 5-minute audio files across the same range.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Sample N audio files evenly across the same date range\n", - "# Reuse sampling_start/sampling_end/total_samples from above\n", - "audio_files_per_request = 2\n", - "\n", - "audio_info = dl.download_sampled_audio(\n", - " DEVICE,\n", - " sampling_start,\n", - " sampling_end,\n", - " total_samples,\n", - " audio_files_per_request,\n", - ")\n", - "\n", - "# json.dumps can't handle datetime objects directly, so serialize them first.\n", - "serializable_info = {\n", - " **audio_info,\n", - " \"start_dt\": audio_info[\"start_dt\"].isoformat(),\n", - " \"end_dt\": audio_info[\"end_dt\"].isoformat(),\n", - " \"request_windows\": [\n", - " (start.isoformat(), end.isoformat())\n", - " for start, end in audio_info[\"request_windows\"]\n", - " ],\n", - "}\n", - "print(json.dumps(serializable_info, indent=2))\n", - "\n", - "plot_first_audio(dl, max_seconds=10.0)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2.4 Event-Based Downloads (Simple)\n", - "Provide event timestamps and let the helper map each one to its containing 5-minute window. This keeps the call minimal.\n", - "\n", - "If you need padding, clipping, JSON/CSV files, or per-event overrides, jump to Section 4 (Event Requests & Request Files).\n", - "\n", - "### Spectrograms (optional audio)\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download spectrograms for event timestamps (mapped to 5-minute windows)\n", - "event_times = [\n", - " datetime(2024, 4, 1, 4, 25, 30, tzinfo=timezone.utc),\n", - " datetime(2024, 4, 1, 14, 10, 15, tzinfo=timezone.utc),\n", - " datetime(2024, 4, 2, 3, 45, 0, tzinfo=timezone.utc),\n", - "]\n", - "\n", - "spectrograms_per_request = 1\n", - "\n", - "info = dl.download_spectrograms_for_events(\n", - " DEVICE,\n", - " event_times,\n", - " spectrograms_per_request,\n", - " tag='event_times',\n", - " # download_audio=True,\n", - " # data_product_options=HSD_OPTIONS,\n", - ")\n", - "print(json.dumps(info, indent=2))\n", - "\n", - "plot_first_spectrogram(dl, title=\"Event-based spectrogram\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Audio only\n", - "Download the 5-minute audio files that contain each event timestamp.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download audio for the same event timestamps\n", - "# Reuse event_times from the spectrogram example above.\n", - "\n", - "dl.download_audio_for_events(DEVICE, event_times)\n", - "plot_first_audio(dl, max_seconds=10.0)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2.5 Centered Audio Clip (Custom Duration)\n", - "Compute which 5-minute files are needed for a shorter clip, then download them in one call.\n", - "\n", - "Use `describe_audio_window` to see which files are needed, and `download_audio_for_center_time` to fetch them.\n", - "This example intentionally crosses a 5-minute boundary, so two adjacent audio files are required before clipping to the exact duration.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download 30 seconds of audio centered on a timestamp near a 5-minute boundary\n", - "center_time = datetime(2024, 4, 1, 12, 34, 50, tzinfo=timezone.utc)\n", - "duration_seconds = 30\n", - "\n", - "window = dl.describe_audio_window(center_time, duration_seconds)\n", - "\n", - "dl.download_audio_for_center_time(DEVICE, center_time, duration_seconds)\n", - "print(\"Files downloaded - use audio utils to stitch and clip\")\n", - "\n", - "plot_first_audio(dl, max_seconds=10.0)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "# 3. Custom Spectrogram Generation (Local)\n", - "Generate spectrograms locally when you need full control over parameters and FFT settings.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3.1 SpectrogramGenerator Basics\n", - "Minimal setup for producing custom spectrograms from audio files.\n", - "\n", - "Defaults work well for most cases; the example below narrows the frequency range for readability. The next section lists every tunable parameter.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from onc_hydrophone_data.audio import SpectrogramGenerator\n", - "\n", - "# Create generator with default settings\n", - "generator = SpectrogramGenerator(\n", - " win_dur=1.0, # 1 second FFT window\n", - " overlap=0.5, # 50% overlap\n", - " window_type='hann', # Window function\n", - " freq_lims=(10, 1000) # 10 Hz to 1 kHz\n", - ")\n", - "\n", - "print(\"SpectrogramGenerator created with:\")\n", - "print(f\" Window: {generator.win_dur}s\")\n", - "print(f\" Overlap: {generator.overlap}\")\n", - "print(f\" Window type: {generator.window_type}\")\n", - "print(f\" Freq range: {generator.freq_lims}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3.2 Custom Parameters\n", - "Every SpectrogramGenerator argument is optional; below are the tunable knobs and what they do.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Full customization example (all arguments are optional)\n", - "custom = SpectrogramGenerator(\n", - " win_dur=0.5,\n", - " overlap=0.75,\n", - " window_type=('kaiser', 14.0),\n", - " nfft=None,\n", - " win_length=None,\n", - " hop_length=None,\n", - " freq_lims=(10, 24000),\n", - " colormap='magma',\n", - " clim=(-80, 0),\n", - " log_freq=True,\n", - " max_duration=120.0,\n", - " clip_start=5.0,\n", - " clip_end=65.0,\n", - " backend='auto',\n", - " scaling='density',\n", - " quiet=False,\n", - " use_logging=True,\n", - ")\n", - "\n", - "# Presets for common use cases\n", - "high_res = SpectrogramGenerator(\n", - " win_dur=0.1, # 100ms window (higher time resolution)\n", - " overlap=0.9, # 90% overlap\n", - " window_type='hann',\n", - " freq_lims=(1, 24000), # Full frequency range\n", - " clim=(-80, 0) # dB scale limits\n", - ")\n", - "\n", - "low_freq = SpectrogramGenerator(\n", - " win_dur=2.0, # 2s window (better freq resolution)\n", - " overlap=0.5,\n", - " window_type=('kaiser', 8.0),\n", - " freq_lims=(10, 200), # Focus on low frequencies\n", - ")\n", - "\n", - "print(\"Custom config window type:\", custom.window_type)\n", - "print(\"High-res config:\", high_res.win_dur, high_res.freq_lims)\n", - "print(\"Low-freq config:\", low_freq.win_dur, low_freq.freq_lims)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Example: exact-duration trimming with extra context for the STFT.\n", - "\n", - "By default, `clip_pad_seconds` uses `auto` (half the window length) to reduce edge artifacts. You can override it with an explicit value if needed.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Exact-duration trimming with STFT context\n", - "from IPython.display import display\n", - "\n", - "target_seconds = 30.0\n", - "clip_start = 10.0\n", - "clip_end = clip_start + target_seconds\n", - "audio_path = find_first_file(dl.audio_path, [\"*.flac\", \"*.wav\"])\n", - "\n", - "if audio_path:\n", - " local_gen = SpectrogramGenerator(\n", - " win_dur=0.5,\n", - " overlap=0.5,\n", - " window_type='hann',\n", - " freq_lims=(10, 1000),\n", - " clip_start=clip_start,\n", - " clip_end=clip_end,\n", - " quiet=True,\n", - " )\n", - " # clip_pad_seconds defaults to 'auto' (half-window); set it if you want more context.\n", - " audio_data, sr, clip_meta = local_gen.load_audio(audio_path)\n", - " freqs, times, _, db = local_gen.compute_spectrogram(\n", - " audio_data,\n", - " sr,\n", - " clip_meta=clip_meta,\n", - " )\n", - " fig = local_gen.plot_spectrogram(\n", - " freqs,\n", - " times,\n", - " db,\n", - " title=f\"Local spectrogram {target_seconds:.1f}s (trimmed after STFT)\",\n", - " )\n", - " display(fig)\n", - "else:\n", - " print(\"No audio file found; run a download cell first.\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3.3 Batch Processing Audio Directory\n", - "Process an entire directory of audio files in one call.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Process all audio files in a directory\n", - "# Uses the most recent audio download path (run an audio download above).\n", - "audio_dir = Path(dl.audio_path)\n", - "output_dir = audio_dir.parent / \"custom_spectrograms\"\n", - "\n", - "if audio_dir.exists():\n", - " results = generator.process_directory(\n", - " input_dir=audio_dir,\n", - " save_dir=output_dir,\n", - " save_plot=True, # Save PNG plots\n", - " save_mat=True, # Save MAT files\n", - " )\n", - " print(f\"Processed {len(results)} files\")\n", - "else:\n", - " print(f\"No audio files at {audio_dir}\")\n", - " print(\"Run audio downloads first, then process them here (or set audio_dir manually)\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "# 4. Event Requests & Request Files\n", - "Use this section when you need padding, clipping, or per-event overrides. The simple event-based download in Section 2.4 just maps timestamps to 5-minute windows.\n", - "\n", - "For downloading data around specific events/annotations.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4.1 Direct Timestamps (Python Lists / Datetime Objects)\n", - "When you have event times, ONC spectrograms are stored in 5-minute blocks. Map each event time to the surrounding 5-minute window (e.g., 04:27 -> 04:25-04:30) and request those windows. The example below passes those explicit start/end windows directly so you control exactly what gets downloaded.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Explicit 5-minute windows (e.g., derived from event timestamps)\n", - "explicit_windows = {\n", - " DEVICE: [\n", - " (datetime(2024, 4, 1, 4, 25, tzinfo=timezone.utc), \n", - " datetime(2024, 4, 1, 4, 30, tzinfo=timezone.utc)),\n", - " (datetime(2024, 4, 1, 14, 0, tzinfo=timezone.utc), \n", - " datetime(2024, 4, 1, 14, 5, tzinfo=timezone.utc)),\n", - " (datetime(2024, 4, 2, 3, 15, tzinfo=timezone.utc), \n", - " datetime(2024, 4, 2, 3, 20, tzinfo=timezone.utc)),\n", - " ]\n", - "}\n", - "\n", - "print(f\"Downloading {len(explicit_windows[DEVICE])} specific time windows\")\n", - "\n", - "info = dl.download_spectrogram_windows(\n", - " DEVICE,\n", - " explicit_windows,\n", - " spectrograms_per_request=1,\n", - " tag='explicit_times',\n", - ")\n", - "print(json.dumps(info, indent=2))\n", - "\n", - "plot_first_spectrogram(dl, title=\"Explicit window spectrogram\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Example formats (datetime objects and legacy tuples):\n", - "Use these directly in Python, or convert them into the request schema below.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Python datetime objects\n", - "timestamps_datetime = [\n", - " datetime(2024, 4, 1, 12, 30, 0, tzinfo=timezone.utc),\n", - " datetime(2024, 4, 1, 14, 45, 30, tzinfo=timezone.utc),\n", - " datetime(2024, 4, 2, 8, 15, 0, tzinfo=timezone.utc),\n", - "]\n", - "\n", - "# Tuple format (legacy)\n", - "timestamps_tuple = [\n", - " [2024, 4, 1, 12, 30, 0],\n", - " [2024, 4, 1, 14, 45, 30],\n", - "]\n", - "\n", - "print(\"Datetime list:\", timestamps_datetime[:2])\n", - "print(\"Tuple list:\", timestamps_tuple)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4.2 Request Files (JSON + CSV)\n", - "Use request files when you have many events or want per-request overrides in one batch.\n", - "Examples below run the request file as-is; pass keyword args to override JSON defaults if needed.\n", - "\n", - "\n", - "**Request schema (JSON/CSV)**\n", - "\n", - "JSON uses a `{defaults, requests}` payload. CSV is a flat table with the same fields (one row per request). Required per request: `deviceCode` and either `timestamp` *or* a `start`/`end` window.\n", - "You can set `timezone` in `defaults` or per-request; it applies to naive timestamps.\n", - "You can mix multiple devices in one file by setting `deviceCode` per request (JSON) or per row (CSV).\n", - "\n", - "JSON format:\n", - "```\n", - "{\n", - " \"defaults\": { ... },\n", - " \"requests\": [ { ... }, { ... } ]\n", - "}\n", - "```\n", - "\n", - "| Field | Type | Required | Notes |\n", - "| --- | --- | --- | --- |\n", - "| `deviceCode` | string | yes | Hydrophone device code (e.g., `ICLISTENHF6324`) |\n", - "| `timestamp` | string | if no `start`/`end` | ISO 8601 timestamp (UTC or offset, e.g., `2024-04-01T12:30:00Z`) |\n", - "| `timezone` | string | no | Timezone for naive timestamps (e.g., `America/Vancouver`, `UTC`, `-07:00`) |\n", - "| `start` | string | if no `timestamp` | ISO 8601 timestamp (UTC or offset) |\n", - "| `end` | string | no | ISO 8601 timestamp (UTC or offset) |\n", - "| `duration_seconds` | number | no | Used when `start` is set but `end` is omitted |\n", - "| `pad_seconds` | number | no | Symmetric padding around `timestamp` or `start`/`end` |\n", - "| `pad_before_seconds` | number | no | Override padding before |\n", - "| `pad_after_seconds` | number | no | Override padding after |\n", - "| `download_audio` | bool | no | Download audio files (default: false) |\n", - "| `download_spectrogram` | bool | no | Download ONC spectrograms (default: true) |\n", - "| `spectrogram_format` | string | no | `mat` or `png` |\n", - "| `clip` | bool | no | Clip outputs to the padded window |\n", - "| `audio_extension` | string | no | `flac` or `wav` |\n", - "| `output_tag` | string | no | Output folder tag |\n", - "| `output_name` | string | no | Override clip basename |\n", - "| `label` / `description` | string | no | Metadata label |\n", - "| `data_product_options` | object | no | ONC `dpo_*` options (same as `HSD_OPTIONS`) |\n", - "\n", - "Tip: In CSV, use a `deviceCode` column to match the JSON field name. For `data_product_options`, store a JSON string per row and parse it in Python.\n", - "\n", - "**Padding + clipping behavior**\n", - "Each request expands to the 5-minute coverage windows needed for downloads. If padding crosses a window boundary, the downloader fetches adjacent files.\n", - "\n", - "- Audio clips are trimmed to the exact padded interval.\n", - "- ONC spectrogram clips are trimmed to the nearest time-bin boundaries on the fixed 5-minute grid, so the spectrogram duration can differ slightly from the audio (up to ~one bin).\n", - "\n", - "The bin width is `300s / num_time_bins` for each 5-minute file and is stored as `seconds_per_column` in the clip metadata.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 4.2a JSON Example + Execution\n", - "Write a request file, then execute it to download audio and/or ONC spectrograms.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# New JSON format with defaults\n", - "json_new = {\n", - " \"defaults\": {\n", - " \"deviceCode\": \"ICLISTENHF6324\",\n", - " \"pad_seconds\": 30,\n", - " \"data_product_options\": HSD_OPTIONS\n", - " },\n", - " \"requests\": [\n", - " {\"timestamp\": \"2024-04-01T12:30:00Z\"},\n", - " {\"start\": \"2024-04-01T14:00:00Z\", \"end\": \"2024-04-01T14:05:00Z\"}\n", - " ]\n", - "}\n", - "\n", - "# Legacy JSON format\n", - "json_legacy = {\n", - " \"ICLISTENHF6324\": [\n", - " [2024, 4, 1, 12, 30, 0],\n", - " [2024, 4, 1, 14, 45, 30]\n", - " ]\n", - "}\n", - "\n", - "print(\"New format:\", json.dumps(json_new, indent=2)[:200] + \"...\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# New JSON format with defaults\n", - "# Example includes multiple devices (set DEVICE_2 to a different device to test)\n", - "json_requests = {\n", - " \"defaults\": {\n", - " \"pad_seconds\": 15,\n", - " \"download_audio\": True,\n", - " \"clip\": True,\n", - " \"data_product_options\": HSD_OPTIONS\n", - " },\n", - " \"requests\": [\n", - " {\n", - " \"deviceCode\": DEVICE,\n", - " \"timestamp\": \"2024-04-01T12:34:50Z\",\n", - " \"label\": \"whale call 1\"\n", - " },\n", - " {\n", - " \"deviceCode\": DEVICE_2,\n", - " \"start\": \"2024-04-01T12:30:00Z\",\n", - " \"end\": \"2024-04-01T12:33:30Z\",\n", - " \"pad_before_seconds\": 10,\n", - " \"pad_after_seconds\": 20,\n", - " \"label\": \"ship noise event\"\n", - " }\n", - " ]\n", - "}\n", - "json_path = Path(DATA_DIR) / \"example_requests.json\"\n", - "json_path.write_text(json.dumps(json_requests, indent=2))\n", - "print(f\"Saved requests to: {json_path}\")\n", - "print(json.dumps(json_requests, indent=2))\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "json_path = Path(DATA_DIR) / \"example_requests.json\"\n", - "# Execute JSON requests (uses settings from the JSON file)\n", - "results = dl.download_requests_from_json(\n", - " str(json_path),\n", - ")\n", - "print(json.dumps(results, indent=2))\n", - "\n", - "plot_request_results(results, downloader=dl)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 4.2b CSV Example + Execution\n", - "CSV can be executed directly with `download_requests_from_csv` (no pandas required).\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example CSV content (data_product_options is JSON per row; use double quotes)\n", - "# Example includes multiple devices (set DEVICE_2 to a different device to test)\n", - "csv_content = f\"\"\"deviceCode,timestamp,label,data_product_options\n", - "{DEVICE},2024-04-01T12:30:00Z,whale call,\"{{\"\"dpo_spectralDataDownsample\"\": 2}}\"\n", - "{DEVICE_2},2024-04-15T14:45:30Z,ship noise,\"{{\"\"dpo_spectralDataDownsample\"\": 1}}\"\n", - "{DEVICE},2024-04-02T08:15:00Z,unknown,\"\"\n", - "\"\"\"\n", - "\n", - "csv_path = Path(DATA_DIR) / \"example_requests.csv\"\n", - "csv_path.write_text(csv_content)\n", - "print(f\"Saved CSV to: {csv_path}\")\n", - "\n", - "# Execute CSV requests (uses settings from the CSV file)\n", - "results = dl.download_requests_from_csv(\n", - " str(csv_path),\n", - ")\n", - "print(json.dumps(results, indent=2))\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4.3 Supported Date/Time Formats\n", - "These formats are parsed consistently by the downloader utilities and are accepted anywhere a timestamp is expected.\n", - "All inputs are converted to UTC. If you use naive datetimes or strings, set a `timezone` in request files or pass a tz-aware datetime.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# All supported input formats\n", - "from onc_hydrophone_data.data.hydrophone_downloader import HydrophoneDownloader\n", - "\n", - "formats = [\n", - " datetime(2024, 4, 1, 12, 30, tzinfo=timezone.utc), # datetime object\n", - " \"2024-04-01T12:30:00Z\", # ISO 8601\n", - " \"2024-04-01T12:30:00.000Z\", # ISO with ms\n", - " \"2024-04-01T12:30:00-07:00\", # ISO with offset\n", - " [2024, 4, 1, 12, 30, 0], # list\n", - " (2024, 4, 1, 12, 30, 0), # tuple\n", - "]\n", - "\n", - "print(\"All these formats are parsed correctly:\")\n", - "for f in formats:\n", - " parsed = HydrophoneDownloader._parse_timestamp_value(f)\n", - " print(f\" {type(f).__name__:10} → {parsed}\")\n", - "\n", - "print(\"Timezone override for naive strings:\")\n", - "local_parsed = HydrophoneDownloader._parse_timestamp_value(\n", - " \"2024-04-01 12:30:00\",\n", - " timezone_str=\"America/Vancouver\",\n", - ")\n", - "print(f\" America/Vancouver → {local_parsed}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 5. End-to-End Pipelines\n", - "End-to-end examples that combine download, processing, and multi-device usage.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5.1 Request-Driven Audio Downloads + Local Spectrograms (JSON/CSV)\n", - "Request files can drive custom spectrograms by downloading audio clips first and then running `SpectrogramGenerator` locally.\n", - "This helper automatically grabs extra audio context (`clip_pad_seconds`, default `auto`) so the STFT has padding and edge artifacts are reduced.\n", - "Use `generator_defaults` (applies to all requests) and optional per-request `generator_options` for settings like `freq_lims`.\n", - "When `freq_lims` are provided, the saved outputs are cropped to that range (set `\"crop_freq_lims\": false` to keep full-band saves).\n", - "Control outputs with `save_png`, `save_mat`, and `save_npy` (PNG defaults to off; MAT defaults to on).\n", - "Saved MAT/NPY files include metadata describing the generator settings, FFT params, and clip context.\n", - "The saved audio clip matches the requested window length; extra context is used internally for spectrogram generation.\n", - "Use `save_context_audio=True` if you want to keep the longer context clip alongside the final trimmed clip.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "custom_json_path = Path(DATA_DIR) / \"custom_spectrogram_requests.json\"\n", - "\n", - "custom_requests = {\n", - " \"defaults\": {\n", - " \"deviceCode\": DEVICE,\n", - " \"pad_seconds\": 15,\n", - " \"label\": \"custom_clip\",\n", - " },\n", - " \"generator_defaults\": {\n", - " \"win_dur\": 0.5,\n", - " \"overlap\": 0.5,\n", - " \"window_type\": \"hann\",\n", - " \"quiet\": True,\n", - " },\n", - " \"requests\": [\n", - " {\n", - " \"timestamp\": \"2024-04-01T12:30:00Z\",\n", - " \"label\": \"early_april\",\n", - " \"generator_options\": {\"freq_lims\": [10, 500]},\n", - " },\n", - " {\n", - " \"timestamp\": \"2024-04-15T12:30:00Z\",\n", - " \"label\": \"mid_april\",\n", - " \"generator_options\": {\"freq_lims\": [10, 2000]},\n", - " },\n", - " {\n", - " \"timestamp\": \"2024-04-30T12:30:00Z\",\n", - " \"label\": \"late_april\",\n", - " \"generator_options\": {\"freq_lims\": [10, 10000]},\n", - " },\n", - " ],\n", - "}\n", - "\n", - "custom_json_path.write_text(json.dumps(custom_requests, indent=2))\n", - "\n", - "custom_results = dl.create_custom_spectrograms_from_json(\n", - " str(custom_json_path),\n", - " save_png=False,\n", - " save_mat=True,\n", - " save_npy=False,\n", - ")\n", - "print(json.dumps(custom_results, indent=2))\n", - "\n", - "for result in custom_results:\n", - " custom_spec = result.get(\"custom_spectrogram\") or {}\n", - " mat_file = custom_spec.get(\"mat_file\")\n", - " if not mat_file:\n", - " continue\n", - " timestamp = result.get(\"timestamp\", \"\")\n", - " plot_onc_mat_spectrogram(\n", - " mat_file,\n", - " title=f\"Custom spectrogram ({timestamp})\",\n", - " freq_lims=custom_spec.get(\"freq_lims\"),\n", - " log_freq=custom_spec.get(\"log_freq\", True),\n", - " )\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Inspect the first saved spectrogram file\n", - "import scipy.io\n", - "\n", - "first_mat = None\n", - "first_npy = None\n", - "for result in custom_results:\n", - " custom_spec = result.get(\"custom_spectrogram\") or {}\n", - " first_mat = first_mat or custom_spec.get(\"mat_file\")\n", - " first_npy = first_npy or custom_spec.get(\"npy_file\")\n", - " if first_mat or first_npy:\n", - " break\n", - "\n", - "if first_mat:\n", - " mat = scipy.io.loadmat(first_mat)\n", - " keys = sorted(k for k in mat.keys() if not k.startswith(\"__\"))\n", - " print(f\"MAT keys: {keys}\")\n", - " for key in keys:\n", - " value = mat[key]\n", - " if hasattr(value, \"shape\"):\n", - " print(f\" {key}: shape={value.shape}, dtype={getattr(value, 'dtype', None)}\")\n", - " else:\n", - " print(f\" {key}: type={type(value).__name__}\")\n", - " meta_json = mat.get(\"metadata_json\")\n", - " if meta_json is not None:\n", - " try:\n", - " meta_text = meta_json.item()\n", - " except Exception:\n", - " meta_text = str(meta_json)\n", - " try:\n", - " meta = json.loads(meta_text)\n", - " print(f\"metadata_json keys: {sorted(meta.keys())}\")\n", - " except json.JSONDecodeError:\n", - " print(\"metadata_json: \")\n", - "elif first_npy:\n", - " data = np.load(first_npy, allow_pickle=True).item()\n", - " print(f\"NPY keys: {sorted(data.keys())}\")\n", - " metadata = data.get(\"metadata\")\n", - " if isinstance(metadata, dict):\n", - " print(f\"metadata keys: {sorted(metadata.keys())}\")\n", - "else:\n", - " print(\"No saved spectrogram files found to inspect.\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5.2 Batch Pipeline: Download Audio → Local Spectrograms\n", - "Download audio, then generate custom spectrograms for analysis.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Complete workflow: Download audio + generate custom spectrograms\n", - "pipeline_start = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)\n", - "pipeline_end = pipeline_start + timedelta(minutes=10)\n", - "\n", - "# Step 1: Download audio\n", - "dl.download_audio_for_range(DEVICE, pipeline_start, pipeline_end, tag=\"pipeline_demo\")\n", - "print(f\"1. Audio downloaded to: {dl.audio_path}\")\n", - "\n", - "plot_first_audio(dl, max_seconds=10.0)\n", - "\n", - "# Step 2: Generate custom spectrograms\n", - "custom_out = Path(dl.audio_path).parent / \"custom_spectrograms\"\n", - "results = generator.process_directory(\n", - " input_dir=dl.audio_path,\n", - " save_dir=custom_out,\n", - ")\n", - "print(f\"2. Custom spectrograms saved to: {custom_out}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5.3 Multi-Device Downloads\n", - "Repeat the same download pattern across multiple hydrophones.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download from multiple hydrophones\n", - "# Keep the range short for a quick multi-device check.\n", - "devices = ['ICLISTENHF6324', 'ICLISTENHF6020', 'ICLISTENHF6019']\n", - "multi_start = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)\n", - "multi_end = multi_start + timedelta(minutes=15)\n", - "\n", - "for device in devices:\n", - " windows = {device: [(multi_start, multi_end)]}\n", - " print(f\"Device {device}: {multi_start} to {multi_end}\")\n", - "\n", - " info = dl.download_spectrogram_windows(\n", - " device,\n", - " windows,\n", - " spectrograms_per_request=3,\n", - " tag='multi_device',\n", - " )\n", - " print(f\" Downloaded: {info.get('runs_downloaded', 0)} runs\")\n", - "\n", - "plot_first_spectrogram(dl, title=f\"Multi-device spectrogram ({devices[-1]})\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 6. Output Folder Structure\n", - "Overview of where downloads and generated files are stored.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# View current paths for a known tag/date range\n", - "example_tag = 'basic_download'\n", - "example_start = EXAMPLE_DATE\n", - "example_end = example_start + timedelta(minutes=10)\n", - "\n", - "dl.setup_directories('mat', DEVICE, example_tag, example_start, example_end)\n", - "print(\"Output paths:\")\n", - "print(f\" Spectrograms: {dl.spectrogram_path}\")\n", - "print(f\" Audio: {dl.audio_path}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 7. Troubleshooting & Tips\n", - "Common issues and ways to speed up or stabilize downloads.\n", - "\n", - "## Common Issues\n", - "\n", - "| Issue | Solution |\n", - "| --- | --- |\n", - "| \"Missing ONC_TOKEN\" | Add `ONC_TOKEN=...` to `.env` in the repo root |\n", - "| \"Device not deployed\" | Run Section 1.1 (HydrophoneDeploymentChecker) and choose dates within deployment |\n", - "| \"Waiting on file system\" | Normal - ONC is generating data; wait and retry |\n", - "| Timeout errors | Reduce request size or increase wait time (`max_wait_minutes`) |\n", - "| Rate limiting | Reduce request size or add delays between runs |\n", - "\n", - "## Performance Tips\n", - "\n", - "1. **Batch requests**: Group spectrograms into 6-12 per request\n", - "2. **Avoid full resolution**: `downsample=0` is much slower\n", - "3. **Check archive first**: Archived data downloads instantly\n", - "4. **Prefer shorter ranges**: Large ranges are easier to handle in chunks\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": ".venv", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.12" - } - }, - "nbformat": 4, - "nbformat_minor": 4 + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# ONC Hydrophone Audio and Spectrogram Tutorial\n", + "\n", + "## Outline\n", + "\n", + "1. [Set up and choose a deployed hydrophone](#1-introduction-and-setup)\n", + "2. [Download audio and make a local spectrogram](#2-download-audio-and-onc-products)\n", + "3. [Tune local spectrograms and process known events safely](#3-generate-local-spectrograms)\n", + "4. [Download ONC products with Python, JSON, or CSV](#4-download-onc-products-and-request-files)\n", + "5. [Create your own event spectrograms from JSON](#5-local-json-and-end-to-end-workflows)\n", + "6. [Inspect output folders](#6-output-folder-structure)\n", + "7. [Troubleshoot and practise](#7-troubleshooting-practice-and-next-steps)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 1. Introduction and setup\n", + "\n", + "This tutorial is for researchers and analysts who are new to ONC hydrophone data. It leads with downloading audio and computing spectrograms locally because that is the most common workflow, then introduces ONC-generated spectrogram products as another valid method.\n", + "\n", + "By the end, you will be able to:\n", + "\n", + "- find deployment dates for a hydrophone;\n", + "- download a short FLAC/WAV range;\n", + "- generate and display a real spectrogram from that audio;\n", + "- generate an edge-safe spectrogram around a known signal time;\n", + "- use JSON either to create local spectrograms or to download ONC products; and\n", + "- sample longer ranges without downloading every file.\n", + "\n", + "## Two different spectrogram workflows\n", + "\n", + "| Goal | Method | Who computes the spectrogram? |\n", + "| --- | --- | --- |\n", + "| Control the FFT, frequency range, and event padding | Download `HAF` audio, then use `SpectrogramGenerator` | This package, locally |\n", + "| Retrieve a standard ONC `HSD` MAT/PNG product | Use a `download_spectrograms_*` method or `download_requests_from_json()` | ONC servers |\n", + "\n", + "## Prerequisites\n", + "\n", + "- Install the package and notebook dependencies.\n", + "- Create a repo-root `.env` containing `ONC_TOKEN=...`. Never place the token in this notebook.\n", + "- Optionally set `DATA_DIR=/path/to/data`; otherwise downloads use `data/`.\n", + "- Use timezone-aware timestamps. Every request is converted to UTC.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Standard imports\n", + "import sys\n", + "import json\n", + "import numpy as np\n", + "from pathlib import Path\n", + "from datetime import datetime, timedelta, timezone\n", + "from IPython.display import Image, display\n", + "\n", + "# Ensure repo is in path\n", + "REPO_ROOT = Path(\"..\").resolve()\n", + "if str(REPO_ROOT) not in sys.path:\n", + " sys.path.append(str(REPO_ROOT))\n", + "\n", + "# Core imports\n", + "from onc_hydrophone_data.onc.common import load_config\n", + "from onc_hydrophone_data.data import HydrophoneDownloader\n", + "\n", + "from onc_hydrophone_data.utils.plotting import (\n", + " find_first_file,\n", + " plot_first_spectrogram,\n", + " plot_first_audio,\n", + " plot_request_results,\n", + ")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Load configuration\n", + "ONC_TOKEN, DATA_DIR = load_config()\n", + "dl = HydrophoneDownloader(ONC_TOKEN, DATA_DIR)\n", + "print(f\"✅ Data directory: {DATA_DIR}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## 1.1 Hydrophone Deployments & Inventory\n", + "Use deployment dates to pick time ranges that actually contain data before making requests.\n", + "\n", + "In this section we:\n", + "- Pull a full hydrophone inventory (current + history)\n", + "- Select devices and set an example date for the rest of the notebook\n", + "\n", + "### 1.1a Hydrophone Inventory (Current + History)\n", + "Collect all hydrophones, their current deployments, and a history view with location metadata. The public reference is the [ONC Hydrophone Location Codes & Data Types page](https://wiki.oceannetworks.ca/spaces/O2KB/pages/72548584/ONC+Hydrophone+Location+Codes+Data+Types).\n", + "---\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Hydrophone Inventory**\n", + "Pulls deployment metadata for all hydrophones and builds two views: current deployments and full history.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from onc_hydrophone_data.data.deployment_checker import HydrophoneDeploymentChecker\n", + "\n", + "checker = HydrophoneDeploymentChecker(ONC_TOKEN)\n", + "inventory = checker.collect_hydrophone_inventory()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Table 1: Current Deployments (Active Devices)**\n", + "One row per active device with `device_id`, location metadata, depth/coords, and mapping labels.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "_ = checker.show_hydrophone_inventory_table(inventory, view='current')\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Table 2: Deployment History (All Deployments)**\n", + "One row per deployment (includes `device_id`). Increase `max_rows` or set it to `None` to show everything.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "_ = checker.show_hydrophone_inventory_table(inventory, view='history', max_rows=20)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Select target devices**\n", + "Choose device codes or device IDs after reviewing the inventory tables above.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Global settings\n", + "# Default device for examples (update to your target device code)\n", + "DEVICE = 'ICLISTENHF6324'\n", + "# Optional second device for multi-device request examples\n", + "# (set to another device code you have access to)\n", + "DEVICE_2 = 'ICLISTENHF1332'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Table 3: Deployments for Selected Devices**\n", + "Shows full deployment history for the devices you selected (code or ID).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "_ = checker.show_device_deployments(device_codes=[DEVICE, DEVICE_2], inventory=inventory)\n", + "# Or filter by numeric device IDs if you have them:\n", + "# _ = checker.show_device_deployments(device_ids=[12345, 67890], inventory=inventory)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.1b Interactive Availability Widget\n", + "Explore deployment availability by device and date.\n", + "\n", + "Tip: leave the dates empty to use the full deployment history, or set a tighter window to speed things up.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from onc_hydrophone_data.utils import availability_widget\n", + "\n", + "device_codes = sorted({row['device_code'] for row in inventory['history']})\n", + "availability_widget(\n", + " checker,\n", + " device_codes=device_codes,\n", + " default_device=DEVICE,\n", + " start_date=datetime(2024, 1, 1, tzinfo=timezone.utc),\n", + " end_date=datetime(2024, 3, 1, tzinfo=timezone.utc),\n", + " auto_run=False, # Click \"Update\" to query ONC and display availability.\n", + ")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Example date used throughout the notebook\n", + "# Choose a time within the deployment ranges shown above\n", + "EXAMPLE_DATE = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "# 2. Download audio and ONC products\n", + "\n", + "Section 2.1 performs the most common complete workflow and displays the spectrogram generated from downloaded audio. The following sections introduce ONC-generated products, range downloads, sampling, and event downloads.\n", + "\n", + "ONC archive files are organized in five-minute coverage windows. The download helpers determine the required windows, submit requests in parallel, and resume around files already present locally.\n", + "\n", + "`HSD_OPTIONS` is only for ONC-generated HSD products. It does not change local `SpectrogramGenerator` output. See the online **ONC Spectrogram Products and Server Options** guide for all server settings and product trade-offs.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Optional ONC HSD settings used only by server-product examples below.\n", + "# Leave empty to use the package defaults.\n", + "HSD_OPTIONS = {\n", + " # \"dpo_spectralDataDownsample\": 1, # 1=one-minute, 2=plot resolution, 0=full resolution\n", + "}\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2.1 Download audio and make a spectrogram\n", + "\n", + "This cell downloads a short audio range, selects the first FLAC/WAV file, computes a local spectrogram, saves PNG and MAT outputs, and displays the generated PNG. Change `DEVICE` and the UTC interval to a deployment selected in Section 1.1.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from onc_hydrophone_data.audio import SpectrogramGenerator\n", + "\n", + "quickstart_start = EXAMPLE_DATE\n", + "quickstart_end = quickstart_start + timedelta(minutes=10)\n", + "\n", + "audio_summary = dl.download_audio_for_range(\n", + " device_code=DEVICE,\n", + " start_dt=quickstart_start,\n", + " end_dt=quickstart_end,\n", + " tag=\"audio_to_spectrogram\",\n", + ")\n", + "audio_path = find_first_file(dl.audio_path, [\"*.flac\", \"*.wav\"])\n", + "if audio_path is None:\n", + " raise FileNotFoundError(f\"No audio was downloaded to {dl.audio_path}\")\n", + "\n", + "quickstart_generator = SpectrogramGenerator(\n", + " win_dur=0.5,\n", + " overlap=0.75,\n", + " freq_lims=(20, 10_000),\n", + " crop_freq_lims=True,\n", + " log_freq=False,\n", + ")\n", + "quickstart_output = Path(dl.audio_path).parent / \"custom_spectrograms\"\n", + "quickstart_result = quickstart_generator.process_single_file(\n", + " audio_path,\n", + " quickstart_output,\n", + " save_plot=True,\n", + " save_mat=True,\n", + ")\n", + "print(f\"Audio: {audio_path}\")\n", + "print(f\"Spectrogram: {quickstart_result['png_file']}\")\n", + "display(Image(filename=quickstart_result[\"png_file\"]))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2.2 Range downloads\n", + "Download every five-minute file that overlaps an interval.\n", + "\n", + "### ONC-generated spectrograms\n", + "This requests HSD products computed by ONC. Set `spectrograms_per_batch` to control request size; add `download_audio=True` if you also need the matching source audio.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Download ALL spectrograms between two dates, batched by spectrograms_per_batch\n", + "range_start = datetime(2024, 4, 1, 0, 0, tzinfo=timezone.utc)\n", + "range_end = range_start + timedelta(minutes=30) # keep short for tutorial\n", + "spectrograms_per_batch = 3 # number of 5-min spectrograms per request\n", + "\n", + "print(f\"Date range: {range_start} to {range_end}\")\n", + "\n", + "result = dl.download_spectrograms_for_range(\n", + " DEVICE,\n", + " range_start,\n", + " range_end,\n", + " spectrograms_per_batch,\n", + " # download_audio=True,\n", + " # data_product_options=HSD_OPTIONS,\n", + ")\n", + "print(json.dumps(result, indent=2))\n", + "\n", + "plot_first_spectrogram(dl, title=\"Date range spectrogram\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Audio only\n", + "Download all 5-minute audio files that overlap the range (FLAC, with WAV fallback).\n", + "\n", + "This is the audio-only equivalent of the spectrogram range download above.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Download audio for a time range\n", + "audio_start = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)\n", + "audio_end = audio_start + timedelta(minutes=10) # 2 files\n", + "\n", + "print(f\"Audio range: {audio_start} to {audio_end}\")\n", + "\n", + "dl.download_audio_for_range(\n", + " DEVICE,\n", + " audio_start,\n", + " audio_end,\n", + ")\n", + "print(f\"Audio saved to: {dl.audio_path}\")\n", + "plot_first_audio(dl, max_seconds=10.0)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2.3 Sample uniformly across a range\n", + "Sampling selects evenly spaced five-minute windows across the full date range. This gives a representative subset without downloading every file.\n", + "\n", + "You control:\n", + "- start/end date\n", + "- total samples (number of 5-minute windows)\n", + "- per-request batch size (how many windows per request)\n", + "\n", + "### ONC-generated spectrograms (optional audio)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Sample N spectrograms evenly across a date range\n", + "sampling_start = datetime(2024, 4, 1, 0, 0, tzinfo=timezone.utc)\n", + "sampling_end = datetime(2024, 4, 1, 2, 0, tzinfo=timezone.utc) # 2 hours\n", + "total_samples = 4\n", + "spectrograms_per_request = 2\n", + "\n", + "print(f\"Sampling {total_samples} spectrograms from {sampling_start} to {sampling_end}\")\n", + "\n", + "info = dl.download_sampled_spectrograms(\n", + " DEVICE,\n", + " sampling_start,\n", + " sampling_end,\n", + " total_samples,\n", + " spectrograms_per_request,\n", + " # download_audio=True,\n", + " # data_product_options=HSD_OPTIONS,\n", + ")\n", + "print(json.dumps(info, indent=2))\n", + "\n", + "plot_first_spectrogram(dl, title=\"Sampled spectrogram\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Audio only\n", + "Sample evenly spaced five-minute audio files across the same range. Use this to build a smaller local-spectrogram dataset.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Sample N audio files evenly across the same date range\n", + "# Reuse sampling_start/sampling_end/total_samples from above\n", + "audio_files_per_request = 2\n", + "\n", + "audio_info = dl.download_sampled_audio(\n", + " DEVICE,\n", + " sampling_start,\n", + " sampling_end,\n", + " total_samples,\n", + " audio_files_per_request,\n", + ")\n", + "\n", + "# json.dumps can't handle datetime objects directly, so serialize them first.\n", + "serializable_info = {\n", + " **audio_info,\n", + " \"start_dt\": audio_info[\"start_dt\"].isoformat(),\n", + " \"end_dt\": audio_info[\"end_dt\"].isoformat(),\n", + " \"request_windows\": [\n", + " (start.isoformat(), end.isoformat())\n", + " for start, end in audio_info[\"request_windows\"]\n", + " ],\n", + "}\n", + "print(json.dumps(serializable_info, indent=2))\n", + "\n", + "plot_first_audio(dl, max_seconds=10.0)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2.4 Download files containing event timestamps\n", + "Provide UTC event timestamps and let the helper map each one to its containing five-minute ONC archive window.\n", + "\n", + "For a locally generated spectrogram with retained event padding and automatic STFT context, use `process_event()` in Section 3.3 or the local JSON workflow in Section 5.1.\n", + "\n", + "### ONC-generated spectrograms (optional audio)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Download spectrograms for event timestamps (mapped to 5-minute windows)\n", + "event_times = [\n", + " datetime(2024, 4, 1, 4, 25, 30, tzinfo=timezone.utc),\n", + " datetime(2024, 4, 1, 14, 10, 15, tzinfo=timezone.utc),\n", + " datetime(2024, 4, 2, 3, 45, 0, tzinfo=timezone.utc),\n", + "]\n", + "\n", + "spectrograms_per_request = 1\n", + "\n", + "info = dl.download_spectrograms_for_events(\n", + " DEVICE,\n", + " event_times,\n", + " spectrograms_per_request,\n", + " tag='event_times',\n", + " # download_audio=True,\n", + " # data_product_options=HSD_OPTIONS,\n", + ")\n", + "print(json.dumps(info, indent=2))\n", + "\n", + "plot_first_spectrogram(dl, title=\"Event-based spectrogram\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Audio only\n", + "Download the 5-minute audio files that contain each event timestamp.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Download audio for the same event timestamps\n", + "# Reuse event_times from the spectrogram example above.\n", + "\n", + "dl.download_audio_for_events(DEVICE, event_times)\n", + "plot_first_audio(dl, max_seconds=10.0)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2.5 Centered audio clip\n", + "Compute which 5-minute files are needed for a shorter clip, then download them in one call.\n", + "\n", + "Use `describe_audio_window` to see which files are needed, and `download_audio_for_center_time` to fetch them.\n", + "This example intentionally crosses a 5-minute boundary, so two adjacent audio files are required before clipping to the exact duration.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Download 30 seconds of audio centered on a timestamp near a 5-minute boundary\n", + "center_time = datetime(2024, 4, 1, 12, 34, 50, tzinfo=timezone.utc)\n", + "duration_seconds = 30\n", + "\n", + "window = dl.describe_audio_window(center_time, duration_seconds)\n", + "\n", + "dl.download_audio_for_center_time(DEVICE, center_time, duration_seconds)\n", + "print(\"Files downloaded - use audio utils to stitch and clip\")\n", + "\n", + "plot_first_audio(dl, max_seconds=10.0)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "# 3. Generate local spectrograms\n", + "\n", + "Local spectrograms are computed from FLAC/WAV audio. Complete-file processing returns complete STFT windows without adding artificial samples beyond the file. Interior clips and known-event modes add computation-only context at clip boundaries, compute the STFT, and then remove that context.\n", + "\n", + "| Mode | Use it for | Edge handling |\n", + "| --- | --- | --- |\n", + "| `process_single_file()` / `process_directory()` | Complete local files | Complete STFT windows only |\n", + "| `clip_start` / `clip_end` | A selected interval | Automatic half-window context by default |\n", + "| `process_event()` | A known signal offset in one file | Automatic half-window context plus retained before/after padding |\n", + "| `create_custom_spectrograms_from_json()` | Many timestamped ONC events | Downloads context, computes locally, and trims to each event interval |\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3.1 SpectrogramGenerator basics\n", + "Minimal setup for producing custom spectrograms from audio files.\n", + "\n", + "Defaults work well for most cases; the example below narrows the frequency range for readability. The next section lists every tunable parameter.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from onc_hydrophone_data.audio import SpectrogramGenerator\n", + "\n", + "# Create generator with default settings\n", + "generator = SpectrogramGenerator(\n", + " win_dur=1.0, # 1 second FFT window\n", + " overlap=0.5, # 50% overlap\n", + " window_type='hann', # Window function\n", + " freq_lims=(10, 1000) # 10 Hz to 1 kHz\n", + ")\n", + "\n", + "print(\"SpectrogramGenerator created with:\")\n", + "print(f\" Window: {generator.win_dur}s\")\n", + "print(f\" Overlap: {generator.overlap}\")\n", + "print(f\" Window type: {generator.window_type}\")\n", + "print(f\" Freq range: {generator.freq_lims}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3.2 Custom parameters\n", + "Every SpectrogramGenerator argument is optional; below are the tunable knobs and what they do.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Full customization example (all arguments are optional)\n", + "custom = SpectrogramGenerator(\n", + " win_dur=0.5,\n", + " overlap=0.75,\n", + " window_type=('kaiser', 14.0),\n", + " nfft=None,\n", + " win_length=None,\n", + " hop_length=None,\n", + " freq_lims=(10, 24000),\n", + " colormap='magma',\n", + " clim=(-80, 0),\n", + " log_freq=True,\n", + " max_duration=120.0,\n", + " clip_start=5.0,\n", + " clip_end=65.0,\n", + " backend='auto',\n", + " scaling='density',\n", + " quiet=False,\n", + " use_logging=True,\n", + ")\n", + "\n", + "# Presets for common use cases\n", + "high_res = SpectrogramGenerator(\n", + " win_dur=0.1, # 100ms window (higher time resolution)\n", + " overlap=0.9, # 90% overlap\n", + " window_type='hann',\n", + " freq_lims=(1, 24000), # Full frequency range\n", + " clim=(-80, 0) # dB scale limits\n", + ")\n", + "\n", + "low_freq = SpectrogramGenerator(\n", + " win_dur=2.0, # 2s window (better freq resolution)\n", + " overlap=0.5,\n", + " window_type=('kaiser', 8.0),\n", + " freq_lims=(10, 200), # Focus on low frequencies\n", + ")\n", + "\n", + "print(\"Custom config window type:\", custom.window_type)\n", + "print(\"High-res config:\", high_res.win_dur, high_res.freq_lims)\n", + "print(\"Low-freq config:\", low_freq.win_dur, low_freq.freq_lims)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3.3 Generate around a known signal time\n", + "\n", + "`process_event()` retains five seconds before and after the event by default. `edge_padding_seconds=\"auto\"` adds an extra half-window on both sides only for STFT computation, discards those context frames before relative-dB normalization, and returns time bins centred inside the requested event interval. This prevents incomplete-window artifacts at the clip boundaries.\n", + "\n", + "The event time is an offset in seconds from the beginning of the selected audio file. The command-line equivalent is `python scripts/generate_spectrograms.py --input-file FILE --event-time 30 --output-dir OUTPUT`.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Generate a ten-second spectrogram around a signal 30 seconds into a file.\n", + "audio_path = find_first_file(dl.audio_path, [\"*.flac\", \"*.wav\"])\n", + "if audio_path is None:\n", + " raise FileNotFoundError(\"Run the audio download in Section 2.1 first.\")\n", + "\n", + "event_generator = SpectrogramGenerator(\n", + " win_dur=0.5,\n", + " overlap=0.75,\n", + " freq_lims=(20, 10_000),\n", + " crop_freq_lims=True,\n", + " log_freq=False,\n", + ")\n", + "event_output = Path(dl.audio_path).parent / \"event_spectrograms\"\n", + "event_result = event_generator.process_event(\n", + " audio_path,\n", + " event_output,\n", + " event_time_seconds=30.0,\n", + " pad_before_seconds=5.0,\n", + " pad_after_seconds=5.0,\n", + " edge_padding_seconds=\"auto\",\n", + " save_plot=True,\n", + " save_mat=True,\n", + ")\n", + "print(f\"Retained interval: {event_result['target_start_seconds']:.1f}–{event_result['target_end_seconds']:.1f} s\")\n", + "print(f\"Computation-only context on each side: {event_result['edge_padding_seconds']:.3f} s\")\n", + "display(Image(filename=event_result[\"png_file\"]))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3.4 Batch-process an audio directory\n", + "Process an entire directory of audio files in one call.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Process all audio files in a directory\n", + "# Uses the most recent audio download path (run an audio download above).\n", + "audio_dir = Path(dl.audio_path)\n", + "output_dir = audio_dir.parent / \"custom_spectrograms\"\n", + "\n", + "if audio_dir.exists():\n", + " results = generator.process_directory(\n", + " input_dir=audio_dir,\n", + " save_dir=output_dir,\n", + " save_plot=True, # Save PNG plots\n", + " save_mat=True, # Save MAT files\n", + " )\n", + " print(f\"Processed {len(results)} files\")\n", + "else:\n", + " print(f\"No audio files at {audio_dir}\")\n", + " print(\"Run audio downloads first, then process them here (or set audio_dir manually)\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "# 4. Download ONC products and request files\n", + "\n", + "The methods in this section download HSD spectrogram products computed by ONC. They do not run the local `SpectrogramGenerator`, so local FFT settings and edge-context handling do not apply. Use Section 5 when you want JSON timestamps to drive local spectrogram generation from ONC audio.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4.1 Direct timestamps\n", + "When you have event times, ONC spectrograms are stored in 5-minute blocks. Map each event time to the surrounding 5-minute window (e.g., 04:27 -> 04:25-04:30) and request those windows. The example below passes those explicit start/end windows directly so you control exactly what gets downloaded.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Explicit 5-minute windows (e.g., derived from event timestamps)\n", + "explicit_windows = {\n", + " DEVICE: [\n", + " (datetime(2024, 4, 1, 4, 25, tzinfo=timezone.utc), \n", + " datetime(2024, 4, 1, 4, 30, tzinfo=timezone.utc)),\n", + " (datetime(2024, 4, 1, 14, 0, tzinfo=timezone.utc), \n", + " datetime(2024, 4, 1, 14, 5, tzinfo=timezone.utc)),\n", + " (datetime(2024, 4, 2, 3, 15, tzinfo=timezone.utc), \n", + " datetime(2024, 4, 2, 3, 20, tzinfo=timezone.utc)),\n", + " ]\n", + "}\n", + "\n", + "print(f\"Downloading {len(explicit_windows[DEVICE])} specific time windows\")\n", + "\n", + "info = dl.download_spectrogram_windows(\n", + " DEVICE,\n", + " explicit_windows,\n", + " spectrograms_per_request=1,\n", + " tag='explicit_times',\n", + ")\n", + "print(json.dumps(info, indent=2))\n", + "\n", + "plot_first_spectrogram(dl, title=\"Explicit window spectrogram\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Example formats (datetime objects and legacy tuples):\n", + "Use these directly in Python, or convert them into the request schema below.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Python datetime objects\n", + "timestamps_datetime = [\n", + " datetime(2024, 4, 1, 12, 30, 0, tzinfo=timezone.utc),\n", + " datetime(2024, 4, 1, 14, 45, 30, tzinfo=timezone.utc),\n", + " datetime(2024, 4, 2, 8, 15, 0, tzinfo=timezone.utc),\n", + "]\n", + "\n", + "# Tuple format (legacy)\n", + "timestamps_tuple = [\n", + " [2024, 4, 1, 12, 30, 0],\n", + " [2024, 4, 1, 14, 45, 30],\n", + "]\n", + "\n", + "print(\"Datetime list:\", timestamps_datetime[:2])\n", + "print(\"Tuple list:\", timestamps_tuple)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4.2 JSON and CSV for ONC downloads\n", + "Use `download_requests_from_json()` or `download_requests_from_csv()` when you want to download ONC products for many events. `download_spectrogram` defaults to `true`, meaning ONC computes the spectrogram. Set `download_audio` when the matching source audio is also needed.\n", + "\n", + "Do not confuse this with `create_custom_spectrograms_from_json()` in Section 5, which downloads audio and computes new spectrograms locally with your generator settings.\n", + "\n", + "\n", + "**Request schema (JSON/CSV)**\n", + "\n", + "JSON uses a `{defaults, requests}` payload. CSV is a flat table with the same fields (one row per request). Required per request: `deviceCode` and either `timestamp` *or* a `start`/`end` window.\n", + "You can set `timezone` in `defaults` or per-request; it applies to naive timestamps.\n", + "You can mix multiple devices in one file by setting `deviceCode` per request (JSON) or per row (CSV).\n", + "\n", + "JSON format:\n", + "```\n", + "{\n", + " \"defaults\": { ... },\n", + " \"requests\": [ { ... }, { ... } ]\n", + "}\n", + "```\n", + "\n", + "| Field | Type | Required | Notes |\n", + "| --- | --- | --- | --- |\n", + "| `deviceCode` | string | yes | Hydrophone device code (e.g., `ICLISTENHF6324`) |\n", + "| `timestamp` | string | if no `start`/`end` | ISO 8601 timestamp (UTC or offset, e.g., `2024-04-01T12:30:00Z`) |\n", + "| `timezone` | string | no | Timezone for naive timestamps (e.g., `America/Vancouver`, `UTC`, `-07:00`) |\n", + "| `start` | string | if no `timestamp` | ISO 8601 timestamp (UTC or offset) |\n", + "| `end` | string | no | ISO 8601 timestamp (UTC or offset) |\n", + "| `duration_seconds` | number | no | Used when `start` is set but `end` is omitted |\n", + "| `pad_seconds` | number | no | Symmetric padding around `timestamp` or `start`/`end` |\n", + "| `pad_before_seconds` | number | no | Override padding before |\n", + "| `pad_after_seconds` | number | no | Override padding after |\n", + "| `download_audio` | bool | no | Download audio files (default: false) |\n", + "| `download_spectrogram` | bool | no | Download ONC spectrograms (default: true) |\n", + "| `spectrogram_format` | string | no | `mat` or `png` |\n", + "| `clip` | bool | no | Clip outputs to the padded window |\n", + "| `audio_extension` | string | no | `flac` or `wav` |\n", + "| `output_tag` | string | no | Output folder tag |\n", + "| `output_name` | string | no | Override clip basename |\n", + "| `label` / `description` | string | no | Metadata label |\n", + "| `data_product_options` | object | no | ONC `dpo_*` options (same as `HSD_OPTIONS`) |\n", + "\n", + "Tip: In CSV, use a `deviceCode` column to match the JSON field name. For `data_product_options`, store a JSON string per row and parse it in Python.\n", + "\n", + "**Padding + clipping behavior**\n", + "Each request expands to the 5-minute coverage windows needed for downloads. If padding crosses a window boundary, the downloader fetches adjacent files.\n", + "\n", + "- Audio clips are trimmed to the exact padded interval.\n", + "- ONC spectrogram clips are trimmed to the nearest time-bin boundaries on the fixed 5-minute grid, so the spectrogram duration can differ slightly from the audio (up to ~one bin).\n", + "\n", + "The bin width is `300s / num_time_bins` for each 5-minute file and is stored as `seconds_per_column` in the clip metadata.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.2a ONC-download JSON example\n", + "Write a request file, then execute it to download audio and/or ONC-generated spectrograms.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Minimal ONC-download JSON payload.\n", + "onc_json_example = {\n", + " \"defaults\": {\n", + " \"deviceCode\": DEVICE,\n", + " \"pad_seconds\": 30,\n", + " \"download_spectrogram\": True,\n", + " \"data_product_options\": HSD_OPTIONS\n", + " },\n", + " \"requests\": [\n", + " {\"timestamp\": \"2024-04-01T12:30:00Z\"},\n", + " {\"start\": \"2024-04-01T14:00:00Z\", \"end\": \"2024-04-01T14:05:00Z\"}\n", + " ]\n", + "}\n", + "print(json.dumps(onc_json_example, indent=2))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Executable ONC-download request file.\n", + "onc_download_requests = {\n", + " \"defaults\": {\n", + " \"pad_seconds\": 15,\n", + " \"download_audio\": True,\n", + " \"download_spectrogram\": True,\n", + " \"clip\": True,\n", + " \"data_product_options\": HSD_OPTIONS\n", + " },\n", + " \"requests\": [\n", + " {\n", + " \"deviceCode\": DEVICE,\n", + " \"timestamp\": \"2024-04-01T12:34:50Z\",\n", + " \"label\": \"whale call 1\"\n", + " },\n", + " {\n", + " \"deviceCode\": DEVICE,\n", + " \"start\": \"2024-04-01T12:30:00Z\",\n", + " \"end\": \"2024-04-01T12:33:30Z\",\n", + " \"pad_before_seconds\": 10,\n", + " \"pad_after_seconds\": 20,\n", + " \"label\": \"ship noise event\"\n", + " }\n", + " ]\n", + "}\n", + "onc_json_path = Path(DATA_DIR) / \"onc_product_requests.json\"\n", + "onc_json_path.write_text(json.dumps(onc_download_requests, indent=2))\n", + "print(f\"Saved ONC product requests to: {onc_json_path}\")\n", + "print(json.dumps(onc_download_requests, indent=2))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# This downloads ONC products; it does not compute a local spectrogram.\n", + "onc_results = dl.download_requests_from_json(\n", + " str(onc_json_path),\n", + ")\n", + "print(json.dumps(onc_results, indent=2))\n", + "\n", + "plot_request_results(onc_results, downloader=dl)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 4.2b ONC-download CSV example\n", + "CSV can be executed directly with `download_requests_from_csv` (no pandas required).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# data_product_options is a JSON string inside the CSV field.\n", + "csv_content = f\"\"\"deviceCode,timestamp,label,data_product_options\n", + "{DEVICE},2024-04-01T12:30:00Z,whale call,\"{{\"\"dpo_spectralDataDownsample\"\": 2}}\"\n", + "{DEVICE},2024-04-15T14:45:30Z,ship noise,\"{{\"\"dpo_spectralDataDownsample\"\": 1}}\"\n", + "{DEVICE},2024-04-02T08:15:00Z,unknown,\"\"\n", + "\"\"\"\n", + "\n", + "csv_path = Path(DATA_DIR) / \"example_requests.csv\"\n", + "csv_path.write_text(csv_content)\n", + "print(f\"Saved CSV to: {csv_path}\")\n", + "\n", + "# Execute CSV requests (uses settings from the CSV file)\n", + "results = dl.download_requests_from_csv(\n", + " str(csv_path),\n", + ")\n", + "print(json.dumps(results, indent=2))\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4.3 Supported date/time formats\n", + "These formats are parsed consistently by the downloader utilities and are accepted anywhere a timestamp is expected.\n", + "All inputs are converted to UTC. If you use naive datetimes or strings, set a `timezone` in request files or pass a tz-aware datetime.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# All supported input formats\n", + "formats = [\n", + " datetime(2024, 4, 1, 12, 30, tzinfo=timezone.utc), # datetime object\n", + " \"2024-04-01T12:30:00Z\", # ISO 8601\n", + " \"2024-04-01T12:30:00.000Z\", # ISO with ms\n", + " \"2024-04-01T12:30:00-07:00\", # ISO with offset\n", + " [2024, 4, 1, 12, 30, 0], # list\n", + " (2024, 4, 1, 12, 30, 0), # tuple\n", + "]\n", + "\n", + "print(\"All these formats are parsed correctly:\")\n", + "for f in formats:\n", + " parsed = HydrophoneDownloader._parse_timestamp_value(f)\n", + " print(f\" {type(f).__name__:10} → {parsed}\")\n", + "\n", + "print(\"Timezone override for naive strings:\")\n", + "local_parsed = HydrophoneDownloader._parse_timestamp_value(\n", + " \"2024-04-01 12:30:00\",\n", + " timezone_str=\"America/Vancouver\",\n", + ")\n", + "print(f\" America/Vancouver → {local_parsed}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 5. Local JSON and end-to-end workflows\n", + "\n", + "This section uses ONC timestamps to download source audio and compute new spectrograms locally. These are not ONC HSD products.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5.1 Generate local event spectrograms from JSON\n", + "\n", + "`create_custom_spectrograms_from_json()` reads `generator_defaults` and per-request `generator_options`, downloads the necessary ONC audio, and runs `SpectrogramGenerator` locally. The JSON `pad_seconds` value controls how much event data is retained.\n", + "\n", + "With `clip_pad_seconds=\"auto\"`, the downloader fetches an extra half of `win_dur` on both sides. That context is used for the STFT, removed before relative-dB normalization, and excluded from the returned event interval. Every retained time bin therefore has a complete analysis window. If you override the FFT with a sample-based `win_length` that differs from `win_dur`, pass the matching context explicitly in seconds.\n", + "\n", + "This is different from Section 4: `download_requests_from_json()` retrieves spectrograms computed by ONC and ignores local generator settings.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "custom_json_path = Path(DATA_DIR) / \"custom_spectrogram_requests.json\"\n", + "\n", + "custom_requests = {\n", + " \"defaults\": {\n", + " \"deviceCode\": DEVICE,\n", + " \"pad_seconds\": 15,\n", + " \"label\": \"custom_clip\",\n", + " },\n", + " \"generator_defaults\": {\n", + " \"win_dur\": 0.5,\n", + " \"overlap\": 0.5,\n", + " \"window_type\": \"hann\",\n", + " \"crop_freq_lims\": True,\n", + " \"log_freq\": False,\n", + " \"quiet\": True,\n", + " },\n", + " \"requests\": [\n", + " {\n", + " \"timestamp\": \"2024-04-01T12:30:00Z\",\n", + " \"label\": \"early_april\",\n", + " \"generator_options\": {\"freq_lims\": [10, 500]},\n", + " },\n", + " {\n", + " \"timestamp\": \"2024-04-15T12:30:00Z\",\n", + " \"label\": \"mid_april\",\n", + " \"generator_options\": {\"freq_lims\": [10, 2000]},\n", + " },\n", + " {\n", + " \"timestamp\": \"2024-04-30T12:30:00Z\",\n", + " \"label\": \"late_april\",\n", + " \"generator_options\": {\"freq_lims\": [10, 10000]},\n", + " },\n", + " ],\n", + "}\n", + "\n", + "custom_json_path.write_text(json.dumps(custom_requests, indent=2))\n", + "\n", + "custom_results = dl.create_custom_spectrograms_from_json(\n", + " str(custom_json_path),\n", + " clip_pad_seconds=\"auto\",\n", + " save_png=True,\n", + " save_mat=True,\n", + " save_npy=False,\n", + ")\n", + "print(json.dumps(custom_results, indent=2, default=str))\n", + "\n", + "for result in custom_results:\n", + " custom_spec = result.get(\"custom_spectrogram\") or {}\n", + " png_file = custom_spec.get(\"png_file\")\n", + " if not png_file:\n", + " continue\n", + " timestamp = result.get(\"timestamp\", \"\")\n", + " print(f\"Local spectrogram for {timestamp}: {png_file}\")\n", + " display(Image(filename=png_file))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Inspect the first saved spectrogram file\n", + "import scipy.io\n", + "\n", + "first_mat = None\n", + "first_npy = None\n", + "for result in custom_results:\n", + " custom_spec = result.get(\"custom_spectrogram\") or {}\n", + " first_mat = first_mat or custom_spec.get(\"mat_file\")\n", + " first_npy = first_npy or custom_spec.get(\"npy_file\")\n", + " if first_mat or first_npy:\n", + " break\n", + "\n", + "if first_mat:\n", + " mat = scipy.io.loadmat(first_mat)\n", + " keys = sorted(k for k in mat.keys() if not k.startswith(\"__\"))\n", + " print(f\"MAT keys: {keys}\")\n", + " for key in keys:\n", + " value = mat[key]\n", + " if hasattr(value, \"shape\"):\n", + " print(f\" {key}: shape={value.shape}, dtype={getattr(value, 'dtype', None)}\")\n", + " else:\n", + " print(f\" {key}: type={type(value).__name__}\")\n", + " meta_json = mat.get(\"metadata_json\")\n", + " if meta_json is not None:\n", + " try:\n", + " meta_text = meta_json.item()\n", + " except Exception:\n", + " meta_text = str(meta_json)\n", + " try:\n", + " meta = json.loads(meta_text)\n", + " print(f\"metadata_json keys: {sorted(meta.keys())}\")\n", + " except json.JSONDecodeError:\n", + " print(\"metadata_json: \")\n", + "elif first_npy:\n", + " data = np.load(first_npy, allow_pickle=True).item()\n", + " print(f\"NPY keys: {sorted(data.keys())}\")\n", + " metadata = data.get(\"metadata\")\n", + " if isinstance(metadata, dict):\n", + " print(f\"metadata keys: {sorted(metadata.keys())}\")\n", + "else:\n", + " print(\"No saved spectrogram files found to inspect.\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5.2 Batch pipeline: download audio → local spectrograms\n", + "Download audio, then generate custom spectrograms for analysis.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Complete workflow: Download audio + generate custom spectrograms\n", + "pipeline_start = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)\n", + "pipeline_end = pipeline_start + timedelta(minutes=10)\n", + "\n", + "# Step 1: Download audio\n", + "dl.download_audio_for_range(DEVICE, pipeline_start, pipeline_end, tag=\"pipeline_demo\")\n", + "print(f\"1. Audio downloaded to: {dl.audio_path}\")\n", + "\n", + "plot_first_audio(dl, max_seconds=10.0)\n", + "\n", + "# Step 2: Generate custom spectrograms\n", + "custom_out = Path(dl.audio_path).parent / \"custom_spectrograms\"\n", + "results = generator.process_directory(\n", + " input_dir=dl.audio_path,\n", + " save_dir=custom_out,\n", + ")\n", + "print(f\"2. Custom spectrograms saved to: {custom_out}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5.3 Multi-device downloads\n", + "Repeat the same download pattern across multiple hydrophones.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Download from multiple hydrophones\n", + "# Keep the range short for a quick multi-device check.\n", + "devices = ['ICLISTENHF6324', 'ICLISTENHF6020', 'ICLISTENHF6019']\n", + "multi_start = datetime(2024, 4, 1, 12, 0, tzinfo=timezone.utc)\n", + "multi_end = multi_start + timedelta(minutes=15)\n", + "\n", + "for device in devices:\n", + " windows = {device: [(multi_start, multi_end)]}\n", + " print(f\"Device {device}: {multi_start} to {multi_end}\")\n", + "\n", + " info = dl.download_spectrogram_windows(\n", + " device,\n", + " windows,\n", + " spectrograms_per_request=3,\n", + " tag='multi_device',\n", + " )\n", + " print(f\" Downloaded: {info.get('runs_downloaded', 0)} runs\")\n", + "\n", + "plot_first_spectrogram(dl, title=f\"Multi-device spectrogram ({devices[-1]})\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 6. Output Folder Structure\n", + "Overview of where downloads and generated files are stored.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# View current paths for a known tag/date range\n", + "example_tag = 'audio_to_spectrogram'\n", + "example_start = EXAMPLE_DATE\n", + "example_end = example_start + timedelta(minutes=10)\n", + "\n", + "dl.setup_directories('mat', DEVICE, example_tag, example_start, example_end)\n", + "print(\"Output paths:\")\n", + "print(f\" Spectrograms: {dl.spectrogram_path}\")\n", + "print(f\" Audio: {dl.audio_path}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 7. Troubleshooting, practice, and next steps\n", + "Common issues and ways to make downloads and local processing reliable.\n", + "\n", + "## Common Issues\n", + "\n", + "| Issue | Solution |\n", + "| --- | --- |\n", + "| \"Missing ONC_TOKEN\" | Add `ONC_TOKEN=...` to `.env` in the repo root |\n", + "| \"Device not deployed\" | Run Section 1.1 (HydrophoneDeploymentChecker) and choose dates within deployment |\n", + "| \"Waiting on file system\" | Normal - ONC is generating data; wait and retry |\n", + "| Timeout errors | Reduce request size or increase wait time (`max_wait_minutes`) |\n", + "| Rate limiting | Reduce request size or add delays between runs |\n", + "| Local generation is slow or uses too much memory | Start with `backend=\"scipy\"`, `max_workers=1`, and a focused `freq_lims` range with `crop_freq_lims=True` |\n", + "| Event edges look incomplete | Use `process_event()` or keep `clip_pad_seconds=\"auto\"`; do not set the computation context to zero |\n", + "\n", + "## Performance Tips\n", + "\n", + "1. **Batch requests**: Group spectrograms into 6-12 per request\n", + "2. **Avoid full resolution**: `downsample=0` is much slower\n", + "3. **Check archive first**: Archived data downloads instantly\n", + "4. **Prefer shorter ranges**: Large ranges are easier to handle in chunks\n", + "\n", + "## Exercise\n", + "\n", + "Using the audio downloaded in Section 2.1, generate a spectrogram around 45 seconds that retains 2 seconds before and 8 seconds after the event. Keep automatic edge context, save PNG and MAT outputs, and display the real generated PNG. A runnable answer follows.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "exercise_audio = find_first_file(dl.audio_path, [\"*.flac\", \"*.wav\"])\n", + "if exercise_audio is None:\n", + " raise FileNotFoundError(\"Run the audio download in Section 2.1 first.\")\n", + "\n", + "exercise_result = event_generator.process_event(\n", + " exercise_audio,\n", + " Path(dl.audio_path).parent / \"event_spectrograms\",\n", + " event_time_seconds=45.0,\n", + " pad_before_seconds=2.0,\n", + " pad_after_seconds=8.0,\n", + " edge_padding_seconds=\"auto\",\n", + " save_plot=True,\n", + " save_mat=True,\n", + ")\n", + "print(f\"Edge context: {exercise_result['edge_padding_seconds']:.3f} s per side\")\n", + "display(Image(filename=exercise_result[\"png_file\"]))\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.12" + } + }, + "nbformat": 4, + "nbformat_minor": 4 } diff --git a/onc_hydrophone_data/audio/spectrogram_generator.py b/onc_hydrophone_data/audio/spectrogram_generator.py index 7209f24..77e07c6 100644 --- a/onc_hydrophone_data/audio/spectrogram_generator.py +++ b/onc_hydrophone_data/audio/spectrogram_generator.py @@ -4,6 +4,7 @@ Translates MATLAB spectrogram functionality to Python. """ +import copy import os import json from datetime import datetime, timezone @@ -548,18 +549,6 @@ def compute_spectrogram( self._last_scaling = scaling self._last_device = device if backend_used == 'torch' else None - # Normalize and convert to dB (following MATLAB: 10*log10(abs(P./max(P,[],'all')))) - max_power = np.max(Sxx) - if max_power > 0: - # Reuse one working array rather than allocating abs(), division, - # clipping, log, and multiplication intermediates. - power_db_norm = np.asarray(Sxx / max_power) - np.maximum(power_db_norm, 1e-10, out=power_db_norm) - np.log10(power_db_norm, out=power_db_norm) - power_db_norm *= 10.0 - else: - power_db_norm = np.full_like(Sxx, -100.0) # Very low dB value - if clip_meta and clip_meta.get('clip_duration_seconds') is not None: offset = float(clip_meta.get('clip_offset_seconds', 0.0)) duration = float(clip_meta.get('clip_duration_seconds', 0.0)) @@ -569,10 +558,22 @@ def compute_spectrogram( if keep_mask.any(): times = times[keep_mask] - offset Sxx = Sxx[:, keep_mask] - power_db_norm = power_db_norm[:, keep_mask] elif not self.quiet: self.log.warning("Clip trimming removed all spectrogram frames; check clip window settings.") + # Normalize after time trimming so computation-only edge context cannot + # change the relative dB values in the retained target interval. + max_power = np.max(Sxx) + if max_power > 0: + # Reuse one working array rather than allocating abs(), division, + # clipping, log, and multiplication intermediates. + power_db_norm = np.asarray(Sxx / max_power) + np.maximum(power_db_norm, 1e-10, out=power_db_norm) + np.log10(power_db_norm, out=power_db_norm) + power_db_norm *= 10.0 + else: + power_db_norm = np.full_like(Sxx, -100.0) # Very low dB value + if not self.quiet: self.log.info(f"Spectrogram computed: {frequencies.shape[0]} freq bins, {times.shape[0]} time frames") return frequencies, times, Sxx, power_db_norm @@ -668,7 +669,8 @@ def process_single_file(self, audio_path: Union[str, Path], save_mat: bool = True, save_npy: bool = False, extra_metadata: Optional[dict] = None, - retain_arrays: bool = True) -> dict: + retain_arrays: bool = True, + output_stem: Optional[str] = None) -> dict: """Process a single audio file and generate a spectrogram. Args: @@ -681,6 +683,8 @@ def process_single_file(self, audio_path: Union[str, Path], extra_metadata: Optional extra metadata to store in outputs. retain_arrays: Keep full spectrogram arrays in the returned dict. Disable this for batch jobs to release memory after each file. + output_stem: Optional filename stem for saved outputs. Defaults to + the input audio filename stem. Returns: Dict with file paths, arrays, and metadata. Keys include: @@ -698,7 +702,7 @@ def process_single_file(self, audio_path: Union[str, Path], \"example.flac\", \"./out\", save_mat=True, - save_png=False, + save_plot=False, ) print(result[\"mat_file\"]) ``` @@ -723,7 +727,9 @@ def process_single_file(self, audio_path: Union[str, Path], ) # Create output filenames - base_name = audio_path.stem + base_name = Path(output_stem).name if output_stem else audio_path.stem + if not base_name or base_name in {'.', '..'}: + raise ValueError("output_stem must contain a valid filename stem") mat_path = save_dir / f"{base_name}.mat" png_path = save_dir / f"{base_name}.png" npy_path = save_dir / f"{base_name}.npy" @@ -801,6 +807,138 @@ def process_single_file(self, audio_path: Union[str, Path], return results + def process_event( + self, + audio_path: Union[str, Path], + save_dir: Union[str, Path], + event_time_seconds: float, + *, + pad_before_seconds: float = 5.0, + pad_after_seconds: Optional[float] = None, + edge_padding_seconds: Union[float, str, None] = 'auto', + save_plot: bool = True, + save_mat: bool = True, + save_npy: bool = False, + extra_metadata: Optional[dict] = None, + retain_arrays: bool = True, + output_stem: Optional[str] = None, + ) -> dict: + """Generate an edge-safe spectrogram around an event in an audio file. + + The retained target interval extends before and after + ``event_time_seconds``. Additional audio context is included only while + computing the STFT, then frames are trimmed back to the target interval. + The default ``'auto'`` context is half the resolved STFT window on each + side, so every retained frame has a complete analysis window. + + Args: + audio_path: Source audio file. + save_dir: Directory for generated outputs. + event_time_seconds: Event offset from the start of the audio file. + pad_before_seconds: Target data retained before the event. Defaults + to five seconds. + pad_after_seconds: Target data retained after the event. Defaults to + ``pad_before_seconds``. + edge_padding_seconds: Extra computation-only context on each side. + Use ``'auto'`` (default) to derive half the STFT window length. + save_plot: Save a PNG plot. + save_mat: Save MATLAB output. + save_npy: Save NumPy output. + extra_metadata: Optional additional output metadata. + retain_arrays: Keep arrays in the returned result. + output_stem: Optional saved-output filename stem. + + Returns: + The usual single-file result plus event and target-window metadata. + """ + try: + event_time_seconds = float(event_time_seconds) + pad_before_seconds = float(pad_before_seconds) + if pad_after_seconds is None: + pad_after_seconds = pad_before_seconds + pad_after_seconds = float(pad_after_seconds) + except (TypeError, ValueError) as exc: + raise ValueError("Event time and padding values must be numeric") from exc + + if not all(np.isfinite(value) for value in ( + event_time_seconds, + pad_before_seconds, + pad_after_seconds, + )): + raise ValueError("Event time and padding values must be finite") + if event_time_seconds < 0: + raise ValueError("event_time_seconds must be >= 0") + if pad_before_seconds < 0 or pad_after_seconds < 0: + raise ValueError("Event padding values must be >= 0") + if pad_before_seconds + pad_after_seconds <= 0: + raise ValueError("At least one event padding value must be > 0") + + target_start = max(0.0, event_time_seconds - pad_before_seconds) + target_end = event_time_seconds + pad_after_seconds + if target_end <= target_start: + raise ValueError("The event target interval is empty") + + if edge_padding_seconds is not None and not ( + isinstance(edge_padding_seconds, str) + and edge_padding_seconds.strip().lower() == 'auto' + ): + try: + explicit_edge_padding = float(edge_padding_seconds) + except (TypeError, ValueError) as exc: + raise ValueError( + "edge_padding_seconds must be a non-negative float or 'auto'" + ) from exc + if not np.isfinite(explicit_edge_padding) or explicit_edge_padding < 0: + raise ValueError( + "edge_padding_seconds must be a non-negative finite value" + ) + edge_padding_seconds = explicit_edge_padding + + # A shallow copy keeps the thread-safe FFT cache but isolates clip + # settings so this convenience method never mutates the caller's + # generator, including when it is reused for directory processing. + event_generator = copy.copy(self) + event_generator.clip_start = target_start + event_generator.clip_end = target_end + event_generator.clip_pad_seconds = edge_padding_seconds + event_generator.max_duration = None + + event_metadata = { + **(extra_metadata or {}), + 'event_spectrogram': { + 'event_time_seconds': event_time_seconds, + 'target_start_seconds': target_start, + 'target_end_seconds': target_end, + 'pad_before_seconds': event_time_seconds - target_start, + 'pad_after_seconds': pad_after_seconds, + 'edge_padding_requested': edge_padding_seconds, + }, + } + if output_stem is None: + event_milliseconds = int(round(event_time_seconds * 1000.0)) + output_stem = f"{Path(audio_path).stem}_event_{event_milliseconds}ms" + + result = event_generator.process_single_file( + audio_path, + save_dir, + save_plot=save_plot, + save_mat=save_mat, + save_npy=save_npy, + extra_metadata=event_metadata, + retain_arrays=retain_arrays, + output_stem=output_stem, + ) + clip_meta = result.get('metadata', {}).get('clip_meta') or {} + result.update({ + 'event_time_seconds': event_time_seconds, + 'target_start_seconds': target_start, + 'target_end_seconds': target_end, + 'pad_before_seconds': event_time_seconds - target_start, + 'pad_after_seconds': pad_after_seconds, + 'edge_padding_seconds': clip_meta.get('clip_pad_seconds'), + }) + return result + def save_numpy_format( self, frequencies: np.ndarray, diff --git a/onc_hydrophone_data/data/__init__.py b/onc_hydrophone_data/data/__init__.py index e69de29..1d6a20e 100644 --- a/onc_hydrophone_data/data/__init__.py +++ b/onc_hydrophone_data/data/__init__.py @@ -0,0 +1,11 @@ +"""Public data-download API.""" + +from ..onc.common import ensure_timezone_aware +from .downloader import FIVE_MINUTES_SECONDS, HydrophoneDownloader, TimestampRequest + +__all__ = [ + "FIVE_MINUTES_SECONDS", + "HydrophoneDownloader", + "TimestampRequest", + "ensure_timezone_aware", +] diff --git a/scripts/generate_spectrograms.py b/scripts/generate_spectrograms.py index b0fd878..4534e9a 100644 --- a/scripts/generate_spectrograms.py +++ b/scripts/generate_spectrograms.py @@ -26,6 +26,9 @@ # Process single file python scripts/generate_spectrograms.py --input-file audio.flac --output-dir spectrograms/ + + # Process an event at 123.4 seconds with edge-safe STFT context + python scripts/generate_spectrograms.py --input-file audio.flac --event-time 123.4 """ import os @@ -490,6 +493,9 @@ def main(): # Single file processing python %(prog)s --input-file audio.flac --output-dir spectrograms/ + + # Event processing (five retained seconds per side; automatic edge context) + python %(prog)s --input-file audio.flac --event-time 123.4 # MATLAB files only python %(prog)s --input-dir audio/ --no-plots @@ -552,10 +558,26 @@ def main(): default=None, help='Clip end time in seconds (default: none)') - parser.add_argument('--clip-pad-seconds', + parser.add_argument('--event-time', + type=float, + default=None, + help='Event offset in a single input file, in seconds') + + parser.add_argument('--event-pad-before', + type=float, + default=5.0, + help='Target seconds retained before --event-time (default: 5)') + + parser.add_argument('--event-pad-after', type=float, default=None, - help='Extra context (seconds) on each side before STFT (default: auto)') + help='Target seconds retained after --event-time (default: same as before)') + + parser.add_argument('--clip-pad-seconds', '--edge-pad-seconds', + dest='clip_pad_seconds', + type=float, + default=None, + help='Extra computation-only context on each side before STFT (default: auto half-window)') parser.add_argument('--freq-min', type=float, @@ -657,6 +679,11 @@ def main(): if not input_path_obj.exists(): print_status(f"Input path not found: {input_path}", "ERROR") return + if args.event_time is not None: + if is_directory: + raise ValueError("--event-time requires --input-file") + if args.clip_start is not None or args.clip_end is not None: + raise ValueError("--event-time cannot be combined with --clip-start/--clip-end") # Determine output directory output_dir = determine_output_directory(input_path, is_directory, args.output_dir) @@ -722,11 +749,34 @@ def main(): print_header("PROCESSING") start_time = time.time() - summary = process_audio_files( - input_path, output_dir, is_directory, - generator, save_mat, save_plot, - max_workers=args.max_workers if (args.input_dir or args.input_file) else params['max_workers'], - ) + if args.event_time is not None: + result = generator.process_event( + input_path, + output_dir, + event_time_seconds=args.event_time, + pad_before_seconds=args.event_pad_before, + pad_after_seconds=args.event_pad_after, + edge_padding_seconds=( + args.clip_pad_seconds + if args.clip_pad_seconds is not None + else 'auto' + ), + save_mat=save_mat, + save_plot=save_plot, + ) + summary = { + 'total_files': 1, + 'successful': 1, + 'failed': 0, + 'output_directory': str(output_dir), + 'results': [result], + } + else: + summary = process_audio_files( + input_path, output_dir, is_directory, + generator, save_mat, save_plot, + max_workers=args.max_workers if (args.input_dir or args.input_file) else params['max_workers'], + ) processing_time = time.time() - start_time diff --git a/tests/test_hydrophone_downloader.py b/tests/test_hydrophone_downloader.py index 2341828..4047cd5 100644 --- a/tests/test_hydrophone_downloader.py +++ b/tests/test_hydrophone_downloader.py @@ -30,7 +30,7 @@ sys.modules['onc'] = MagicMock() sys.modules['onc.onc'] = MagicMock() -from onc_hydrophone_data.data.hydrophone_downloader import ( +from onc_hydrophone_data.data import ( HydrophoneDownloader, TimestampRequest, ensure_timezone_aware, diff --git a/tests/test_spectrogram_generator.py b/tests/test_spectrogram_generator.py index d907954..35aadc0 100644 --- a/tests/test_spectrogram_generator.py +++ b/tests/test_spectrogram_generator.py @@ -84,3 +84,126 @@ def test_hashable_windows_are_cached(): first = generator._resolve_window(321) second = generator._resolve_window(321) assert first is second + + +def test_process_event_uses_complete_windows_and_trims_context(tmp_path: Path): + sample_rate = 1_000 + duration_seconds = 20 + time_axis = np.arange(sample_rate * duration_seconds) / sample_rate + signal = ( + np.sin(2 * np.pi * 80 * time_axis) + + 0.4 * np.sin(2 * np.pi * 180 * time_axis) + ) + # A much stronger signal outside the target interval verifies that extra + # computation context is excluded before relative-dB normalization. + signal[(time_axis >= 14.0) & (time_axis < 15.0)] *= 20.0 + audio_path = tmp_path / "events.wav" + sf.write(audio_path, signal, sample_rate, subtype="FLOAT") + + generator = SpectrogramGenerator( + win_dur=2.0, + overlap=0.5, + backend="scipy", + quiet=True, + ) + loaded_audio, loaded_rate, _ = generator.load_audio(audio_path) + _, full_times, full_power, _ = generator.compute_spectrogram( + loaded_audio, + loaded_rate, + ) + + result = generator.process_event( + audio_path, + tmp_path / "spectrograms", + event_time_seconds=10.0, + pad_before_seconds=3.0, + pad_after_seconds=3.0, + save_plot=False, + save_mat=False, + ) + + target_mask = (full_times >= 7.0) & (full_times <= 13.0) + np.testing.assert_allclose( + result["times"], + full_times[target_mask] - 7.0, + ) + np.testing.assert_allclose( + result["power_spectrogram"], + full_power[:, target_mask], + rtol=1e-6, + atol=1e-12, + ) + assert result["duration"] == pytest.approx(6.0) + assert result["edge_padding_seconds"] == pytest.approx(1.0) + assert result["target_start_seconds"] == pytest.approx(7.0) + assert result["target_end_seconds"] == pytest.approx(13.0) + assert generator.clip_start is None + assert generator.clip_end is None + + wider_context_result = generator.process_event( + audio_path, + tmp_path / "spectrograms", + event_time_seconds=10.0, + pad_before_seconds=3.0, + pad_after_seconds=3.0, + edge_padding_seconds=2.0, + save_plot=False, + save_mat=False, + ) + assert np.max(wider_context_result["power_db_norm"]) == pytest.approx(0.0) + + default_result = generator.process_event( + audio_path, + tmp_path / "spectrograms", + event_time_seconds=10.0, + save_plot=False, + ) + assert default_result["target_start_seconds"] == pytest.approx(5.0) + assert default_result["target_end_seconds"] == pytest.approx(15.0) + assert default_result["pad_before_seconds"] == pytest.approx(5.0) + assert default_result["pad_after_seconds"] == pytest.approx(5.0) + assert default_result["edge_padding_seconds"] == pytest.approx(1.0) + assert Path(default_result["mat_file"]).name == "events_event_10000ms.mat" + event_metadata = default_result["metadata"]["extra_metadata"]["event_spectrogram"] + assert event_metadata["event_time_seconds"] == pytest.approx(10.0) + assert event_metadata["edge_padding_requested"] == "auto" + + +@pytest.mark.parametrize( + ("event_time", "pad_before", "pad_after", "message"), + [ + (-1.0, 5.0, 5.0, "event_time_seconds"), + (10.0, -1.0, 5.0, "padding values"), + (10.0, 0.0, 0.0, "At least one"), + (float("nan"), 5.0, 5.0, "finite"), + ], +) +def test_process_event_validates_window( + tmp_path: Path, + event_time: float, + pad_before: float, + pad_after: float, + message: str, +): + generator = SpectrogramGenerator(quiet=True) + + with pytest.raises(ValueError, match=message): + generator.process_event( + tmp_path / "missing.wav", + tmp_path, + event_time_seconds=event_time, + pad_before_seconds=pad_before, + pad_after_seconds=pad_after, + ) + + +def test_process_event_rejects_negative_edge_padding_before_loading(tmp_path: Path): + generator = SpectrogramGenerator(quiet=True) + + with pytest.raises(ValueError, match="edge_padding_seconds"): + generator.process_event( + tmp_path / "missing.wav", + tmp_path, + event_time_seconds=10.0, + edge_padding_seconds=-0.1, + )