OA Service
The OA service provides a comprehensive pipeline for environmental data processing, storage, and visualization. It is composed of four primary integrated systems, with scheduled jobs orchestrated by Prefect.
1. Front-end
The frontend is a high-performance dashboard that visualizes spatial and temporal data simultaneously, with focus on speed and performance.
Map Visualization
- Vector Tile Layering: Built the logic to overlay the tiles generated by the
process service onto the map.
- Interactive Probing: Implemented "click-to-extract" functionality, where clicking the map triggers the API to fetch deep-dive data for that specific location.
Global Chart Footer
- Multi-Series Plotting: Heavily customized ECharts integration to support multiple lines. It plots model data and sensor data (e.g., "Sensor Temperature") on the same axis.
- Dynamic Updating: Built logic to add/remove lines on the fly as users toggle different sensors or variables.
2. API Services
The API acts as the bridge between ClickHouse and the frontend, providing on-demand data extraction. It never reads NetCDF: all data comes from ClickHouse.
Data Extraction Endpoints
extractTimeseries.py: Pulls historical data for a specific coordinate (lat/lon) across time for ECharts graphs.extract_profile.py: Generates vertical "depth profiles," showing variable changes from surface to seabed.- On-Demand Computation: Designed to scale with computational demand, performing real-time slicing and interpolation of massive datasets as user requests arrive.
External Sensor Integration
extractSensorTimeseries.py: Serves sensor observations from ClickHouse (ingested hourly from ERDDAP and ONC by the sensors service).
Variable Metadata
- Manages logic in
variables.py to serve unit definitions, display names, and color scales, ensuring consistency across all services.
3. Processing
The "engine" that transforms raw external data into precisely-tiled, color-corrected images and derived values.
ERDDAP Ingestion & Management
- Intelligent Fetching: Logic in
SSC/downloader.py for ERDDAP downloads with automatic backfilling and compression.
- State Tracking: ClickHouse integration to track download status and manage re-queuing.
Imaging & Tiling
- Tiling: Developed
nc2tile.py to convert NetCDF grids into Mapbox-compatible tiles.
- WebP Pipeline:
SSC/imaging.py renders tiles with PST/UTC timezone handling and consistent color scaling.
Derived Calculations
- Biogeochemical Models: The live pipeline derives pH, Aragonite Saturation (Omega), and Inorganic Carbon via PyCO2SYS in
process/SSC/compute.py.
- Shared Memory Scaling: A standalone
calc_carbon_grid_shm_memmap_year_aware.py offers shared-memory (shm_memmap) high-concurrency computation for ad-hoc, DB-free runs.
Statistics & Climatology
- Climatology: Day-of-year climatology (mean/min/max) is aggregated in ClickHouse (
SalishSeaCast_daily) and served by the API's /extract_climateTimeseries; the Analysis Builder derives trends, thresholds, and anomalies client-side.
4. Database (ClickHouse)
Technology & Foundation
- Spatial Support: The curvilinear model grid (
grid_SSC) and sensor locations are stored as ClickHouse tables, queried with geoDistance() for nearest-neighbor lookups.
- State Machine Logic:
SalishSeaCast_status tracks the processing queue, letting workers coordinate downloading, computing, and tiling tasks.
Pipeline Orchestration
- Status Evolution: Managed processing statuses (e.g.,
downloading, computing, imaging).
- Self-Healing Queue: Failed or interrupted tasks are automatically re-queued.
5. Scheduling (Prefect)
Scheduled jobs run as Prefect flows reporting to the shared CIOOS Pacific server at https://pipelines.cioospacific.ca. Prefect is production-only: dev has no Prefect server or schedulers, so run the pipeline with the CLI (python -m SSC.cli ...).
| Job | Flow code | Container (compose file) | Deployment | Schedule |
|---|
| SalishSeaCast model pipeline | process/SSC/flows.py | scheduler (docker-compose.prod.process.yml, tools profile) | OceanECO-SSC | RUN_CRON, every 3 hours |
| ERDDAP sensor ingestion | sensors/flows.py | sensors-scheduler (docker-compose.prod.api.yml) | OceanECO-sensors-ERDDAP | ERDDAP_CRON, hourly on the hour |
| ONC sensor ingestion | sensors/flows.py | sensors-scheduler (docker-compose.prod.api.yml) | OceanECO-sensors-ONC | ONC_CRON, hourly at half past |
- Our own containers: Each scheduler serves its flows with
serve() and runs them in-process, so the shared server only schedules and records runs. All deployments carry the oceaneco tag because the server is shared with other apps.
- Configuration: Both compose files require
PREFECT_API_URL (https://pipelines.cioospacific.ca/api) and take the server's basic auth from PREFECT_AUTH_STRING, so always pass --env-file.
- One run at a time: A run that comes due while the previous one is still going is cancelled, not queued.
- Pausing:
RUN_SCHEDULE_PAUSED / SENSORS_SCHEDULE_PAUSED are re-applied on every container start, so a pause set in the UI lasts only until the next restart. Stop the container for a pause that holds.
- Updating: After a code change, rebuild and recreate the scheduler container. Keep the
prefect== pin in process/pyproject.toml and sensors/pyproject.toml at or below the server's version.
Filling a missed SSC date
Scheduled runs only queue dates newer than the latest date in SalishSeaCast_status, so a day that was skipped in the past is never picked up on its own. Process it explicitly on the process machine (or use Run → custom in the Prefect UI with date set, which won't overlap a scheduled run):
C="docker compose -f docker-compose.prod.process.yml --env-file .env.process.remote exec scheduler uv run python -m SSC.cli"
$C status --date YYYY-MM-DD # where it stopped (no rows = never queued)
$C run --date YYYY-MM-DD # seeds missing rows, then resumes from the date's current stage through sync
Don't start it while a scheduled run is in progress. Add --force only if rows are stuck at failed_* after exhausting retries or the date must be redone from scratch; if the API already holds partial data for that date, clear it first (see docs/clickhouse-ops.md).
Details: process/README.md (SSC pipeline) and sensors/README.md (sensor ingestion).
