OA Service

The OA service provides a comprehensive pipeline for environmental data processing, storage, and visualization. It is composed of four primary integrated systems, with scheduled jobs orchestrated by Prefect.

---

1. Front-end

The frontend is a high-performance dashboard that visualizes spatial and temporal data simultaneously, with focus on speed and performance.

Map Visualization

• Vector Tile Layering: Built the logic to overlay the tiles generated by the process service onto the map.
• Interactive Probing: Implemented "click-to-extract" functionality, where clicking the map triggers the API to fetch deep-dive data for that specific location.

Global Chart Footer

• Multi-Series Plotting: Heavily customized ECharts integration to support multiple lines. It plots model data and sensor data (e.g., "Sensor Temperature") on the same axis.
• Dynamic Updating: Built logic to add/remove lines on the fly as users toggle different sensors or variables.

---

2. API Services

The API acts as the bridge between ClickHouse and the frontend, providing on-demand data extraction. It never reads NetCDF: all data comes from ClickHouse.

Data Extraction Endpoints

• extractTimeseries.py: Pulls historical data for a specific coordinate (lat/lon) across time for ECharts graphs.
• extract_profile.py: Generates vertical "depth profiles," showing variable changes from surface to seabed.
• On-Demand Computation: Designed to scale with computational demand, performing real-time slicing and interpolation of massive datasets as user requests arrive.

External Sensor Integration

• extractSensorTimeseries.py: Serves sensor observations from ClickHouse (ingested hourly from ERDDAP and ONC by the sensors service).

Variable Metadata

• Manages logic in variables.py to serve unit definitions, display names, and color scales, ensuring consistency across all services.

---

3. Processing

The "engine" that transforms raw external data into precisely-tiled, color-corrected images and derived values.

ERDDAP Ingestion & Management

• Intelligent Fetching: Logic in SSC/downloader.py for ERDDAP downloads with automatic backfilling and compression.
• State Tracking: ClickHouse integration to track download status and manage re-queuing.

Imaging & Tiling

• Tiling: Developed nc2tile.py to convert NetCDF grids into Mapbox-compatible tiles.
• WebP Pipeline: SSC/imaging.py renders tiles with PST/UTC timezone handling and consistent color scaling.

Derived Calculations

• Biogeochemical Models: The live pipeline derives pH, Aragonite Saturation (Omega), and Inorganic Carbon via PyCO2SYS in process/SSC/compute.py.
• Shared Memory Scaling: A standalone calc_carbon_grid_shm_memmap_year_aware.py offers shared-memory (shm_memmap) high-concurrency computation for ad-hoc, DB-free runs.

Statistics & Climatology

• Climatology: Day-of-year climatology (mean/min/max) is aggregated in ClickHouse (SalishSeaCast_daily) and served by the API's /extract_climateTimeseries; the Analysis Builder derives trends, thresholds, and anomalies client-side.

---

4. Database (ClickHouse)

Technology & Foundation

• Spatial Support: The curvilinear model grid (grid_SSC) and sensor locations are stored as ClickHouse tables, queried with geoDistance() for nearest-neighbor lookups.
• State Machine Logic: SalishSeaCast_status tracks the processing queue, letting workers coordinate downloading, computing, and tiling tasks.

Pipeline Orchestration

• Status Evolution: Managed processing statuses (e.g., downloading, computing, imaging).
• Self-Healing Queue: Failed or interrupted tasks are automatically re-queued.

---

5. Scheduling (Prefect)

Scheduled jobs run as Prefect flows reporting to the shared CIOOS Pacific server at https://pipelines.cioospacific.ca. Prefect is production-only: dev has no Prefect server or schedulers, so run the pipeline with the CLI (python -m SSC.cli ...).

Job	Flow code	Container (compose file)	Deployment	Schedule
SalishSeaCast model pipeline	process/SSC/flows.py	scheduler (docker-compose.prod.process.yml, tools profile)	OceanECO-SSC	RUN_CRON, every 3 hours
ERDDAP sensor ingestion	sensors/flows.py	sensors-scheduler (docker-compose.prod.api.yml)	OceanECO-sensors-ERDDAP	ERDDAP_CRON, hourly on the hour
ONC sensor ingestion	sensors/flows.py	sensors-scheduler (docker-compose.prod.api.yml)	OceanECO-sensors-ONC	ONC_CRON, hourly at half past

• Our own containers: Each scheduler serves its flows with serve() and runs them in-process, so the shared server only schedules and records runs. All deployments carry the oceaneco tag because the server is shared with other apps.
• Configuration: Both compose files require PREFECT_API_URL (https://pipelines.cioospacific.ca/api) and take the server's basic auth from PREFECT_AUTH_STRING, so always pass --env-file.
• One run at a time: A run that comes due while the previous one is still going is cancelled, not queued.
• Pausing: RUN_SCHEDULE_PAUSED / SENSORS_SCHEDULE_PAUSED are re-applied on every container start, so a pause set in the UI lasts only until the next restart. Stop the container for a pause that holds.
• Updating: After a code change, rebuild and recreate the scheduler container. Keep the prefect== pin in process/pyproject.toml and sensors/pyproject.toml at or below the server's version.

Filling a missed SSC date

Scheduled runs only queue dates newer than the latest date in SalishSeaCast_status, so a day that was skipped in the past is never picked up on its own. Process it explicitly on the process machine (or use Run → custom in the Prefect UI with date set, which won't overlap a scheduled run):

C="docker compose -f docker-compose.prod.process.yml --env-file .env.process.remote exec scheduler uv run python -m SSC.cli"
$C status --date YYYY-MM-DD   # where it stopped (no rows = never queued)
$C run --date YYYY-MM-DD      # seeds missing rows, then resumes from the date's current stage through sync

Don't start it while a scheduled run is in progress. Add --force only if rows are stuck at failed_* after exhausting retries or the date must be redone from scratch; if the API already holds partial data for that date, clear it first (see docs/clickhouse-ops.md).

Details: process/README.md (SSC pipeline) and sensors/README.md (sensor ingestion).