metasalmon

The Problem

You've spent years collecting salmon data. But when you try to share it:

• Colleagues ask "What does SPAWN_EST mean?"
• Combining datasets fails because everyone uses different column names
• Your future self opens old data and can't remember what the codes mean
• Other researchers can't use your data without emailing you for explanations

The Solution

metasalmon wraps your salmon data with a data dictionary that travels with it—explaining every column, every code, and linking to standard scientific definitions. These definitions come from the Salmon Domain Ontology (shared layer) and the DFO Salmon Ontology (DFO-specific layer), alongside other published controlled vocabularies, and the data is packaged according to the Salmon Data Package Specification. The preferred review workflow now happens inside the R package: metasalmon can retrieve candidate terms, optionally ask an LLM to review them, write draft REVIEW-prefixed IRIs into the package metadata, and then hand you back a package you finish reviewing in R: review_semantics() prints each candidate with its definition and the exact call that accepts it, and review_metadata() prints the call that fills every remaining required field.

Integration context: See the Salmon Data Integration System overview page (https://br-johnson.github.io/salmon-data-integration-system/) and walkthrough video (https://youtu.be/B0Zqac49zng?si=VmOjbfMDMd2xW9fH).

Think of it like adding a detailed legend to your spreadsheet that never gets lost.

What You Get

Your Data	+ metasalmon	= Data Package
Raw CSV files	Data dictionary	Self-documenting dataset
Cryptic column names	Clear descriptions	Anyone can understand it
Inconsistent codes	Linked to standards	Works with other datasets

Quick Example

Before you start, do the one-time Setup and Credentials check so GitHub installs work cleanly and any optional LLM provider is ready in advance.

Install, run one function on the bundled Fraser Coho 2023-2024 example (173 rows), then finish the review in R.

# Install from GitHub (recommended)
# install.packages("remotes")
# remotes::install_github("salmon-data-mobilization/metasalmon")

library(metasalmon)

data_path <- system.file("extdata", "nuseds-fraser-coho-2023-2024.csv", package = "metasalmon")
fraser_coho <- readr::read_csv(data_path, show_col_types = FALSE)

pkg_path <- create_sdp(
  fraser_coho,
  path = "fraser-coho-2023-2024-sdp",
  dataset_id = "fraser-coho-2023-2024",
  table_id = "escapement",
  check_updates = FALSE,
  overwrite = TRUE
)

# Finish the review without leaving R:
review <- review_semantics(pkg_path)   # decide the seeded IRIs
apply_sdp_semantics(pkg_path, review)
review_metadata(pkg_path)              # fill the remaining required fields
validate_salmon_datapackage(pkg_path, require_iris = TRUE)

create_sdp() is the main path. It writes the canonical metadata/*.csv files plus your data/*.csv tables, adds a short review checklist, writes prefilled semantic drafts directly into metadata/column_dictionary.csv and metadata/tables.csv only where target fields were blank, and keeps semantic_suggestions.csv as a fallback shortlist when you want more context or a better match. Code-level semantic seeding stays conservative by default for factor and low-cardinality character source columns. Before SPSR/EDH upload, run validate_salmon_datapackage(pkg_path, require_iris = TRUE) to catch package/data/codes mismatches in one pass. In interactive use create_sdp() can also mention an available package update; set check_updates = FALSE to skip that check.

Review the semantics in R, not in a spreadsheet

create_sdp() seeds draft REVIEW:-prefixed IRIs. You can review them without leaving R, and — this is the point — the review becomes an ordinary script you can re-run, diff, and hand to someone else:

review <- review_semantics(pkg_path)
review
#> ── spawners · spawner_count · variable ────────────────────────────────
#>    field:   column_dictionary.csv · term_iri
#>    current: REVIEW: https://w3id.org/smn/SpawnerAbundance
#>
#>   [1] Spawner Abundance   smn   score 4.9
#>       The number of mature salmon returning to spawn in a stream.
#>       https://w3id.org/smn/SpawnerAbundance
#>       review <- accept_suggestion(review, "spawner_count", "variable", rank = 1)
#>
#>   [2] Escapement   smn   score 3.2
#>       Fish that escape the fishery and reach the spawning grounds.
#>       https://w3id.org/smn/Escapement
#>       review <- accept_suggestion(review, "spawner_count", "variable", rank = 2)

The console prints the exact call that decides each candidate. Paste it into your script — that paste is the audit trail. There is no interactive prompt, deliberately: a prompt would make the decision as unreproducible as the spreadsheet it replaces.

review <- accept_suggestion(review, "spawner_count", "variable", rank = 1)
review <- reject_suggestion(review, "gear_code", "variable",
                            reason = "no candidate describes gear type")

apply_sdp_semantics(pkg_path, review)
validate_salmon_datapackage(pkg_path, require_iris = TRUE)

apply_sdp_semantics() strips the REVIEW: prefix from decided fields, clears rejected ones, leaves undecided slots untouched, records the decision in semantic_suggestions.csv, and does not touch your data CSV bytes. Running it twice produces identical bytes.

Decisions persist in the package, so a review you stop halfway through picks up where you left it rather than asking the same questions again.

review_semantics() shows shortlists, not gaps — a slot retrieval found nothing for never enters its queue, and free-text fields have no shortlist at all. review_metadata() is the other half, and it reads the package against the rules that decide strict validation rather than against the suggestions:

review_metadata(pkg_path)
#> ── dataset.csv ────────────────────────────────────────────────────────
#>    creator: placeholder text, refused by strict validation
#>       MISSING METADATA: add creator, team, or originating program.
#>    contact_email: placeholder text, refused by strict validation
#>       MISSING METADATA: add primary contact email.
#>
#>    set_sdp_dataset(pkg_path,
#>      creator = "<add creator, team, or originating program>",
#>      contact_email = "<add primary contact email>"
#>    )

Replace each <…> with the real value and paste. Pasting one unedited is refused, because a package whose creator reads <add creator…> would pass strict validation while saying nothing. The setters are set_sdp_dataset(), set_sdp_table(), set_sdp_column() and set_sdp_code(); each keeps datapackage.json in step in the same transactional write.

When review_metadata() reports nothing, require_iris = TRUE passes — so a package created by create_sdp() can be taken all the way to strict validation without opening a spreadsheet at any point.

Reviewing in a spreadsheet still works and remains supported; the R path is the recommended one because it leaves a record of why each value was chosen.

Bundled NuSEDS Example

The default quickstart and get-started flow use nuseds-fraser-coho-2023-2024.csv, a 173-row Fraser coho slice derived from the official Open Government Canada Fraser and BC Interior workbook.

Open Government Canada record: https://open.canada.ca/data/en/dataset/c48669a3-045b-400d-b730-48aafe8c5ee6

A smaller nuseds-fraser-coho-sample.csv file is still bundled for tiny smoke tests, but it is no longer the default walkthrough dataset.

The package also now ships a matching starter dictionary for the fuller example (system.file("extdata", "nuseds-fraser-coho-2023-2024-column_dictionary.csv", package = "metasalmon")), which is useful when you want a ready-made context file for the package-native LLM review path.

See example-data-README.md for the record/resource URLs, row counts, licensing note, and the data-raw/ script that reproduces the 2023-2024 example.

To continue:

• Setup and Credentials — one-time GitHub credential setup for installs plus optional LLM provider setup.
• 5-Minute Quickstart — create the full package with metadata and export it.
• After Excel Review — reload the reviewed package, detect unresolved ontology gaps, route shared vs DFO-specific requests, and finish publication.
• Publishing Data Packages — manual package assembly path when you are not continuing from create_sdp().
• Linking to Standard Vocabularies — pick term_iri, property_iri, and entity_iri with confidence.

Need the full context-file workflow? See LLM Review With Context Files.

Package-native LLM semantic review (optional)

If you want an LLM to judge the shortlisted semantic matches directly from R, keep the deterministic search path and add an opt-in review pass:

context_files <- c(
  file.path(pkg_path, "metadata", "column_dictionary.csv"),
  "README.md",
  "data-dictionary.xlsx",
  "methods-report.pdf"
)

suggested <- suggest_semantics(
  df = fraser_coho,
  dict = infer_dictionary(fraser_coho, dataset_id = "fraser-coho-2023-2024", table_id = "escapement"),
  llm_assess = TRUE,
  llm_provider = "openrouter",
  llm_context_files = context_files
)

suggestions <- attr(suggested, "semantic_suggestions")
assessments <- attr(suggested, "semantic_llm_assessments")

# In create_sdp(...), any auto-applied semantic IRI drafts are written back
# into the package metadata as REVIEW-prefixed values for manual cleanup,
# including table-level observation-unit selections in metadata/tables.csv.

This keeps find_terms() as the canonical candidate generator. For each measurement column, the LLM reviews the variable, property, entity, unit, constraint, and method shortlists together. Generic column, code, table, and dataset targets keep their existing per-target review path. Deterministic validators can downgrade an unsupported accept to review; the original model confidence remains in the assessment as provenance.

Only accepted variable, property, entity, and unit candidates can be written back automatically. Drafts are stored as REVIEW: <iri> so you can confirm or replace them in Excel rather than treating them as final. Constraint and method assessments always remain manual, as do dataset keywords and code-level suggestions. Compatible table-level observation-unit matches can still be written into metadata/tables.csv.

Omitting sources (or semantic_sources on the package-creation helpers) uses role-aware defaults. Supplying sources explicitly creates a strict allowlist for both initial retrieval and the single retry round. For example, sources = "smn" cannot introduce a QUDT unit candidate.

The target-level semantic_llm_assessments table has a stable 30-column schema. Its final two fields record whether a term request was escalated from reject_shortlist and why a retry query was rejected. An exact retry duplicate, after case and whitespace normalization, remains a retry_search decision with llm_retry_query_rejection_reason = "duplicate_original_query" and does not trigger another query-generation, search, or reassessment call.

llm_context_files must be a character vector of existing local file paths. Do not pass a tibble returned by readr::read_csv(), an xml2 document returned by read_html(), or another parsed object; those inputs now fail early instead of being silently ineffective. Supplying context also never enables an LLM call: set llm_assess = TRUE explicitly, or the package warns that the context will be ignored and continues with deterministic retrieval only. Use llm_context_text for inline text that is not stored in a file.

Supported paths include text and notes (.md, .txt, .rst), delimited/data files (.csv, .tsv, .json, .yaml, .yml), source and notebook-style files (.R, .Rmd, .qmd), HTML (.htm, .html), DOCX (.docx), Excel workbooks (.xls, .xlsx, .xlsm via readxl), and PDF reports (.pdf via pdftools). Validation should only pass after the REVIEW prefix is removed. When you use llm_provider = "openrouter" without specifying llm_model, metasalmon defaults to openrouter/free.

For the full workflow across dataset.csv, tables.csv, column_dictionary.csv, codes.csv, and the post-review EDH rebuild, use the LLM Review With Context Files guide.

The quickstart path does not require an API key. Only set up one of these providers when you want create_sdp(..., llm_assess = TRUE) or suggest_semantics(..., llm_assess = TRUE).

For DFO internal users on the internal network or VPN, open https://chapi-dev.intra.azure.cloud.dfo-mpo.gc.ca/, click the user icon in the bottom left, open Settings, click Show next to API Keys, and copy the key value. Then run:

file.edit("~/.Renviron")
CHAPI_API_KEY="paste key here"

chapi defaults to ollama2.mistral:7b and https://chapi-dev.intra.azure.cloud.dfo-mpo.gc.ca/api. Optional overrides are CHAPI_MODEL and CHAPI_BASE_URL. gpt-oss:latest is also supported, but it can be slower to warm up and now gets a longer automatic timeout.

For external users, OpenRouter is the easiest free option:

file.edit("~/.Renviron")
OPENROUTER_API_KEY="paste key here"

llm_provider = "openrouter" defaults to openrouter/free.

If you already have OpenAI API credits, use:

file.edit("~/.Renviron")
OPENAI_API_KEY="paste key here"

Then pass an OpenAI model explicitly, for example llm_model = "gpt-4.1-mini".

Recommended workflow

For the current package-native review path, use this order:

1. Run create_sdp(...) to create the Salmon Data Package.
2. If you want semantic review, set llm_assess = TRUE.
3. Review the seeded IRIs. Preferred: review_semantics(pkg_path) in the console, then paste the printed accept_suggestion() / reject_suggestion() calls into a script and finish with apply_sdp_semantics(pkg_path, review). Alternative: open README-review.txt and edit metadata/column_dictionary.csv and metadata/tables.csv in a spreadsheet.
4. For any prefilled or REVIEW:-prefixed IRI, click through and read the term definition before keeping it.
5. Run review_metadata(pkg_path) for the free-text fields and for any required IRI nothing was suggested for, and paste the set_sdp_*() calls it prints. Steps 3 and 5 together are what make step 9 reachable without a spreadsheet.
6. Use semantic_suggestions.csv only as a fallback shortlist if you are unsure or want a better match.
7. If no candidate fits, request a new term instead of forcing a bad match:
   • shared cross-organization/domain terms -> https://github.com/salmon-data-mobilization/salmon-domain-ontology/issues/new/choose
   • DFO-specific policy/operations terms -> https://github.com/dfo-pacific-science/dfo-salmon-ontology/issues/new/choose
8. Follow the After Excel Review guide to reload the package, detect unresolved semantic gaps, and produce a concrete shared-vs-DFO term-request plan.
9. If you are preparing EDH metadata, regenerate the XML from the reviewed package with write_edh_xml_from_sdp(pkg_path) (the reviewed-package wrapper around the canonical edh_build_hnap_xml() builder). It now refuses to rebuild while REVIEW: markers or unresolved dataset/table placeholder text remain.
10. Re-run validation with validate_salmon_datapackage(pkg_path, require_iris = TRUE).
11. For KNB, add the reviewed metadata/eml-mapping.yml sidecar, generate schema-valid EML 2.2.0 with write_eml_from_sdp(pkg_path), and inspect the credential-free restricted-review plan with . A live private review still creates persistent production objects; it verifies authenticated exact readback and anonymous non-disclosure, and explicitly disables DataONE peer replication so the restricted bytes remain KNB-only. The plan contains the original data files plus each validated canonical SDP artifact as a named object with its package-relative path. No ZIP is required, and the KNB-specific EML mapping and publication receipts remain local. KNB has no separate server-side draft state, and metasalmon does not mint a DOI. DOI minting and public release are a later, per-metadata-version decision in KNB. Referenced vocabulary rows labelled as review candidates are blocked; the transformation workflow must separately pin and verify the governed vocabulary release. Corrections use a new sidecar and the preceding verified manifest, preserving the KNB series instead of overwriting immutable objects.

In other words: create -> decide the IRIs -> fill the remaining metadata -> check gaps -> validate -> build the required export -> dry-run -> publish. All of it from R; the spreadsheet is the fallback, not the route.

Who Is This For?

If you are...	Start here
A biologist who wants to share data	5-Minute Quickstart
Finished Excel review and need to publish	After Excel Review
Curious how it works	How It Fits Together
A data steward standardizing datasets	Data Dictionary & Publication
Reading CSVs from private GitHub repos	GitHub CSV Access

Video Walkthrough

Watch: Creating Your First Data Package

Installation

# Install from GitHub
install.packages("remotes")
remotes::install_github("salmon-data-mobilization/metasalmon")

What's In a Data Package?

When you create a package, you get a folder containing:

my-data-package/
  +-- README-review.txt         # Step-by-step review checklist for manual Excel cleanup
  +-- semantic_suggestions.csv  # Detailed semantic evidence + LLM review trail (when present)
  +-- datapackage.json          # Machine-readable export
  +-- metadata/
  |   +-- dataset.csv           # Dataset-level metadata (canonical)
  |   +-- tables.csv            # Table-level metadata and file paths
  |   +-- column_dictionary.csv # What each column means
  |   +-- codes.csv             # What each code value means (if applicable)
  +-- data/
      +-- escapement.csv        # Your data table(s)

Anyone opening this folder - whether a colleague, a reviewer, or your future self - can immediately understand your data. The metadata/*.csv files are the canonical package metadata; datapackage.json is a derived interoperability export. When you share the package, send the whole folder (or a zip of the whole folder), not just datapackage.json.

Key Features

For everyday use:

• Automatically generate data dictionaries from your data frames
• Validate that your dictionary is complete and correct
• Create shareable packages that work across R, Python, and other tools
• Read CSVs directly from private GitHub repositories

For data stewards (optional):

• Link columns to standard DFO Salmon Ontology terms
• Add I-ADOPT measurement metadata (property, entity, unit, constraint)
• Use AI assistance to help write descriptions
• Suggest Darwin Core Data Package table/field mappings for biodiversity data
• Opt in to DwC-DP export hints via suggest_semantics(..., include_dwc = TRUE) while keeping the Salmon Data Package as the canonical deliverable.
• Generate HNAP-aware EDH metadata XML for DFO Enterprise Data Hub upload workflows via the canonical edh_build_hnap_xml() builder, the reviewed-package helper write_edh_xml_from_sdp(), or create_sdp(..., include_edh_xml = TRUE). Create-time XML is a draft when review markers remain; finalize the metadata and rebuild it before submission.
• Generate deterministic, schema-validated EML 2.2.0 from a strictly reviewed SDP with write_eml_from_sdp(), then produce an offline exact-object KNB manifest or explicitly run a verified private/public DataONE deposit with publish_sdp_to_knb(). Expanded KNB plans retain the original data resources and publish the closed canonical SDP inventory as named package-relative objects without a ZIP.
• Role-aware vocabulary search with find_terms() and sources_for_role():
  • Units: QUDT preferred, then NVS P06
  • Salmon-domain roles: shared SMN terms first, then GCDFO DFO-specific terms where needed
  • Entities/taxa: SMN and GCDFO first, then GBIF and WoRMS taxon resolvers
  • Properties/variables/methods: shared salmon-domain terms first, then broader ontology fallbacks
  • Cross-source agreement boosting for high-confidence matches
• Per-source diagnostics, scoring, and optional rerank explain why find_terms() matches rank where they do and expose failures, so you can tune role-aware queries with confidence.
• End-to-end semantic QA loop with fetch_salmon_ontology() + validate_semantics(), plus deduplicate_proposed_terms() to prevent term proliferation before opening ontology issues.
• Optional package-native LLM review for semantic suggestions: suggest_semantics(..., llm_assess = TRUE) reviews all six measurement roles as one bundle, keeps explicit source lists strict across retries, and uses deterministic validators to downgrade unsupported acceptances.
• Structured ontology-gap handling combines deterministic candidate gaps with final LLM request_new_term decisions, preserves escalation evidence, and renders curator-reviewable request bodies for shared SMN, DFO-specific GCDFO, or local profile governance.
• NuSEDS method crosswalk helpers: and for mapping legacy values to canonical method families.

Getting Help

• Frequently Asked Questions
• Glossary of Terms
• Report a bug
• Request a feature
• Salmon Domain Ontology
• Salmon Data Package Specification

How It Fits Together

metasalmon brings together four pieces: your raw data, the Salmon Data Package specification, the Salmon Domain Ontology (and other vocabularies), optional in-package LLM review, and the review files written into the package itself. When you finish the workflow, the dictionary, dataset/table metadata, and optional code lists are already aligned with the specification, which makes the package ready to publish. The ontology keeps the column meanings consistent, and the package-native review workflow helps draft descriptions and term choices without forcing you into a separate prompt-export side path.

The high-level flow is:

• Start here: create_sdp() takes raw tables, infers the package metadata, writes a review-ready package, gives you a checklist, auto-fills top column/table semantic suggestions only where fields are blank, and keeps default code-level semantic seeding conservative by limiting it to factor and low-cardinality character source columns.
• Advanced/manual path: write_salmon_datapackage() is for cases where you already assembled dataset.csv, tables.csv, column_dictionary.csv, and optional codes.csv yourself.
• Raw tables lead into metadata/column_dictionary.csv (and metadata/codes.csv when there are categorical columns).
• Dataset/table metadata fill the required specification fields (title, description, creator, contact, etc.), so the package folder can be shared or uploaded.
• The Salmon Domain Ontology and published vocabularies supply term_iri/entity_iri links that describe what each column and row represents.
• Post-review publication helpers let you reopen the package, re-run semantic checks, detect unresolved ontology gaps, and route curator-reviewed drafts to shared SMN, DFO-specific GCDFO, or a local profile.
• write_salmon_datapackage() consumes the metadata, dictionary, codes, and data to write the files in the Salmon Data Package format; the preferred review loop is now the package itself plus README-review.txt / semantic_suggestions.csv, not an external prompt-export workflow.

For Developers

Installation for Development

install.packages(c("devtools", "roxygen2", "testthat", "knitr", "rmarkdown",
                   "tibble", "readr", "jsonlite", "cli", "rlang", "dplyr",
                   "tidyr", "purrr", "withr", "frictionless"))

Build and Check

devtools::document()
devtools::test()
devtools::check()
devtools::build_vignettes()

Rscript scripts/build-pkgdown.R

# Canonical source-tarball build path (writes into the repo root, not ../)
./scripts/build-package.sh

Package Structure

• R/: Core functions for dictionary and package operations
• inst/extdata/: Example data files and templates
• tests/testthat/: Automated tests
• vignettes/: Long-form documentation
• docs/: pkgdown site output

Internal Review Architecture

• R/semantic-suggestions.R owns the internal semantic-suggestion row interface: target keys, candidate row normalization, column-term filtering, and LLM assessment merge rules.
• R/llm-review-adapter.R owns shared LLM/chat review parsing, validation, and assessment row construction.
• R/semantics-helpers.R, R/llm-semantic-helpers.R, and R/chat-decomposition.R keep the public workflows while delegating shared row and review behavior to those internal modules.

Salmon Domain Ontology

This package can link your data to the Salmon Domain Ontology for shared terms and to the DFO Salmon Ontology for DFO-specific terms. Canonical IRIs are explicit: SMN uses https://w3id.org/smn/<Term> and GCDFO uses https://w3id.org/gcdfo/salmon#<Term>. metasalmon does not silently rewrite legacy salmon: IRIs.

See the Reusing Standards for Salmon Data Terms guide for details.