suMEDS

suMEDS (pronounced "summed-s", past tense of "sum" + s) creates a compact code catalog for performing tasks Medical Event Data Standard (MEDS) datasets. MEDS data should be "summed" in this way in a storage-efficient format, without the need to access the full dataset.

It can be seen as a form of "extended metadata," as we sometimes need more information than vanilla MEDS metadata provides, but don't want to expose our complete dataset (either because of privacy concerns, storage capacity, or token use efficiency for LLMs). It joins canonical descriptions and configurable parent, child, and sibling codes to event and unique-subject counts, then masks or removes rare codes before writing Parquet, CSV, or JSON.

[!WARNING] This package is in early development and is human-guided, but AI-agent-generated. It is not yet reviewed for production use. Please report issues and suggestions.

Polars scans, aggregations, joins, and writes stay lazy. This should mean that it could run with little resources on large MEDS datasets.

Install and run

Normal install:

pip install sumeds

Datasets with more than 2³² rows require Polars' 64-bit runtime:

pip install "sumeds[rt64]"

Development

uv sync
uv run suMEDS tests/resources/MIMICIV_DEMO/MEDS_cohort \
  --output code-summary.parquet \
  --min-subjects 20 \
  --split-columns

Or use YAML:

summary:
  per_split: false
  split_columns: true
  partitions: 256
privacy:
  min_subjects: 20
  min_split_subjects: 1
  rare_code_action: bucket
  rare_code_label: __RARE__
  round_counts_to: 5

uv run suMEDS /path/to/MEDS -o summary.parquet -c examples/summary.yaml
# The suffix selects CSV or JSON instead:
uv run suMEDS /path/to/MEDS -o summary.json -c examples/summary.yaml

CLI flags override YAML. The defaults bucket codes seen in fewer than 20 unique subjects and retain exact counts.

Optionally fill missing descriptions, all ancestors, OMOP concept IDs, domains, and standard-concept markers from OHDSI Athena. Child and sibling expansion are opt-in:

uv run suMEDS /path/to/MEDS -o summary.parquet \
  --athena-csv /path/to/athena --child-codes --child-depth 3 --sibling-codes
# Or use PostgreSQL through the installed psql client:
uv run suMEDS /path/to/MEDS -o summary.parquet \
  --athena-postgres postgresql://postgres@localhost/omop

Standalone enrichment avoids rescanning event data:

uv run suMEDS-enrich metadata/codes.parquet -o codes-enriched.parquet \
  --athena-csv /path/to/athena

The standalone command shows phase progress and reports Athena matches and before/after metadata coverage. Pass the input as -o to enrich it in place, for example a MEDS dataset's metadata/codes.parquet; the input is replaced only once the enriched table is complete.

The VOCABULARY//CODE//..., VOCABULARY//VOCABULARY_VERSION//CODE and OMOP-style VOCABULARY/CODE layouts are resolved, and so is a code's first existing parent reference (ICD9CM/428.0). Every existing parent reference gets its name and its ancestors. See How enrichment works for the steps and a worked example, and Athena enrichment for what is read from Athena, how ICD parents are added, and how Athena compares with the packs.

Rules and public ontology packs

Without Athena, or in addition to it, a sumeds-rules/0.1 file maps dataset codes to vocabulary codes and adds parents that exist in the same table:

schema: sumeds-rules/0.1
ontology:
  - id: icd10cm-diagnoses
    pattern: '^DIAGNOSIS//ICD//10//(?P<code>[A-Z][0-9][0-9A-Z][0-9A-Z]{0,4})$'
    vocabulary: ICD10CM
    normalize: [{pattern: '^(...)(.+)$', replace: '\1.\2'}]
    validate: pack            # pack | athena | none
crosswalk:
  - id: nwicu-lab-loinc
    pattern: '^LAB//(?P<key>[0-9]+)//'
    table: nwicu/loinc_parents
    key_column: itemid
    code_column: loinc_code
    vocabulary: LOINC
hierarchy:
  - {id: eicu-diagnosis-paths, pattern: '^(?P<prefix>DIAGNOSIS//)(?P<path>.+)$', group: path, separator: '|'}
  - {id: icd-prefix, pattern: '^(?P<p>DIAGNOSIS//ICD10_CN//)(?P<code>.+)$', group: code, mode: prefix, ignore: '.'}
  - {id: ontology-parents-in-dataset, from_ontology: true}

uv run suMEDS-enrich metadata/codes.parquet -o codes-enriched.parquet --rules rules.yaml

An ETL package can register its rules file under its MEDS-Extract spec name (entry-point group sumeds.rules), so --rules MC-MED finds it. With MEDS-Extract 0.8.1 or later, its spec can declare the sumeds_enrich stage, and then meds-extract-run alone runs the ETL and enriches its metadata in place:

This is the beginning of the README. Read the rest at the original. Read the full README ↗