ESSE

Essential Source of Schemas and Examples (ESSE) contains data format definitions (schemas) and examples for common entities used in digital materials science (see refs. ¹, ² below).

Although the schemas are used to facilitate the operations of mat3ra.com, they are designed to be generic and can be used in other applications. The open-source packages developed by Mat3ra.com use the schemas available in this repository.

The latest variants of schemas and examples are available at schemas.mat3ra.com, which has three surfaces:

• Documentation — how ESSE is put together and why: the schema layering, entity anatomy, categorization, conventions and the build pipeline. Start here if you are new.
• Explorer — a file browser over every resolved schema and example.
• Ontology — an interactive map of the ontology: every entity type and every relationship the schemas declare between them (extends, contains, variant), laid out by architectural layer — primitives at the centre, root entities around them, catalogues on the rim. Search for a schema, fly to it, and follow what it extends, contains and is used by.

The map is built from graph.json, an extracted reference graph that is also published at schemas.mat3ra.com/graph.json and doubles as a lint over the corpus.

ESSE has a dual-nature as both a Python and a Node.js package.

1. Installation

1.1. Python

ESSE is compatible with Python 3.8+.

1.1.1. PyPI

pip install mat3ra-esse

1.1.2. Repository

virtualenv .venv
source .venv/bin/activate
pip install -e PATH_TO_ESSE_REPOSITORY

1.2. Node

1.2.1. NPM

npm install @mat3ra/esse

2. Usage

ESSE contains separate but equivalent interfaces for Python and Javascript. The package provides ESSE class that can be initialized and used as below.

2.1. Usage in Python

from mat3ra.esse import ESSE

helper = ESSE()
schema = helper.get_schema_by_id("material")

# standalone schema validation and cleaning
from mat3ra.esse import validate_and_clean

data = {"a": "x", "b": 1, "c": "drop"}
schema = {
    "type": "object",
    "properties": {
        "a": {"type": "string"},
        "b": {"type": "integer"}
    }
}

validate_and_clean(data, schema)
print(data) # {"a": "x", "b": 1}

# alternative: schema-aware cleaning via generated pydantic models
# also see how it is used in ade: https://github.com/mat3ra/ade/blob/main/src/py/mat3ra/ade/application.py
from pydantic import ConfigDict
from mat3ra.esse.models.software.application import ApplicationSchemaBase


class Application(ApplicationSchemaBase):
    # drop keys the schema doesn't declare at construction time
    model_config = ConfigDict(extra="ignore")


config = {"name": "espresso", "version": "6.3", "buildConfig": {"moduleName": "6.3-gnu"}}
app = Application(**config)
print(app.model_dump(exclude_none=True)) # {"name": "espresso", "version": "6.3", ...}

2.2. Usage in Node/JS/TS

const { ESSE } = require("@mat3ra/esse/lib/js/esse");

const helper = new ESSE();
const schema = helper.getSchemaById("material");

# schema validation and cleaning
const { validateAndClean } = require("@mat3ra/esse/lib/js/esse");

const data = {"a": "x", "b": 1, "c": "drop"};
const schema = {
    "type": "object",
    "properties": {
        "a": {"type": "string"},
        "b": {"type": "integer"}
    }
}

const result = validateAndClean(data, schema);
console.log(result); // {"a": "x", "b": 1}

3. Directory Structure

ESSE contains 3 main directories, schema, example and src outlined below, plus docs (documentation sources) and plan (design documents).

3.1. Schema

The schema directory contains the schemas specifying the rules to structure data. A set of core schemas, outlined below, are defined to facilitate the schema modularity.

• Primitive directory contains a set of custom primitives that extends default standard primitive types allowed by schema, such as String and Number. Primitives are solely defined by the default primitives and can not be re-constructed from each other.
• Abstract directory contains unit-less schemas that are constructed from default and custom primitives.
• Reusable directory contains the schemas that are widely used in other schemas to avoid duplication, constructed from the abstract and primitive schemas.
• Reference directory contains the schemas defining the rules to structure the references to data sources.

3.2. Example

This directory contains the examples formed according to the schemas and implements the same directory structure as the schema directory.

3.3. src

This directory contains Python and Javascript interfaces implementing the functionality to access and validate schemas and examples.

3.4. docs

Markdown sources for the concept documentation, rendered to HTML at deploy time. Pages may embed generated fragments (<!-- generated:name -->) that are expanded from the entity graph, so counts and relationship listings cannot drift from the schemas.

3.5. plan

Design documents, filed by status (upcoming/, review/, implemented/, context/) per the convention in mat3ra/agents. See plan/README.md.

Footnotes

1. Data-centric online ecosystem for digital materials science ↩

2. CateCom: A Practical Data-Centric Approach to Categorization of Computational Models ↩

This is the beginning of the README. Read the rest at the original. Read the full README ↗