shex.js

shex.js javascript implementation of Shape Expressions (try online)

install

npm install --save shex

test

npm ci
npm test

See building and testing below for the full development story.

branch-specific tests

The shex.js repo includes several branches for features that are in-flight in the ShEx Community Group. Each branch depends on the corresponding branch of the shexTest test suite. The package.json file for each branch SHOULD have that corresponding shexTest branch à la:

  "shex-test": "github:shexSpec/shexTest#extends"

The test harness prefers a ../shexTest sibling checkout and otherwise uses the shex-test npm dependency. If you test from a sibling clone, keep its branch aligned with your shex.js branch; the pre-commit hook (npm run check-branch-deps) whines when main isn't testing against shexTest#main.

validation

You can validate RDF data using the shex-validate executable or the validation library described below.

validation executable

Validate something in HTTP-land:

npx shex-validate \
    -x http://shex.io/examples/Issue.shex \
    -d http://shex.io/examples/Issue1.ttl \
    -s http://shex.io/examples/IssueShape \
    -n http://shex.io/examples/Issue1

That validates node http://shex.io/examples/Issue1 in http://shex.io/examples/Issue1.ttl against shape http://shex.io/examples/IssueShape in http://shex.io/examples/Issue.shex. The result is a JSON structure which tells you exactly how the data matched the schema.

{
  "type": "ShapeTest",
  "node": "http://shex.io/examples/Issue1",
  "shape": "http://shex.io/examples/IssueShape",
  "solution": {
...
  }
}

A result with "errors" tells you the data was invalid with respect to the schema. See the ShExJ primer for a description of ShEx validation and the ShExJ specification for more details about the results format.

relative resolution

shex-validate's -n and -s arguments are evaluated as IRIs relative to the (first) data and schema sources respectively. The above invocation validates the node <Issue1> in http://shex.io/examples/Issue1.ttl. This and the shape can be written as relative IRIs:

npx shex-validate \
    -x http://shex.io/examples/Issue.shex \
    -d http://shex.io/examples/Issue1.ttl \
    -s IssueShape \
    -n Issue1

validation library

The ShExLoader fetches and parses the schema and data and wires them up for validation:

const shexc = "http://shex.io/examples/IssueSchema";  // schema location
const data = "http://shex.io/examples/Issue1";        // data location
const node = "http://shex.io/examples/Issue1#Issue1"; // node in that data

const N3 = require("n3");
const ShExLoader = require("@shexjs/loader")({        // initialize with:
  fetch: globalThis.fetch,                            //   fetch implementation (Node >=18)
  rdfjs: N3,                                          //   RdfJs Turtle parser
});
const { ctor: RdfJsDb } = require('@shexjs/neighborhood-rdfjs');
const {ShExValidator} = require("@shexjs/validator");

ShExLoader.load({shexc: [shexc]}, {turtle: [data]})
  .then(function (loaded) {
    var db = RdfJsDb(loaded.data);
    var validator = new ShExValidator(loaded.schema, db, { results: "api" });
    const smap = [                                // array of node/shape pairs
      {node: node,                                //   JSON-LD @id for node
       shape: ShExValidator.Start}                //   schemas's start shape
    ]
    var result = validator.validateShapeMap(smap);  // success if no "errors"
    console.log(JSON.stringify(result, null, 2));
  } );

Note that the shex loader takes an array of ShExC schemas, either file paths or URLs, and an array of JSON schemas (empty in this invocation) and an array of Turtle files.

conversion

As with validation (above), you can convert by either executable or library.

conversion executable

ShEx can be represented in the compact syntax

PREFIX ex: <http://ex.example/#>
<IssueShape> {                       # An <IssueShape> has:
    ex:state [ex:unassigned            # state which is
              ex:assigned],            #   unassigned or assigned.
    ex:reportedBy @<UserShape>        # reported by a <UserShape>.
}

or in JSON:

{ "type": "Schema", "start": "http://shex.io/examples/IssueShape",
  "shapes": [
    { "id": "http://shex.io/examples/IssueShape", "type": "ShapeDecl",
      "shapeExpr": { "type": "Shape",
        "expression": { "type": "EachOf",
          "expressions": [
            { "type": "TripleConstraint", "predicate": "http://ex.example/#state",
              "valueExpr": { "type": "NodeConstraint", "values": [
                  "http://ex.example/#unassigned", "http://ex.example/#assigned"
              ] } },
            { "type": "TripleConstraint", "predicate": "http://ex.example/#reportedBy",
              "valueExpr": "http://shex.io/examples/UserShape" }
] } } } ] }

You can convert between them with shex-to-json:

npx shex-to-json http://shex.io/examples/Issue.shex

and, less elegantly, back with json-to-shex.

conversion by library

As with validation, the ShExLoader wraps the fetching and parsing:

const shexc = "http://shex.io/examples/Issue.shex";

const ShEx = require("shex");
const ShExLoader = ShEx.Loader({fetch: globalThis.fetch, rdfjs: require("n3")});
ShExLoader.load({shexc: [shexc]}, null).then(function (loaded) {
    console.log(JSON.stringify(loaded.schema, null, "  "));
});

There's no actual conversion; the JSON representation is just the stringification of the parsed schema.

local files

Command line arguments which don't start with http:// or https:// are assumed to be file paths. We can create a local JSON version of the Issues schema:

npx shex-to-json http://shex.io/examples/Issue.shex > Issue.json

and use it to validate the Issue1.ttl as we did above:

npx shex-validate \
    -j Issue.json \
    -d http://shex.io/examples/Issue1.ttl \
    -s http://shex.io/examples/IssueShape \
    -n http://shex.io/examples/Issue1

Of course the data file can be local as well.

Happy validating!

materialize

shexmap-materialize (from @shexjs/extension-map) transforms data from a source schema to a target schema after validation is done.

The syntax is:

shexmap-materialize `-t <target schema>`|-h [-j `<JSON Vars File>`] [-r `<RDF root IRI>`]

It reads the output of shex-validate --extension from STDIN and maps it to the specified target schema (--extension takes a package name or file glob for the extension module).

If supplied, a JSON vars file will be referenced to fill in constant values not specified from the source. This is useful in assigning default fields to the target when there is no equivalent value in the source schema and source data.

Here is an example of a simple JSON vars file:

{
  "urn:local:Demographics:constSys": "System"
}

If this vars file content is used, then any time a variable in the target file with value "urn:local:Demographics:constSys" is seen, the value "System" will be substituted.

The RDF root IRI specifies the root node from which all nodes in the schema will descend. The default root if none is specified is: tag:[email protected]/2016/root

Here are some examples:

npx shexmap-materialize -h

npx shex-validate -x source_schema.shex -d data.ttl -s ProblemShape -n prob1 \
    --extension @shexjs/extension-map \
  | npx shexmap-materialize -t target_schema.shex -j vars.json

cat problem.val | npx shexmap-materialize -t target_schema.shex -j vars.json -r http://hl7.org/fhir/shape/problem

See doc/threaded-materializer.md for how materialization works.

ShEx2 features

ShEx IMPORT Demo (with relative IRIs):

1. open a browser window (we'll call validator) with https://shex.io/webapps/packages/shex-webapp/doc/shex.html
2. open another browser window (we'll call viewer) with https://shex.io/shexTest/main/viewer?validation
3. wait 'till viewer loads and look for "3circRefS1-IS2-IS3-IS3" (near the bottom)
4. drag the "#3circRefS1-IS2-IS3-IS3" cell (or the ✓ to the left of it) to the right of the QueryMap area of validator
5. click on the long label under "Manifest:", then the long label under "Passing:" and validate.

It should validate, which involves the IMPORT of 3circRefS2-IS3 and 3circRefS3. 3circRefS2-IS3 also IMPORTs 3circRefS3 which shows that IMPORT is idempotent (has a side effect only the first time).

npm workspaces monorepo

This repo uses npm workspaces to manage multiple NPM packages located in packages/*:

• shex -- meta-package aggregating the packages below
• shape-map -- a ShapeMap parser
• @shexjs/parser -- parse ShExC into ShExJ
• @shexjs/writer -- serialize ShExJ as ShExC
• @shexjs/term -- RDF terms used in ShEx
• @shexjs/util -- some utilities for transforming schemas or validation output
• @shexjs/visitor -- a visitor for schemas
• @shexjs/validator -- validate nodes in an RDF graph against shapes in a schema
• @shexjs/eval-simple-1err -- eval-simple-1err
• @shexjs/eval-threaded-nerr -- eval-threaded-nerr
• -- an API for loading and using ShEx schemas

building

git clone [email protected]:shexjs/shex.js.git
cd shex.js
npm ci                  # install and cross-link all workspace packages

npm ci links every packages/* package into node_modules, so the packages resolve each other at their workspace versions and their executables land in node_modules/.bin/.

Generated artifacts are committed, so these are only needed after changing the corresponding sources:

npm run parser-all      # regenerate the Jison parsers (ShExC and ShapeMap)
npm run compile         # make ALL: recompile the TypeScript packages' lib/ output
npm run webpack         # rebuild the browser bundles in packages/*/doc/webpacks/

To try the web apps locally, serve the repository root with any static web server (the pages reference scripts across packages/* by relative path). Apache pointed at the checkout works; so does the zero-dependency server that ships in @shexjs/webapp:

npm run serve           # prints the shex.html / shexmap-simple URLs
                        # (npx shex-serve [--port N] [--root DIR] [--coi] outside the repo)

The app URLs open with language-aware CodeMirror editors; add ?editors=textarea for plain textareas (see doc/editor-integration-plan.md). What is still to do, across the whole repository, is in doc/plan.md; the design notes it links hold the rationale. With the editors on, the 🐞 buttons step through a materialization (shexmap-simple) or any of a validation's triple-expression matches (both apps) with schema-gutter breakpoints and a live threads pane; shexmap-debug and shex-debug are the command-line equivalents (see doc/debugger-design.md). shex-serve --coi sends the cross-origin-isolation headers that browser-side validation debugging will need.

To add a dependency to one package, use npm's --workspace flag from the repo root, e.g. npm install promise-worker --workspace=@shexjs/webapp. Development tooling (mocha, webpack, eslint...) lives in the root devDependencies.

testing

npm test                # quick suite, as run by the pre-commit hook
npm run test-all        # everything, including the cli, browser and server tests
TESTS='ThreadedMaterializer|Map' npm test  # filter by test name pattern
npm run lint
npm run coverage        # test-all under c8; writes coverage/lcov.info

npm run test-all sets TEST_cli/TEST_browser/TEST_server; the same suite runs in CI on every supported Node version. The test suite validates against a checkout of shexTest; it uses the copy installed as the shex-test dependency, or a ../shexTest sibling checkout if you have one. On main, the shex-test dependency must track shexTest#main (enforced by npm run check-branch-deps in the pre-commit hook).

publishing

The packages share one version line (formerly lerna's "fixed" mode; lerna is no longer used). Publishing runs in CI, not locally: pushing a v* tag triggers .github/workflows/release.yml, which runs node tools/publish-ordered.js under npm OIDC trusted publishing. The tag is only the trigger — each package publishes the version in its own package.json.

# 1. Bump every workspace package (independents @shexjs/term / shape-map kept) + sync the lock
node tools/bumpVersions.js --dry-run 1.0.0-alpha.NN   # preview first (optional)
node tools/bumpVersions.js 1.0.0-alpha.NN             # (== npm run bump-versions 1.0.0-alpha.NN)
npm install --package-lock-only                       # sync package-lock.json to the new versions

# 2. Commit (the pre-commit hook runs the suite + check-branch-deps)
git add -u
git commit -m "chore(release): 1.0.0-alpha.NN"

# 3. Annotated tag, then push — the v* tag is what triggers publishing
git tag -a v1.0.0-alpha.NN -m "1.0.0-alpha.NN"
git push origin main --follow-tags

tools/publish-ordered.js publishes each workspace package after the ones it depends on and skips any version the registry already has (--list shows the order, --dry-run rehearses it); per-package publishConfig grants public access.

Keep the lock in sync with a current npm. The release job's npm ci is strict: a lock left stale after a dependency's major bump fails it with "package.json and package-lock.json … not in sync" (older Node lanes' npm tolerates it, so CI can go half-red). If that bites, regenerate under the newest npm — npx -y npm@latest install --package-lock-only — and cut a fresh tag.