vortex-rdflib

An rdflib Store implementation for Vortex-RDF, a columnar zero-copy RDF serialization format — so .vortex files can be queried with SPARQL.

The native layer is the vortex-rdf package (PyO3 bindings over the vortex-rdf-core Rust crate), pulled in as a dependency: stores are memory-mapped from .vortex files by default, and queried in place without loading the dataset into memory. However, a .vortex file can also be fully loaded in memory, with exactly the same data structure and queried.

Install

pip install vortex-rdflib

Python 3.12+ (tested on 3.12–3.15). The vortex-rdf dependency ships prebuilt wheels for Linux (x86_64, aarch64), macOS (x86_64, arm64) and Windows (x64); on other platforms it builds from source.

Usage

from rdflib import Graph
from vortex_rdflib import VortexRdflibStore

graph = Graph(store=VortexRdflibStore("data.vortex"))
for row in graph.query("""
    SELECT ?s ?o WHERE {
        ?s <http://xmlns.com/foaf/0.1/name> ?o
    }
    LIMIT 10
"""):
    print(row.s, row.o)

SPARQL evaluation is rdflib's engine; the store serves quad patterns from the Vortex file. The store is read-only; mutation support is on the roadmap.

A .vortex file holds quads, so the store is context-aware. A Dataset gives the named graphs, and GRAPH works in SPARQL:

from rdflib import Dataset
from vortex_rdflib import VortexRdflibStore

# default_union=True makes the SPARQL default graph the union of every graph
dataset = Dataset(store=VortexRdflibStore("data.vortex"), default_union=True)

for graph in dataset.graphs():
    print(graph.identifier, len(graph))

for row in dataset.query("""
    SELECT ?g (COUNT(*) AS ?n) WHERE {
        GRAPH ?g { ?s ?p ?o }
    }
    GROUP BY ?g
"""):
    print(row.g, row.n)

A plain Graph(store=VortexRdflibStore(path)) — as in the example above — is the view over the whole file, every graph included. Like the union default graph of a Dataset, it is a set of triples — a triple in two graphs is yielded once, with both graphs as its contexts. Triples are the file's own: two spellings rdflib reads as one value, such as "042"^^xsd:integer and "42"^^xsd:integer, are two triples, as len() and SPARQL count them. A match over every graph returns a triple's quads next to each other (every sort order a store holds ends in the graph), so the store keeps one row of each run as it streams them (vortex_rdflib.union). A file where no triple is in two graphs — a file of one graph, such as any N-Triples file, or graphs that partition the data — has no runs to drop and pays nothing for it: whether a triple repeats is read once per open file, from one pass over its triple codes when it holds several graphs. A layout without term codes takes a file of several graphs to have repeats.

To produce a .vortex file from an RDF file, use the binding layer directly (or the vortex-rdf CLI):

from vortex_rdf import serialize_rdf

serialize_rdf("data.nq", "data.vortex", format="nquads", layout="dictionary")

layout accepts "default", "typed-object" and "dictionary" (described here); opening auto-detects the layout. The "dictionary" layout is the fastest to query from Python — it enables the SPARQL pushdowns described below.

Cancelling a query

cancel_scope gives the queries started inside it a vortex_rdf.CancelToken. Cancel the token, from any thread, and the query raises vortex_rdf.Cancelled at the next point that checks it:

• A call into vortex-rdf that races the token: every batch and chunk call, the string fallback's streamed read (a layout other than "dictionary", or VORTEX_RDF_DISABLE_CODE_PATH=1) on the call and every chunk it pulls, and each one-shot call that can scan a file-backed store.
• Every other one-shot call, which checks the token before it starts and then runs to completion: an in-memory call, a point lookup and a call capped at one row. The probes a join sends per row are calls of one kind or the other, so each probe is a check point of its own.
• Between calls: the pieces of a streamed join, the rows rdflib finishes when a join over the pair budget is left to it, and the slices and decode chunks the rows of a result are read in.

A cancel takes effect at the next of these points, so rows that are already fetched are delivered until then: a result whose rows are decoded as they are read stops within one decode chunk, at most 4,096 rows.

import threading

from vortex_rdf import Cancelled, CancelToken
from vortex_rdflib import cancel_scope

token = CancelToken()
deadline = threading.Timer(30.0, token.cancel)  # or cancel from any other thread
deadline.start()
try:
    with cancel_scope(token):
        result = graph.query("SELECT ?s ?o WHERE { ?s ?p ?o }")
    for row in result:
        print(row.s, row.o)
except Cancelled:
    print("cancelled")
finally:
    deadline.cancel()

This is the beginning of the README. Read the rest at the original. Read the full README ↗