Dataset Project
The Dataset Project provides tools for working with collections of JSON documents. It uses a simple key and object pair to organize JSON documents into a collection. It supports SQL querying of the objects stored in a collection.
It is suitable for temporary storage of JSON objects in data processing pipelines as well as a persistent storage mechanism for collections of JSON objects.
The Dataset Project provides a command line program and a web service for working with JSON objects as a collection or individual objects. As such it is well suited for data science projects as well as building web applications that work with metadata.
dataset, a command line tool
dataset is a command line tool for working with collections of JSON documents. Collections can be stored on the file system in a pairtree or stored in a SQL database that supports JSON columns like SQLite3 and Postgres.
The dataset command line tool supports common data management operations as
- initialization of a collection
- dump and load JSON lines files into collection
- CRUD operations on a collection
- Query a collection using SQL
See Getting started with dataset for a tour and tutorial.
datasetd is dataset implemented as a web service
datasetd is a JSON REST web service and static file host. It provides a JSON API supporting the main operations found in the dataset command line program. This allows dataset collections to be integrated safely into web applications or be used concurrently by multiple processes.
The Dataset Web Service can host multiple collections each with their own custom query API defined in a simple YAML configuration file.
Design choices
dataset and datasetd are intended to be simple tools for managing collections JSON object documents in a predictable structured way. The dataset web service allows multi process or multi user access to a dataset collection via HTTP.
dataset is guided by the idea that you should be able to work with JSON documents as easily as you can any plain text document on the Unix command line. dataset is intended to be simple to use with minimal setup (e.g. dataset init mycollection.ds creates a new collection called 'mycollection.ds').
- dataset and datasetd store JSON object documents in collections
- Storage of the JSON documents may be either in a pairtree on disk or in a SQL database using JSON columns (e.g. SQLite3 or Postgres)
- dataset collections are made up of a directory containing a collection.json and codemeta.json files.
- collection.json metadata file describing the collection, e.g. storage type, name, description, if versioning is enabled
- codemeta.json is a codemeta file describing the nature of the collection, e.g. authors, description, funding
- collection objects are accessed by their key, a unique identifier, made up of lower case alpha numeric characters
- collection names are usually lowered case and usually have a
.ds extension for easy identification
dataset collection storage options
- SQL store stores JSON documents in a JSON column
- SQLite3 (default), Postgres >= 12 are the current SQL databases support
- A "DSN URI" is used to identify and gain access to the SQL database
- The DSN URI maybe passed through the environment
- pairtree (depricated, will be removed in v3)
- the pairtree path is always lowercase
- non-JSON attachments can be associated with a JSON document and found in a directories organized by semver (semantic version number)
- versioned JSON documents are created along side the current JSON document but are named using both their key and semver
datasetd is a web service
- it is intended as a back end web service run on localhost
- it runs on localhost and a designated port (port 8485 is the default)
- supports multiple collections each can have their own configuration for global object permissions and supported SQL queries
The choice of plain UTF-8 is intended to help future proof reading dataset collections. Care has been taken to keep dataset simple enough and light weight enough that it will run on a machine as small as a Raspberry Pi Zero while being equally comfortable on a more resource rich server or desktop environment. dataset can be re-implement in any programming language supporting file input and output, common string operations and along with JSON encoding and decoding functions. The current implementation is in the Go language.
Features
dataset supports
- Collection level
- Initialize a new dataset collection
- Codemeta file support for describing the collection contents
- Dump a collection to a JSON lines document
- Load a collection from a JSON lines document
- Listing Keys in a collection
- Object level actions
datasetd supports
- List collections available from the
web service
- List a collection's metadata
- List a collection's Keys
- Object level actions
- Schema validation
- Define reusable schemas in settings.yaml for validating JSON documents
- Enable per-collection validation with schema_name and validate configuration
- Returns X-Validation-Errors header with detailed error information on validation failures
Both dataset and datasetd maybe useful for general data science applications needing JSON object management or in implementing repository systems in research libraries and archives.
Limitations of dataset and datasetd
dataset has many limitations, some are listed below
- the pairtree implementation it is not a multi-process, multi-user data store
- it is not a general purpose database system
- it stores all keys in lower case in order to deal with file systems
- it stores collection names as lower case to deal with file systems that
are not case sensitive
- it should NOT be used for sensitive, confidential or secret information because it lacks access controls and data encryption
datasetd is a simple web service intended to run on "localhost:8485".
- it does not include support for authentication
- it does not support access control for users or roles
- it does not encrypt the data it stores
- it does not support HTTPS
- it does not provide auto key generation
- it limits the size of JSON documents stored to the size supported by
with host SQL JSON columns
- it limits the size of attached files to less than 250 MiB
- it does not support partial JSON record updates or retrieval
- it does not provide an interactive Web UI for working with dataset
collections
- it should NOT be used for sensitive, confidential or secret information because it lacks access controls and data encryption
Read next ...
Authors and history
- R. S. Doiel
- Tommy Morrell
Releases
Compiled versions are provided for Linux (x86, aarch64), Mac OS X (x86 and M1), Windows 11 (x86, aarch64) and Raspberry Pi OS.
github.com/caltechlibrary/dataset/releases
Related projects
You can use dataset from Python via the py_dataset package.
You can use dataset from Deno+TypeScript by running datasetd and access it with ts_dataset.
