PLATO tools
Status: experimental — under development; not yet in beta testing.
Tools for data about places in the formats of PLATO, the Place Attestation Ontology: check it,
convert it, compare two versions of it, publish it. They run in your browser at
https://pelagios.org/plato-tools/, and from the command line for many
files at a time.
Your files stay on your computer, and nothing is sent to any other site unless you allow it, in the
Permissions panel at the top of each page, which also shows what this browser remembers for the
tools and lets you forget it.
Everything happens in the browser tab, with the working data on disk, in the browser's private file
storage. A dataset of a million places has been checked and converted in the page;
how large a dataset your browser can take,
and what changes on the command line, which also takes many files at once,
is in the guide.
How to use them is in the PLATO guide:
Checking, converting and comparing says what each check looks at, what each format keeps
when converting, and what the version check reports. The rest of the guide explains PLATO
itself. How the tools work, and how they are tested, is in DEVELOPERS.md.
No data to hand? Download example files
(zip, 344 KB) to try each tool with, and a README in it saying what each should show. The zip is
also in this repository, in public/try/.
The toolbox
Still to come, with the plan as a whole in #1,
and the issue for each where it has one:
They read and write PLATO's spreadsheet tables, PLATO JSON and JSON Lines, RDF (N-Triples, N-Quads
and Turtle, written as N-Triples) and Linked Places Format v1, and read the W3C Web Annotations that
Recogito exports, TEI editions (P5; P4, such as Perseus's, its Beta Code Greek reported and not
converted; and TEI with no namespace), and any other CSV or GeoJSON of places: the formats in full.
A CSV, workbook or GeoJSON of places may give its locations as grid references of the Ordnance
Survey National Grid (TQ 33760 80560, SU1234) or the Irish Grid (O 15 34): each becomes the
centre of its square in WGS 84, with a precision from the number of digits (a four-figure reference
is a 1 km square), converted by the Ordnance Survey's Helmert transformation (good to about 3.5 m)
or, for the Irish Grid, EPSG 1641's (about 1 m). Irish Transverse Mercator coordinates are not read.
A TEI edition and a table of places have reading options, each off until chosen: on the page in
Reading options, below the column table, and on the command line as flags (below).
| Format | Reading option | Page | Command line |
|---|
| TEI | Read the <listPlace>: each <place> with a web address as an attestation, its first name the edition's headword, with its coordinates where the address is on the edition's own site | Read the list of places | --list-places |
| TEI | Make a place's address from its @key (with no ref), with a pattern for the key's prefix: tgn,7011179 with http://vocab.getty.edu/tgn/{id} | A row for each prefix found, its suggested pattern filled in, unticked until you tick Use | --key-pattern [PREFIX=]PATTERN, repeatable; no prefix for keys with none |
| TEI | Places in the header (where the object was found or made), and place names in an edition's commentary, translation and notes, as the editors' words, each with the form status plato:Editorial | Read the places in the header (found at, made at), as the editors' words; Read place names in the commentary and notes, as the editors' words | --header-places, --commentary-places |
| CSV, GeoJSON | Rows with the same id are evidence about one place, an attestation each, where a repeated id is otherwise a problem; needs a column read as the place id | Rows with the same id are one place | --same-id |
| CSV, GeoJSON | A column of a gazetteer's ids (pleiades_id: 579885) made into web addresses with a pattern, suggested for Pleiades, GeoNames and Wikidata, never used until confirmed | In the column table, Make web addresses in the column's row | --columns with {"field": "address", "pattern": "https://pleiades.stoa.org/places/{id}"} for the column |
| CSV, GeoJSON | The regions a place lies in (a parish, a county, a country…), guessed from the headings and numbered widest first (1 the widest). Under a base address of your own each region becomes a place, each contained in the next, and each place is plato:ContainedIn its narrowest region, as PLATO's worked example of regions has it; without one they are kept in the notes | Region it lies in, with its Level beside it | --columns with {"field": "within", "level": 2} for the column |
| CSV, GeoJSON |
From the command line
The same engine runs in a terminal, with Node.js 24 or later, and says the same things in the same
words:
npx github:pelagios/plato-tools check data/*.jsonl tables/ # without installing anything
git clone https://github.com/pelagios/plato-tools && cd plato-tools && npm install
node bin/plato-tools.mjs check my-tables/ places.jsonl.gz export.nt # or from a clone
node bin/plato-tools.mjs convert --to plato-jsonl --out converted/ export.nt.gz
node bin/plato-tools.mjs check --json data/*.json > report.jsonl # for scripts
node bin/plato-tools.mjs compare release-1.jsonl.gz release-2.jsonl.gz # was anything deleted or changed?
node bin/plato-tools.mjs preview --limit 5 my-places.csv # the first records, before the rest
| Command | What it does |
|---|
check INPUT… | Reports on each input in turn, then gives a total |
convert --to TARGET INPUT… | Also writes each input as plato-jsonl, plato-json, ntriples, tables, lpf-seq or lpf, into --out (by default the current directory), named after the input |
cluster --column NAME [--method M] INPUT | Proposes groups of similar spellings in one column of a table of places, as JSON for review, and applies nothing (below) |
preview [--limit N] INPUT | Shows the first N records (10 unless --limit says otherwise) of a table of places, a TEI edition or W3C Web Annotations, as a run reads them, and writes nothing (below) |
compare EARLIER LATER | Checks that a published dataset was only added to (what it reports) |
publish PART INPUT | Prepares a dataset for publishing: report, mint, site or w3id (below) |
datacube FILE… | Checks a Data Cube export against Data Cube's integrity constraints (Statistical tables) |
- Which files make one input. Each file is one input, except that the sheets of a set of
spreadsheet tables (
places.csv, names.csv…) are one input per directory, whether the directory
is given or its CSV files are named one by one, so a/*.csv b/*.csv is two sets. Any other CSV
file in the directory, or named, is an input of its own, and so is a lone sheet whose header does
not begin as that sheet's does (a places.csv of one's own). A zip of the CSV files is one set, and so
is a workbook (.xlsx, .ods) whose sheets are named after the tables' (two or more, or one that
begins as that sheet does).
- A workbook of your own (any other
.xlsx or .ods) is a table of places, read as a CSV file
is, one sheet at a time: the first sheet that is not hidden, unless --sheet NAME (or, on the
page, the sheet chosen above the columns table) names another. The report names the sheets not
read, any hidden sheet (Excel workbooks only: SheetJS does not read an ODS file's hidden flag), an
empty sheet, a formula saved without its value, and a cell holding an error (#DIV/0!, #N/A,
#REF!…), which carries nothing. Each cell is read as its value, not as the workbook displays it: a coordinate formatted 0.00 keeps every digit, and a date is an ISO date
(1990-05-06, or 1990-05-06T10:30:00 with a time). A workbook is read whole into memory, and
one over 50 MB is warned of; a very large sheet is better saved as CSV (UTF-8).
- Exit status: 0 if no input has problems, 1 if any has, 2 if the command is wrong or an input
cannot be read or written (a missing file, an unrecognised format, an output that already
exists). Warnings, and what a conversion cannot carry over, do not count as problems. For
compare: 0 if nothing was deleted or changed, 1 if something was, 2 if the two versions could
not be compared, including when either cannot be read to the end.
- Outputs are never replaced unless
--overwrite is given, and an output left incomplete, by a
file that could not be read to the end or by identity relations that could not be held back for a
PLATO JSON document, is removed.
--json prints one JSON object per input, one per line, then one for the total: the page's
report, with the input's format, counts, outputs and status. Its storeBytes is the size of the
working database (the triple store) for RDF or attestation-centric input, or the tables' for
spreadsheet tables, or null when the input streamed straight through. For a table of places its columns is a list of
{column, field, reason}, with , , or , and where the field has them, to read;
takes the object printed without instead. For a workbook's sheet, names the
sheet read and lists them all. For a TEI edition, holds the
patterns. prints one line per input.
Previewing the first records
preview reads the first records of a table of places (a CSV file, plain GeoJSON, or a sheet of a
workbook), a TEI edition, or W3C Web Annotations (with or without --georef: nothing is fetched),
through the same reader check and convert use, and stops: it checks nothing as a whole and
writes nothing. On the page, Preview the first 10 records does the same, once the columns are
read. It takes --limit N, --columns FILE, --sheet NAME, --base URL and the reading options.
Any other format is refused, with exit 2: a shortened file of the spreadsheet tables or of PLATO
JSON could be taken for the whole.
node bin/plato-tools.mjs preview --limit 3 places.csv > first.jsonl # the records as JSON Lines
node bin/plato-tools.mjs preview --json edition.xml # all of it, as one JSON object
- The records go to stdout, one JSON object a line: a place for a place-centric table, an
attestation (saying what it is about) for annotations, TEI and a table of rows about web
addresses, as the reader gives them before a run gathers them by place. The line saying what the
preview is, why it may be partial, and the losses so far, grouped as the report groups them,
go to stderr.
--json prints one object with all of it (items, line, complete, total,
read, why, report, exitCode).
- The line says
first N of M records; nothing checked or written only where the whole input was
read, so that M is known; otherwise the first N records read; the rest not read. A file is never
read to its end to count it: reading stops at the first record past the limit.
- A partial preview says why its records may not be all there are: the rest was not read; a run
gathers attestations by place at the end; places made from rows with the same id come at the end;
a TEI place name pointing to a
<place> later in the file (ref="#…") waits for it, and a name
in a part that may be the editors' is held until it is known whether the text has an edition div,
either of which can need most of the file. The report says so too (preview-partial, a warning).
- Exit status: 0 if the records read have no problems, 1 if they have, 2 if no preview could be
made.
Grouping similar spellings
Before places are looked up in a gazetteer, variant spellings of one name can be grouped, as
OpenRefine's clustering does, so that Rotherhith, ROTHERHITHE. and Rotherhithe are looked up
by one spelling. Nothing is grouped unless you say so, and the names in the PLATO output always
keep the source's spellings: a grouped row is given a spelling to look it up by, beside its record,
and its attestation a note, Grouped for lookup with: Rotherhith, ROTHERHITHE. (spelling chosen:
Rotherhithe).
node bin/plato-tools.mjs cluster --column name places.csv # propose, as JSON; nothing applied
node bin/plato-tools.mjs cluster --column parish --method phonetic places.csv
node bin/plato-tools.mjs convert --to plato-json --clusters groups.json places.csv # apply the groups kept
- Three ways of grouping (
--method): fingerprint (the default: case, accents, punctuation,
spacing and word order do not count, so Newton, Upper and upper newton are one),
ngram-fingerprint (the same pairs of letters: spacing does not count either; it groups more,
some wrongly) and phonetic (Cologne phonetics, word by word: names that sound alike; made for
German, it serves English and other names in the Latin alphabet passably, and groups the most,
some wrongly: Rotherhithe and Redruth sound alike to it). Each group lists its spellings,
with how many rows have each, and suggests the commonest.
- On the page, Group similar spellings… (below the Reading options) proposes the groups for the
column chosen, each unticked, with the spelling to look it up by, which you can change. Only
ticked groups are used. Save matching then saves them beside the columns, as
{"columns": {…}, "clusters": {"name": {"method": "fingerprint", "groups": [{"chosen": "Rotherhithe", "members": ["Rotherhith", "ROTHERHITHE."]}]}}}, and Load shows them ticked again. A
ticked group that a later Find groups does not find again is kept at the end of the list, still
ticked, and marked as not found this way.
--clusters FILE takes that file, or the groups alone ({"name": {"method": …, "groups": […]}}).
Given only as --columns, such a file's columns are used and its groups are not, and the command
says so: groups are never applied without being asked for.
Given with an input that is not a table of places (a TEI edition, a Recogito export), they are not
used for it, and the command says so on stderr.
Publishing
publish (Agora) takes a checked dataset one step at a time towards being cited and found: each
part reads it through the same check, and writes nothing from a dataset with problems. What each
part reports, and the order to take them in, is in the guide:
Publishing your dataset.
node bin/plato-tools.mjs publish report --out deposit/ my-tables/ # what it lacks to be FAIR; .zenodo.json, CITATION.cff, datacite.json
node bin/plato-tools.mjs publish mint --out . my-tables/ # my-tables-with-ids.jsonl: an @id for every attestation
node bin/plato-tools.mjs publish mint --previous release-1.jsonl --out . my-tables/ # keeping the addresses release 1 published
node bin/plato-tools.mjs publish site --repo owner/name --out build/ my-tables-with-ids.jsonl # a page and a JSON-LD file for every place
node bin/plato-tools.mjs publish w3id --repo owner/name --maintainer you --out . my-tables-with-ids.jsonl # the w3id.org folder, with its tests
- Mint once, and commit the result. The site is made only from a dataset whose attestations
have addresses, and never makes them itself; the workflow
site writes for GitHub Actions builds
the site from the committed copy, so CI never mints.
- On the command line a site or a w3id folder is written as a folder; in the browser, as a zip.
The site's size is estimated first against GitHub Pages' 1 GB: the page refuses a site over it,
and the command line writes it with a warning, for hosting elsewhere. For Pages, give a list of
the places to include, a text file of their keys one to a line (
--only, or on the page Only
these places in Options; the rest are in the site's downloads), or leave out --turtle.
- The w3id folder is for a base address on
w3id.org and a dataset whose status is published;
its STEPS.md says how to test the rules and open the pull request.
--overwrite replaces a whole output folder (such as <stem>-site), not single files in it.
plato-tools --help lists everything, and plato-tools --version names the PLATO commit the
checks follow.
Results that must be citable. npx github:pelagios/plato-tools runs whatever the branch holds
when it is fetched (npx may also reuse a copy it has cached), and npm does not use a git
dependency's lockfile, so its dependencies can differ from those the tools were tested with. For a
result someone must be able to reproduce, run a named ref from a clone, with that ref's own
lockfile, and record the two commits:
REF=main # a branch or tag; a clone of a ref that does not exist fails
git clone --depth 1 --branch "$REF" https://github.com/pelagios/plato-tools "plato-tools-$REF"
cd "plato-tools-$REF" && npm ci # exactly the dependencies in the ref's package-lock.json
git rev-parse HEAD # the tools' commit
node bin/plato-tools.mjs --version # and the PLATO commit they check against
node bin/plato-tools.mjs check my-data.json
For a single commit rather than a branch or tag, replace the clone with
git init plato-tools-C && cd plato-tools-C && git fetch --depth 1 https://github.com/pelagios/plato-tools C && git checkout FETCH_HEAD,
which also fails if the commit does not exist, then run npm ci as above.
Development
npm install
npm test # the engine and the command line, in Node
python3 e2e/app_test.py # the page itself, in a headless browser
npm run dev # a local server
DEVELOPERS.md has the rest: how the tools are built, how they are pinned to PLATO,
every test and what it proves, and the conventions to keep.
Citing
If you use the tools in your work, please cite them: CITATION.cff gives the
citation (GitHub shows it as "Cite this repository"), and .zenodo.json the record
each release is archived under. PLATO itself is cited separately, as its own repository says.
Acknowledgements
Development has been supported by the
Institute for Spatial History Innovation (ISHI) at the University of
Pittsburgh.
Licence
BSD 3-Clause. PLATO itself is CC BY 4.0.
- Type: Alegreya and Alegreya Sans, as in the PLATO guide, under the SIL Open Font Licence 1.1
(
public/fonts/). They are bundled from the Fontsource packages rather than loaded from Google
Fonts, so that opening the page makes no request to a third party.
- Icons: from Lucide, under the ISC Licence (
public/icons/).
- The logo and the drawing of Plato are PLATO's (
docs/_static/ in the PLATO repository).
