PLATO tools

Status: experimental — under development; not yet in beta testing.

Tools for data about places in the formats of PLATO, the Place Attestation Ontology: check it, convert it, compare two versions of it, publish it. They run in your browser at https://pelagios.org/plato-tools/, and from the command line for many files at a time.

Your files stay on your computer, and nothing is sent to any other site unless you allow it, in the Permissions panel at the top of each page, which also shows what this browser remembers for the tools and lets you forget it. Everything happens in the browser tab, with the working data on disk, in the browser's private file storage. A dataset of a million places has been checked and converted in the page; how large a dataset your browser can take, and what changes on the command line, which also takes many files at once, is in the guide.

How to use them is in the PLATO guide: Checking, converting and comparing says what each check looks at, what each format keeps when converting, and what the version check reports. The rest of the guide explains PLATO itself. How the tools work, and how they are tested, is in DEVELOPERS.md.

No data to hand? Download example files (zip, 344 KB) to try each tool with, and a README in it saying what each should show. The zip is also in this repository, in public/try/.

The toolbox

Still to come, with the plan as a whole in #1, and the issue for each where it has one:

They read and write PLATO's spreadsheet tables, PLATO JSON and JSON Lines, RDF (N-Triples, N-Quads and Turtle, written as N-Triples) and Linked Places Format v1, and read the W3C Web Annotations that Recogito exports, TEI editions (P5; P4, such as Perseus's, its Beta Code Greek reported and not converted; and TEI with no namespace), and any other CSV or GeoJSON of places: the formats in full.

A CSV, workbook or GeoJSON of places may give its locations as grid references of the Ordnance Survey National Grid (TQ 33760 80560, SU1234) or the Irish Grid (O 15 34): each becomes the centre of its square in WGS 84, with a precision from the number of digits (a four-figure reference is a 1 km square), converted by the Ordnance Survey's Helmert transformation (good to about 3.5 m) or, for the Irish Grid, EPSG 1641's (about 1 m). Irish Transverse Mercator coordinates are not read.

A TEI edition and a table of places have reading options, each off until chosen: on the page in Reading options, below the column table, and on the command line as flags (below).

Format	Reading option	Page	Command line
TEI	Read the <listPlace>: each <place> with a web address as an attestation, its first name the edition's headword, with its coordinates where the address is on the edition's own site	Read the list of places	--list-places
TEI	Make a place's address from its @key (with no ref), with a pattern for the key's prefix: tgn,7011179 with http://vocab.getty.edu/tgn/{id}	A row for each prefix found, its suggested pattern filled in, unticked until you tick Use	--key-pattern [PREFIX=]PATTERN, repeatable; no prefix for keys with none
TEI	Places in the header (where the object was found or made), and place names in an edition's commentary, translation and notes, as the editors' words, each with the form status plato:Editorial	Read the places in the header (found at, made at), as the editors' words; Read place names in the commentary and notes, as the editors' words	--header-places, --commentary-places
CSV, GeoJSON	Rows with the same id are evidence about one place, an attestation each, where a repeated id is otherwise a problem; needs a column read as the place id	Rows with the same id are one place	--same-id
CSV, GeoJSON	A column of a gazetteer's ids (pleiades_id: 579885) made into web addresses with a pattern, suggested for Pleiades, GeoNames and Wikidata, never used until confirmed	In the column table, Make web addresses in the column's row	--columns with {"field": "address", "pattern": "https://pleiades.stoa.org/places/{id}"} for the column
CSV, GeoJSON	The regions a place lies in (a parish, a county, a country…), guessed from the headings and numbered widest first (1 the widest). Under a base address of your own each region becomes a place, each contained in the next, and each place is plato:ContainedIn its narrowest region, as PLATO's worked example of regions has it; without one they are kept in the notes	Region it lies in, with its Level beside it	--columns with {"field": "within", "level": 2} for the column
CSV, GeoJSON

From the command line

The same engine runs in a terminal, with Node.js 24 or later, and says the same things in the same words:

npx github:pelagios/plato-tools check data/*.jsonl tables/             # without installing anything
git clone https://github.com/pelagios/plato-tools && cd plato-tools && npm install
node bin/plato-tools.mjs check my-tables/ places.jsonl.gz export.nt     # or from a clone
node bin/plato-tools.mjs convert --to plato-jsonl --out converted/ export.nt.gz
node bin/plato-tools.mjs check --json data/*.json > report.jsonl        # for scripts
node bin/plato-tools.mjs compare release-1.jsonl.gz release-2.jsonl.gz  # was anything deleted or changed?
node bin/plato-tools.mjs preview --limit 5 my-places.csv                 # the first records, before the rest

Command	What it does
check INPUT…	Reports on each input in turn, then gives a total
convert --to TARGET INPUT…	Also writes each input as plato-jsonl, plato-json, ntriples, tables, lpf-seq or lpf, into --out (by default the current directory), named after the input
cluster --column NAME [--method M] INPUT	Proposes groups of similar spellings in one column of a table of places, as JSON for review, and applies nothing (below)
preview [--limit N] INPUT	Shows the first N records (10 unless --limit says otherwise) of a table of places, a TEI edition or W3C Web Annotations, as a run reads them, and writes nothing (below)
compare EARLIER LATER	Checks that a published dataset was only added to (what it reports)
publish PART INPUT	Prepares a dataset for publishing: report, mint, site or w3id (below)
datacube FILE…	Checks a Data Cube export against Data Cube's integrity constraints (Statistical tables)

• Which files make one input. Each file is one input, except that the sheets of a set of spreadsheet tables (places.csv, names.csv…) are one input per directory, whether the directory is given or its CSV files are named one by one, so a/*.csv b/*.csv is two sets. Any other CSV file in the directory, or named, is an input of its own, and so is a lone sheet whose header does not begin as that sheet's does (a places.csv of one's own). A zip of the CSV files is one set, and so is a workbook (.xlsx, .ods) whose sheets are named after the tables' (two or more, or one that begins as that sheet does).
• A workbook of your own (any other .xlsx or .ods) is a table of places, read as a CSV file is, one sheet at a time: the first sheet that is not hidden, unless --sheet NAME (or, on the page, the sheet chosen above the columns table) names another. The report names the sheets not read, any hidden sheet (Excel workbooks only: SheetJS does not read an ODS file's hidden flag), an empty sheet, a formula saved without its value, and a cell holding an error (#DIV/0!, #N/A, #REF!…), which carries nothing. Each cell is read as its value, not as the workbook displays it: a coordinate formatted 0.00 keeps every digit, and a date is an ISO date (1990-05-06, or 1990-05-06T10:30:00 with a time). A workbook is read whole into memory, and one over 50 MB is warned of; a very large sheet is better saved as CSV (UTF-8).
• Exit status: 0 if no input has problems, 1 if any has, 2 if the command is wrong or an input cannot be read or written (a missing file, an unrecognised format, an output that already exists). Warnings, and what a conversion cannot carry over, do not count as problems. For compare: 0 if nothing was deleted or changed, 1 if something was, 2 if the two versions could not be compared, including when either cannot be read to the end.
• Outputs are never replaced unless --overwrite is given, and an output left incomplete, by a file that could not be read to the end or by identity relations that could not be held back for a PLATO JSON document, is removed.
• --json prints one JSON object per input, one per line, then one for the total: the page's report, with the input's format, counts, outputs and status. Its storeBytes is the size of the working database (the triple store) for RDF or attestation-centric input, or the tables' for spreadsheet tables, or null when the input streamed straight through. For a table of places its columns is a list of {column, field, reason}, with , , or , and where the field has them, to read; takes the object printed without instead. For a workbook's sheet, names the sheet read and lists them all. For a TEI edition, holds the patterns. prints one line per input.

Previewing the first records

preview reads the first records of a table of places (a CSV file, plain GeoJSON, or a sheet of a workbook), a TEI edition, or W3C Web Annotations (with or without --georef: nothing is fetched), through the same reader check and convert use, and stops: it checks nothing as a whole and writes nothing. On the page, Preview the first 10 records does the same, once the columns are read. It takes --limit N, --columns FILE, --sheet NAME, --base URL and the reading options. Any other format is refused, with exit 2: a shortened file of the spreadsheet tables or of PLATO JSON could be taken for the whole.

node bin/plato-tools.mjs preview --limit 3 places.csv > first.jsonl    # the records as JSON Lines
node bin/plato-tools.mjs preview --json edition.xml                   # all of it, as one JSON object

• The records go to stdout, one JSON object a line: a place for a place-centric table, an attestation (saying what it is about) for annotations, TEI and a table of rows about web addresses, as the reader gives them before a run gathers them by place. The line saying what the preview is, why it may be partial, and the losses so far, grouped as the report groups them, go to stderr. --json prints one object with all of it (items, line, complete, total, read, why, report, exitCode).
• The line says first N of M records; nothing checked or written only where the whole input was read, so that M is known; otherwise the first N records read; the rest not read. A file is never read to its end to count it: reading stops at the first record past the limit.
• A partial preview says why its records may not be all there are: the rest was not read; a run gathers attestations by place at the end; places made from rows with the same id come at the end; a TEI place name pointing to a <place> later in the file (ref="#…") waits for it, and a name in a part that may be the editors' is held until it is known whether the text has an edition div, either of which can need most of the file. The report says so too (preview-partial, a warning).
• Exit status: 0 if the records read have no problems, 1 if they have, 2 if no preview could be made.

Grouping similar spellings

Before places are looked up in a gazetteer, variant spellings of one name can be grouped, as OpenRefine's clustering does, so that Rotherhith, ROTHERHITHE. and Rotherhithe are looked up by one spelling. Nothing is grouped unless you say so, and the names in the PLATO output always keep the source's spellings: a grouped row is given a spelling to look it up by, beside its record, and its attestation a note, Grouped for lookup with: Rotherhith, ROTHERHITHE. (spelling chosen: Rotherhithe).

node bin/plato-tools.mjs cluster --column name places.csv                         # propose, as JSON; nothing applied
node bin/plato-tools.mjs cluster --column parish --method phonetic places.csv
node bin/plato-tools.mjs convert --to plato-json --clusters groups.json places.csv  # apply the groups kept

• Three ways of grouping (--method): fingerprint (the default: case, accents, punctuation, spacing and word order do not count, so Newton, Upper and upper newton are one), ngram-fingerprint (the same pairs of letters: spacing does not count either; it groups more, some wrongly) and phonetic (Cologne phonetics, word by word: names that sound alike; made for German, it serves English and other names in the Latin alphabet passably, and groups the most, some wrongly: Rotherhithe and Redruth sound alike to it). Each group lists its spellings, with how many rows have each, and suggests the commonest.
• On the page, Group similar spellings… (below the Reading options) proposes the groups for the column chosen, each unticked, with the spelling to look it up by, which you can change. Only ticked groups are used. Save matching then saves them beside the columns, as {"columns": {…}, "clusters": {"name": {"method": "fingerprint", "groups": [{"chosen": "Rotherhithe", "members": ["Rotherhith", "ROTHERHITHE."]}]}}}, and Load shows them ticked again. A ticked group that a later Find groups does not find again is kept at the end of the list, still ticked, and marked as not found this way.
• --clusters FILE takes that file, or the groups alone ({"name": {"method": …, "groups": […]}}). Given only as --columns, such a file's columns are used and its groups are not, and the command says so: groups are never applied without being asked for. Given with an input that is not a table of places (a TEI edition, a Recogito export), they are not used for it, and the command says so on stderr.

Publishing

publish (Agora) takes a checked dataset one step at a time towards being cited and found: each part reads it through the same check, and writes nothing from a dataset with problems. What each part reports, and the order to take them in, is in the guide: Publishing your dataset.

node bin/plato-tools.mjs publish report --out deposit/ my-tables/          # what it lacks to be FAIR; .zenodo.json, CITATION.cff, datacite.json
node bin/plato-tools.mjs publish mint --out . my-tables/                   # my-tables-with-ids.jsonl: an @id for every attestation
node bin/plato-tools.mjs publish mint --previous release-1.jsonl --out . my-tables/   # keeping the addresses release 1 published
node bin/plato-tools.mjs publish site --repo owner/name --out build/ my-tables-with-ids.jsonl   # a page and a JSON-LD file for every place
node bin/plato-tools.mjs publish w3id --repo owner/name --maintainer you --out . my-tables-with-ids.jsonl   # the w3id.org folder, with its tests

• Mint once, and commit the result. The site is made only from a dataset whose attestations have addresses, and never makes them itself; the workflow site writes for GitHub Actions builds the site from the committed copy, so CI never mints.
• On the command line a site or a w3id folder is written as a folder; in the browser, as a zip. The site's size is estimated first against GitHub Pages' 1 GB: the page refuses a site over it, and the command line writes it with a warning, for hosting elsewhere. For Pages, give a list of the places to include, a text file of their keys one to a line (--only, or on the page Only these places in Options; the rest are in the site's downloads), or leave out --turtle.
• The w3id folder is for a base address on w3id.org and a dataset whose status is published; its STEPS.md says how to test the rules and open the pull request.
• --overwrite replaces a whole output folder (such as <stem>-site), not single files in it.

plato-tools --help lists everything, and plato-tools --version names the PLATO commit the checks follow.

Results that must be citable. npx github:pelagios/plato-tools runs whatever the branch holds when it is fetched (npx may also reuse a copy it has cached), and npm does not use a git dependency's lockfile, so its dependencies can differ from those the tools were tested with. For a result someone must be able to reproduce, run a named ref from a clone, with that ref's own lockfile, and record the two commits:

REF=main                                   # a branch or tag; a clone of a ref that does not exist fails
git clone --depth 1 --branch "$REF" https://github.com/pelagios/plato-tools "plato-tools-$REF"
cd "plato-tools-$REF" && npm ci            # exactly the dependencies in the ref's package-lock.json
git rev-parse HEAD                         # the tools' commit
node bin/plato-tools.mjs --version         # and the PLATO commit they check against
node bin/plato-tools.mjs check my-data.json

For a single commit rather than a branch or tag, replace the clone with git init plato-tools-C && cd plato-tools-C && git fetch --depth 1 https://github.com/pelagios/plato-tools C && git checkout FETCH_HEAD, which also fails if the commit does not exist, then run npm ci as above.

Development

npm install
npm test                                  # the engine and the command line, in Node
python3 e2e/app_test.py                   # the page itself, in a headless browser
npm run dev                               # a local server

DEVELOPERS.md has the rest: how the tools are built, how they are pinned to PLATO, every test and what it proves, and the conventions to keep.

Citing

If you use the tools in your work, please cite them: CITATION.cff gives the citation (GitHub shows it as "Cite this repository"), and .zenodo.json the record each release is archived under. PLATO itself is cited separately, as its own repository says.

Acknowledgements

Development has been supported by the Institute for Spatial History Innovation (ISHI) at the University of Pittsburgh.

Licence

BSD 3-Clause. PLATO itself is CC BY 4.0.

• Type: Alegreya and Alegreya Sans, as in the PLATO guide, under the SIL Open Font Licence 1.1 (public/fonts/). They are bundled from the Fontsource packages rather than loaded from Google Fonts, so that opening the page makes no request to a third party.
• Icons: from Lucide, under the ISC Licence (public/icons/).
• The logo and the drawing of Plato are PLATO's (docs/_static/ in the PLATO repository).