Institutional Repository Data Management Tools

irdmtools is a set of tools for working with Invenio RDM and migrating content from EPrints to RDM. It consists of a small set of Go based command line programs along with Python and shell scripts and a wrapping irdm Python module (all in the scripts directory). The Go based tooling is designed to work directory with a copy of you repositories' database (e.g. Postgres for RDM or MySQL for EPrints).

irdmtools is developed around RDM's web services (e.g. REST API and OAI-PMH), PostgreSQL database and external metadata services (e.g. CrossRef, DataCite).

Caltech Library is using irdmtools to migrate content from our legacy EPrints 3.3 repositories (heavily customized) to RDM. Post migration the core Go tools will remain useful for curation at the collection level (e.g. rdmutil)

Featured Tools

rdmutil

This tool is for interacting with an Invenio RDM repository via RDM's REST and OAI-PMH API. It covers most the JSON API documented at https://inveniordm.docs.cern.ch/. This includes listing, submitting and managing records and draft records.

rdmutil configuration is read either from the environment or a JSON formatted configuration file. See the man page for details.

ep3util

This tool is used for migrating data out of EPrints. It can be used on a copy of your EPrints MySQL database. It parallels rdmutil and is an evolution of our tooling developed in eprinttools. See the man page for details.

eprint2rdm

This tool is migrating content from an EPrints repository via the EPrint REST API. It will retrieve an EPrint XML representation of the EPrint record and transform it into a JSON encoded simplified record nearly compatible with Invenio RDM. See the man page for details.

doi2rdm

This tool will query the CrossRef or DataCite API and convert a works record into a JSON structure compatible with an RDM record (e.g. to be inserted via an RDM API call). See the man page for details, and titles and abstracts from CrossRef for how titles, additional titles and descriptions are converted

ep3ds2citations

This tools take an EPrint record in a dataset collection and returns an abbreviated record inspired by citeproc. It also supports harvesting selected EPrint records into a dataset collection using the -harvest and -ids options. We use this feature to facilate creating https://feeds.library.caltech.edu. See the man page for details.

rdmds2citations

This tools take an RDM record in a dataset collection and returns an abbreviated record inspired by citeproc. It also supports harvesting selected RDM records into a dataset collection using the -harvest and -ids options. We use this feature to facilate creating https://feeds.library.caltech.edu. See the man page for details.

Documentation

• Installation
• User manual, with the man page of each program
• Help topics: rdmutil (one topic per action), ep3util (one per action) and doi2rdm have more than one page of help. NAME -help topics lists the topics and NAME -help TOPIC prints one, for example rdmutil -help get_record or doi2rdm -help exit_status. Each topic also has a page in this documentation, named NAME-TOPIC (for example rdmutil-get_record.1.md).
• Migration workflow, moving a repository from EPrints to Invenio RDM

For examples of using irdmtools from Python see the irdm_harvester repository.

Requirements

• An Invenio RDM deployment
• To building the Go based software and documentation
  • git
  • Go >= 1.27.1
  • Make (e.g. GNU Make)
  • Pandoc >= 3
• For harvesting content
  • dataset >= 2
• To migrate content from EPrints 3.3 to RDM
  • Python 3 and the packages listed in scripts/requirements.txt

Quick install

If you're running on Linux, macOS or Raspberry Pi OS you may be able to installed precompiled irdmtools Go based tools with the following curl command:

curl https://software.library.caltech.edu/irdmtools/installer.sh | sh

Installation from source

To install from source you need to download the source code and compile it. Here are the steps to install irdmtools.

git clone [email protected]:caltechlibrary/irdmtools
cd irdmtools
make
make test
make install
python -m pip install -r scripts/requirements.txt

Configuration

The Go based tools rely on a properly configured environment (i.e. environment variables set in your POSIX shell). Specific requirements are listed in the man pages for each of the Go based command line programs.