IOStreams

IOStreams reads and writes files of any size in Ruby, a block at a time, and makes their format, compression, encryption and storage location transparent to your code: CSV, JSON, Excel or Parquet files, compressed with gzip, zip, bzip2 or Zstandard, encrypted with PGP, on local disk, AWS S3, Google Cloud Storage (experimental), an SFTP server or a web server.

require "iostreams"

# Write a CSV file compressed with gzip, a record at a time.
IOStreams.path("people.csv.gz").writer(:hash) do |io|
  io << {"name" => "Jack", "city" => "Oslo"}
  io << {"name" => "Jill", "city" => "Lima"}
end

# Read it back a record at a time, as it is decompressed.
IOStreams.path("people.csv.gz").each(:hash) do |person|
  p person
end
# {"name" => "Jack", "city" => "Oslo"}
# {"name" => "Jill", "city" => "Lima"}

The file name says how each file is stored, so the same code reads a plain CSV file, a spreadsheet, a file encrypted with PGP, or a file on S3 or SFTP. Copy it to S3 as JSON, encrypted with PGP for a partner, by naming the target so:

IOStreams.path("s3://my-bucket/people.json.gz.pgp").
  option(:pgp, recipient: "[email protected]").
  copy_from("people.csv.gz", mode: :hash)

Why IOStreams

• The same code for every format, compression and encryption, chosen by the file name, so the code does not change when a partner sends .csv.pgp instead of .xlsx.
• The same code wherever the file is stored, so a path held in configuration can be a local file in development, and on S3 or SFTP in production.
• Constant memory: files are read and written a block at a time, so a 10 GB file needs about as much memory as a 10 KB one.
• Strict options: a misspelled option raises ArgumentError when it is set, wherever the code runs.

Installation

For Ruby 3.3 or later, add the gem to your Gemfile:

bundle add iostreams

IOStreams loads the gems of the features that need one, such as aws-sdk-s3 for S3, when they are first used. See Installation for the list.

Documentation

Start with the IOStreams tutorial, or the common tasks, each a line of code, then read the rest of the IOStreams documentation.

See the CHANGELOG for the release history and notable changes.

Upgrading

v3.0 is a major release: text is read as UTF-8, a schema describes the columns of tabular files, and many options are now checked strictly. See Upgrading IOStreams for the changes that may need updates to your application, and the security settings to review.

Project Status

Production Ready, heavily used in production environments, many as part of Rocket Job.

Versioning

This project adheres to Semantic Versioning.

Contributing

Contributions are welcome. See CONTRIBUTING.md for how to set up the project, run its tests, its architecture, and the code of conduct.

Author

Reid Morrison

License

Copyright 2020 Reid Morrison

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.