Down

Down is a utility tool for streaming, flexible and safe downloading of remote files. It can use open-uri + Net::HTTP, http.rb or HTTPX as the backend HTTP library.

Installation

gem "down", "~> 5.0"

Downloading

The primary method is Down.download, which downloads the remote file into a Tempfile:

require "down"

tempfile = Down.download("http://example.com/nature.jpg")
tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz.jpg>

Metadata

The returned Tempfile has some additional attributes extracted from the response data:

tempfile.content_type      #=> "text/plain"
tempfile.original_filename #=> "document.txt"
tempfile.charset           #=> "utf-8"

Maximum size

When you're accepting URLs from an outside source, it's a good idea to limit the filesize (because attackers want to give a lot of work to your servers). Down allows you to pass a :max_size option:

Down.download("http://example.com/image.jpg", max_size: 5 * 1024 * 1024) # 5 MB
# Down::TooLarge: file is too large (max is 5MB)

What is the advantage over simply checking size after downloading? Well, Down terminates the download very early, as soon as it gets the Content-Length header. And if the Content-Length header is missing, Down will terminate the download as soon as the downloaded content surpasses the maximum size.

Destination

By default the remote file will be downloaded into a temporary location and returned as a Tempfile. If you would like the file to be downloaded to a specific location on disk, you can specify the :destination option:

Down.download("http://example.com/image.jpg", destination: "/path/to/destination")
#=> nil

In this case Down.download won't have any return value, so if you need a File object you'll have to create it manually.

You can also keep the tempfile, but override the extension:

tempfile = Down.download("http://example.com/some/file", extension: "txt")
File.extname(tempfile.path) #=> ".txt"

You can also override the default tempfile prefix:

tempfile = Down.download("http://example.com/image.jpg", tempfile_name: "custom-prefix")
File.basename(tempfile.path) #=> "custom-prefix20150925-55456-z7vxqz.jpg"

Basic authentication

Down.download and Down.open will automatically detect and apply HTTP basic authentication from the URL:

Down.download("http://user:[email protected]")
Down.open("http://user:[email protected]")

Progress

Down.download supports :content_length_proc, which gets called with the value of the Content-Length header as soon as it's received, and :progress_proc, which gets called with current filesize whenever a new chunk is downloaded.

Down.download "http://example.com/movie.mp4",
  content_length_proc: -> (content_length) { ... },
  progress_proc:       -> (progress)       { ... }

Streaming

Down has the ability to retrieve content of the remote file as it is being downloaded. The Down.open method returns a Down::ChunkedIO object which represents the remote file on the given URL. When you read from it, Down internally downloads chunks of the remote file, but only how much is needed.

remote_file = Down.open("http://example.com/image.jpg")
remote_file.size # read from the "Content-Length" header, nil if unknown

remote_file.read(1024) # downloads and returns first 1 KB
remote_file.read(1024) # downloads and returns next 1 KB

remote_file.eof? #=> false
remote_file.read # downloads and returns the rest of the file content
remote_file.eof? #=> true

remote_file.close # closes the HTTP connection and deletes the internal Tempfile

The size is nil when the server doesn't send a Content-Length. With the Net::HTTP backend it's also nil when the response body is compressed, because Net::HTTP decompresses it on the fly and the header holds the compressed size. If you need the size, you can set your own Accept-Encoding, which turns off decompression; the body may then need to be decompressed by hand if the server sends it compressed anyway:

remote_file = Down.open("http://example.com/file.json", headers: { "Accept-Encoding" => "identity" })

The following IO methods are implemented:

• #read & #readpartial
• #gets
• #seek
• #pos & #tell
• #eof?
• #rewind
• #close

Caching

By default the downloaded content is internally cached into a Tempfile, so that when you rewind the Down::ChunkedIO, it continues reading the cached content that it had already retrieved.

remote_file = Down.open("http://example.com/image.jpg")
remote_file.read(1*1024*1024) # downloads, caches, and returns first 1MB
remote_file.rewind
remote_file.read(1*1024*1024) # reads the cached content
remote_file.read(1*1024*1024) # downloads the next 1MB

If you want to save on IO calls and on disk usage, and don't need to be able to rewind the Down::ChunkedIO, you can disable caching downloaded content:

Down.open("http://example.com/image.jpg", rewindable: false)

Yielding chunks

You can also yield chunks directly as they're downloaded via #each_chunk, in which case the downloaded content is not cached into a file regardless of the :rewindable option.

remote_file = Down.open("http://example.com/image.jpg")
remote_file.each_chunk { |chunk| ... }
remote_file.close

Data

You can access the response status and headers of the HTTP request that was made:

remote_file = Down.open("http://example.com/image.jpg")
remote_file.data[:status]   #=> 200
remote_file.data[:headers]  #=> { "Content-Type" => "image/jpeg", ... } (header names are normalized)
remote_file.data[:response] # returns the response object

Note that a Down::ResponseError exception will automatically be raised if response status was 4xx or 5xx.

This is the beginning of the README. Read the rest at the original. Read the full README ↗