qsv.dathere.com

qsv: Blazing-fast Data-Wrangling toolkit

Table of Contents

Hi-ho "Quicksilver" away!original logo details * Base AI-reimagined logo * Event logo archive

qsv is a neuro-symbolic data-wrangling toolkit for querying,slicing, sorting, analyzing, filtering, enriching, transforming,validating, geocoding, formatting, converting, visualizing,FAIRifying, chatting, joining & documenting tabular data(CSV/TSV/SSV, JSONL, parquet, spreadsheets, etc).Commands are simple, composable & "blazing fast".* Commands* Installation: CLI • MCP Server • Cowork Plugin * Whirlwind Tour / Notebooks / Lessons & Exercises* FAQ* Performance Tuning* 👉 Benchmarks  •  Interactive Benchmarks 🚀* 📊 Visualizations Gallery 🗺️* 🗃️ -, , 📈* * * * * ///   /* series - • • • *

Try it out at qsv.dathere.com!

Command	Description
apply✨📇🧠🤖🚀🔣👆⛩️	Apply series of string, date, math & currency transformations to given CSV column/s. It also has some basic NLP functions (similarity, sentiment analysis, profanity, eudex, language & name gender) detection. Its summarize subcommand condenses a column or group of columns using an OpenAI API-compatible LLM (local or commercial) with customizable, MiniJinja-templated per-record prompts.
applydp✨📇🚀🔣👆	applydp is a slimmed-down version of apply with only Datapusher+ relevant subcommands/operations (qsvdp binary variant only).
behead	Drop headers from a CSV.
blake3🚀	Compute or check BLAKE3 hashes of files.

✨: enabled by a feature flag.
📇: uses an index when available.
🤯: loads entire CSV into memory, though dedup, stats & transpose have "streaming" modes as well.
😣: uses additional memory proportional to the cardinality of the columns in the CSV.
🧠: expensive operations are memoized with available inter-session Redis/Disk caching for fetch commands.
🗄️: Extended input support.
🗃️: Limited Extended input support.
🐻‍❄️: command powered/accelerated by vectorized query engine.
🤖: command uses Natural Language Processing or Generative AI.
🏎️: multithreaded and/or faster when an index (📇) is available.
🚀: multithreaded even without an index.
: has CKAN-aware integration options.
🌐: has web-aware options.
🔣: requires UTF-8 encoded input.
👆: has powerful column selector support. See select for syntax.
🪄: "automagical" commands that uses stats and/or frequency tables to work "smarter" & "faster".
📚: has lookup table support, enabling runtime "lookups" against local or remote reference CSVs.
🌎: has geospatial capabilities.
⛩️: uses MiniJinja template engine.
: uses Luau 0.740 as an embedded scripting DSL.
🎲: randomly generated or randomized output with a --seed option for reproducibility. 🖥️: part of the User Interface (UI) feature group.

Installation Options

[!TIP] To install the qsv MCP Server and/or the qsv Claude Cowork plugin, see the Getting Started guide.

[!TIP] To run qsv from Python, use qsv-client (pip install qsv-client). It stops qsv and every process qsv started on a timeout, raises typed exceptions instead of making you parse stderr, and reports the installed binary's version, features and commands. It has sync and asyncio clients and no runtime dependencies. qsv itself is not bundled: install qsv 25.0.0 or later using one of the options below.

Option 0: qsv pro

If you prefer to explore your data using a graphical interface instead of the command-line, feel free to try out qsv pro. Leveraging qsv, qsv pro can help you quickly analyze spreadsheet data by just dropping a file, along with many other interactive features. Learn more at qsvpro.dathere.com or download qsv pro directly by clicking one of the badges below.

Option 1: Download Prebuilt Binaries

Full-featured prebuilt binary variants of the latest qsv version for Linux, macOS & Windows are available for download, including binaries compiled with Rust Nightly (more info). You may click a badge below based on your platform to download a ZIP with pre-built binaries.

Prebuilt binaries for Apple Silicon, Windows for ARM, IBM Power Servers (PowerPC64 LE Linux) and IBM Z mainframes (s390x) have CPU optimizations enabled (target-cpu=native). The macOS Apple Silicon, Linux x86_64 (GNU), Linux ARM64 (GNU) and Windows x86_64 (MSVC) prebuilts are also compiled with Profile Guided Optimization (PGO) for even more performance gains.

x86_64 prebuilts target the x86-64-v3 microarchitecture level, so they require a CPU with AVX2, FMA and BMI1/BMI2 - e.g. Intel Haswell (2013) / AMD Excavator (2015) or newer. Some newer low-end chips (Intel Pentium/Celeron/Atom parts up to ~2020) and VMs with a conservative CPU model (e.g. QEMU/Proxmox kvm64/qemu64) do NOT qualify. Check with grep -o avx2 /proc/cpuinfo | head -1 on Linux, or Coreinfo on Windows. On a CPU without AVX2 the prebuilts fail with Illegal Instruction (SIGILL) - compile qsv from source instead, which uses the portable baseline by default. (We use x86-64-v3 rather than target-cpu=native, since native ties a binary to the build runner's exact CPU.) qsvdp is the exception: it stays on the portable x86-64 baseline, as DataPusher+ often runs on VMs without AVX2.

For Windows, an MSI "Easy installer" is also available. After downloading and installing the Easy installer, launch the Easy installer and click "Install qsv" to download the latest pre-built binary to a folder that is added to your PATH. Afterwards qsv should be installed and you may launch a new terminal to use qsv.

For macOS, "ad-hoc" signatures are used to sign our binaries, so you will need to set appropriate Gatekeeper security settings or run the following command to remove the quarantine attribute from qsv before you run it for the first time:

# replace qsv with qsvmcp, qsvlite, qsvdp, qsvpy* if you installed those binary variants
sudo xattr -d com.apple.quarantine qsv

An additional benefit of using the prebuilt binaries is that they have the self_update feature enabled, allowing you to quickly update qsv to the latest version with a simple qsv --update. For further security, the self_update feature only fetches releases from this GitHub repo and automatically verifies the signature of the downloaded zip archive before installing the update.

[!NOTE] The luau feature is not available in musl prebuilt binaries².

Manually verifying the Integrity of the Prebuilt Binaries Zip Archives

All prebuilt binaries zip archives are signed with zipsign with the following public key qsv-zipsign-public.key. To verify the integrity of the downloaded zip archives:

# if you don't have zipsign installed yet
cargo install zipsign

# verify the integrity of the downloaded prebuilt binary zip archive
# after downloading the zip archive and the qsv-zipsign-public.key file.
# replace <PREBUILT-BINARY-ARCHIVE.zip> with the name of the downloaded zip archive
# e.g. zipsign verify zip qsv-0.118.0-aarch64-apple-darwin.zip qsv-zipsign-public.key
zipsign verify zip <PREBUILT-BINARY-ARCHIVE.zip> qsv-zipsign-public.key

Option 2: Package Managers & Distributions

qsv is also distributed by several package managers and distros.

Here are the relevant commands for installing qsv using the various package managers and distros:

# Arch Linux Extra Repository (https://archlinux.org/packages/extra/x86_64/qsv/)
pacman -S qsv

# Homebrew on macOS/Linux (https://formulae.brew.sh/formula/qsv#default)
brew install qsv

# MacPorts on macOS (https://ports.macports.org/port/qsv/)
sudo port install qsv

# Mise on Linux/macOS/Windows (https://mise.jdx.dev)
mise use -g qsv@latest

# Nixpkgs on Linux/macOS (https://search.nixos.org/packages?channel=unstable&show=qsv&from=0&size=50&sort=relevance&type=packages&query=qsv)
nix-shell -p qsv

# Scoop on Windows (https://scoop.sh/#/apps?q=qsv)
scoop install qsv

# Void Linux (https://voidlinux.org/packages/?arch=x86_64&q=qsv)
sudo xbps-install qsv

# Conda-forge (https://anaconda.org/conda-forge/qsv)
conda install conda-forge::qsv

Note that qsv provided by these package managers/distros enable different features (Homebrew, for instance, enables the apply, fetch, foreach, geocode, lens, luau and to features. However, it does automatically install shell completion for bash, fish and zsh shells).

To find out what features are enabled in a package/distro's qsv, run qsv --version (more info).

In the true spirit of open source, these packages are maintained by volunteers who wanted to make qsv easier to install in various environments. They are much appreciated, and we loosely collaborate with the package maintainers through GitHub, but know that these packages are maintained by third-parties.

Debian package

datHere also maintains a Debian package targeting the latest Ubuntu LTS on x86_64 architecture to make it easier to install qsv with DataPusher+.

To install qsv on Ubuntu/Debian:

wget -O - https://dathere.github.io/qsv-deb-releases/qsv-deb.gpg | sudo gpg --dearmor -o /usr/share/keyrings/qsv-deb.gpg
echo "deb [signed-by=/usr/share/keyrings/qsv-deb.gpg] https://dathere.github.io/qsv-deb-releases ./" | sudo tee /etc/apt/sources.list.d/qsv.list
sudo apt update
sudo apt install qsv

Option 3: Compile from Source

If you have Rust installed, you can compile from source³:

git clone https://github.com/dathere/qsv.git
cd qsv
cargo build --release --locked --bin qsv --features all_features

The compiled binary will end up in ./target/release/.

To compile different variants and enable optional features:

# to compile qsv with all features enabled
cargo build --release --locked --bin qsv --features feature_capable,apply,fetch,foreach,geocode,geoconnex,get,get_cloud,luau,mcp,magika,polars,profile,self_update,synthesize,to,ui,viz_static
# shorthand
cargo build --release --locked --bin qsv -F all_features
# enable all CPU optimizations for the current CPU (warning: creates non-portable binary)
CARGO_BUILD_RUSTFLAGS='-C target-cpu=native' cargo build --release --locked --bin qsv -F all_features

# or build qsv with only the fetch and foreach features enabled
cargo build --release --locked --bin qsv -F feature_capable,fetch,foreach

# for qsvmcp - MCP server optimized variant
cargo build --release --locked --bin qsvmcp -F qsvmcp

# for qsvlite
cargo build --release --locked --bin qsvlite -F lite

# for qsvdp
cargo build --release --locked --bin qsvdp -F datapusher_plus

[!NOTE] To build with Rust nightly, see Nightly Release Builds. The feature_capable, qsvmcp, lite and datapusher_plus are MUTUALLY EXCLUSIVE features. See Special Build Features for more info.

Variants

There are five binary variants of qsv:

• qsv - feature-capable(✨), with the prebuilt binaries enabling all applicable features except Python ²
• qsvpy - same as qsv but with the Python feature enabled. Four subvariants are available - qsvpy311, qsvpy312, qsvpy313 & qsvpy314 - which are compiled with the latest patch version of Python 3.11, 3.12, 3.13 & 3.14 respectively. We need to have a binary for each Python version as Python is dynamically linked (more info).
• qsvmcp - optimized for MCP (Model Context Protocol) server use with geocode, mcp, polars, profile, readstat, self_update, synthesize, to, and viz features enabled. Shares src/main.rs with qsv.
• qsvlite - all features disabled (~16% of the size of qsv). If you are migrating from xsv and want the same experience and feature set, this is the variant for you.
• qsvdp - optimized for use with DataPusher+ with only DataPusher+ relevant commands; applydp, a slimmed-down version of the apply feature; the --progressbar option disabled; and the self-update only checking for new releases, requiring an explicit --update (~16% of the size of qsv).

[!NOTE] Past releases also shipped "portable" subvariants with a "p" suffix (qsvp, qsvplite, qsvpdp, qsvpmcp), compiled without CPU features for users hitting "Illegal instruction (SIGILL)" errors. They are being discontinued: the x86_64 prebuilts stopped enabling target-cpu=native (it was the cause of those SIGILL faults), so the subvariants no longer differed from the regular binaries. (The x86_64 prebuilts have since moved to the x86-64-v3 baseline - see above - so a CPU without AVX2 now needs a source build.) Which release archives still contain them varies by release and platform; if you have been using one, switch to the regular binary of the same name (qsvp → qsv, qsvplite → qsvlite, qsvpdp → qsvdp, qsvpmcp → qsvmcp).

Tool Definitions

qsv and qsvmcp can describe their own commands in machine-readable form for agents and other tools. The definitions are generated from the binary's own usage text, so they always match your installed version and feature set:

qsv stats --help --format json   # one command's JSON tool definition
qsv stats --help --format md     # its help as Markdown
qsv --export-tool-definitions defs   # all installed commands, plus a manifest.json

--export-tool-definitions <dir> writes tool-definitions/qsv-<command>.json, help/<command>.md (with a TableOfContents.md) and a manifest.json that records the qsv version, the binary, its installed commands and which of them are in the MCP server's curated skill set.

Shell Completion

qsv has extensive, extendable shell completion support. It currently supports the following shells: bash, zsh, powershell, fish, nushell, fig & elvish. You may download a shell completions script for your shell by clicking one of the badges below:

To customize shell completions, see the Shell Completion documentation. If you're using Bash, you can also follow the step-by-step tutorial at 100.dathere.com to learn how to enable the Bash shell completions.

Regular Expression Syntax

The --select option and several commands (apply, applydp, datefmt, exclude, fetchpost, replace, schema, search, searchset, select, sqlp, stats & validate) allow the user to specify regular expressions. We use the regex crate to parse, compile and execute these expressions. ⁴

Its syntax can be found here and "is similar to other regex engines, but it lacks several features that are not known how to implement efficiently. This includes, but is not limited to, look-around and backreferences. In exchange, all regex searches in this crate have worst case O(m * n) time complexity, where m is proportional to the size of the regex and n is proportional to the size of the string being searched."

If you want to test your regular expressions, regex101 supports the syntax used by the regex crate. Just select the "Rust" flavor.

[!CAUTION] JSON SCHEMA VALIDATION REGEX: The schema command, when inferring a JSON Schema Validation file, will derive a regex expression for the selected columns when the --pattern-columns option is used. Though the derived regex is guaranteed to work, it may not be the most efficient.Before using the generated JSON Schema file in production with the validate command, it is recommended that users inspect and optimize the derived regex as required.While doing so, note that the validate command in JSON Schema Validation mode, can also support "fancy" regex expressions with look-around and backreferences using the --fancy-regex option.

File formats

qsv recognizes UTF-8/ASCII encoded, CSV (.csv), SSV (.ssv) and TSV files (.tsv & .tab). CSV files are assumed to have "," (comma) as a delimiter, SSV files have ";" (semicolon) as a delimiter and TSV files, "\t" (tab) as a delimiter. The delimiter is a single ascii character that can be set either by the --delimiter command-line option or with the QSV_DEFAULT_DELIMITER environment variable or automatically detected when QSV_SNIFF_DELIMITER is set.

When using the --output option, qsv will UTF-8 encode the file & automatically change the delimiter used in the generated file based on the file extension - i.e. comma for .csv, semicolon for .ssv, tab for .tsv & .tab files.

JSON files are recognized & converted to CSV with the json command. JSONL/NDJSON files are also recognized & converted to/from CSV with the jsonl and tojsonl commands respectively.

The fetch & fetchpost commands also produces JSONL files when its invoked without the --new-column option & TSV files with the --report option.

The excel, safenames, sniff, sortcheck & validate commands produce JSON files with their JSON options following the JSON API 1.1 specification, so it can return detailed machine-friendly metadata that can be used by other systems.

The schema command produces a JSON Schema Validation (Draft 2020-12) file with the ".schema.json" file extension, which can be used with the validate command to validate other CSV files with an identical schema.

The describegpt and frequency commands also both produce TOON files. TOON is a compact, human-readable encoding of the JSON data model for LLM prompts.

The excel command recognizes Excel & Open Document Spreadsheet(ODS) files (.xls, .xlsx, .xlsm, .xlsb & .ods files).

The readstat command recognizes the binary formats of the major statistical packages - SAS (.sas7bdat, .xpt/.xpt5/.xpt8), Stata (.dta) and SPSS (.sav, .zsav, .por) - & converts them to CSV. It can also dump their variable metadata (labels, value labels & missing-value codes) instead of the data.

Speaking of Excel, if you're having trouble opening qsv-generated CSV files in Excel, set the QSV_OUTPUT_BOM environment variable to add a Byte Order Mark to the beginning of the generated CSV file. This is a workaround for Excel's UTF-8 encoding detection bug.

The to command converts CSVs to Parquet, Excel .xlsx, LibreOffice/OpenOffice Calc .ods & Data Package formats, and populates PostgreSQL and SQLite databases.

The sqlp command returns query results in CSV, JSON, JSONL, Parquet, Apache Arrow IPC & Apache AVRO formats. Polars SQL also supports reading external files directly in various formats with its read_csv, read_ndjson, read_parquet & read_ipc table functions.

The sniff command can also detect the mime type of any file with the --no-infer or --just-mime options, may it be local or remote (http and https schemes supported). It can detect more than 130 file formats, including MS Office/Open Document files, JSON, XML, PDF, PNG, JPEG and specialized geospatial formats like GPX, GML, KML, TML, TMX, TSX, TTML. Click here for a complete list.

[!TIP] When the polars feature is enabled, qsv can also natively read .parquet, .ipc, .arrow, .json & .jsonl files.

Extended Input Support

The cat, headers, sqlp, to & validate commands have extended input support (🗄️). If the input is - or empty, the command will try to use stdin as input. If it's not, it will check if its a directory, and if so, add all the files in the directory as input files.

If its a file, it will first check if it has an .infile-list extension. If it does, it will load the text file and parse each line as an input file path. This is a much faster and convenient way to process a large number of input files, without having to pass them all as separate command-line arguments. Further, the file paths can be anywhere in the file system, even on separate volumes. If an input file path is not fully qualified, it will be treated as relative to the current working directory. Empty lines and lines starting with # are ignored. Invalid file paths will be logged as warnings and skipped.

For both directory and .infile-list input, snappy compressed files with a .sz or .zip extension will be automatically decompressed.

Finally, if its just a regular file, it will be treated as a regular input file.

Limited Extended Input Support

The describegpt, lens, slice & tojsonl commands have limited extended input support (🗃️). They are different in that they only process one file. If provided an .infile-list or a compressed .sz or .zip file, they will only process the first file.

Note on .zip inputs. qsv treats a .zip archive as a container of delimited-text files (CSV/TSV/TAB/SSV). The first such entry (in archive order) is used; commands with full Extended Input Support use all of them. Directory and system entries (__MACOSX, .DS_Store, …) are skipped, and path-traversal ("zip-slip") entries are rejected. Nesting a special binary format (Parquet, Avro, or Arrow) inside a .zip is not a recommended workflow — those formats are already compressed, so zipping them serves no purpose — and it is not handled uniformly: most commands select only the first CSV/TSV/TAB/SSV entry (and error if a .zip contains none), while the commands with full Extended Input Support (cat, headers, sqlp, to & validate) do extract non-tabular supported entries. For predictable results, provide .parquet/.avro/.arrow files directly instead (qsv reads them natively). See #3988.

Automatic Compression/Decompression

qsv supports automatic compression/decompression using the Snappy frame format. Snappy was chosen instead of more popular compression formats like gzip because it was designed for high-performance streaming compression & decompression (up to 2.58 gb/sec compression, 0.89 gb/sec decompression).

For all commands except the index, extdedup & extsort commands, if the input file has an ".sz" extension, qsv will automatically do streaming decompression as it reads it. Further, if the input file has an extended CSV/TSV ".sz" extension (e.g nyc311.csv.sz/nyc311.tsv.sz/nyc311.tab.sz), qsv will also use the file extension to determine the delimiter to use.

Similarly, if the --output file has an ".sz" extension, qsv will automatically do streaming compression as it writes it. If the output file has an extended CSV/TSV ".sz" extension, qsv will also use the file extension to determine the delimiter to use.

Note however that compressed files cannot be indexed, so index-accelerated commands (frequency, schema, split, stats, tojsonl) will not be multithreaded. Random access is also disabled without an index, so slice will not be instantaneous and luau's random-access mode will not be available.

There is also a dedicated snappy command with four subcommands for direct snappy file operations — a multithreaded compress subcommand (5-6x faster than the built-in, single-threaded auto-compression); a decompress subcommand with detailed compression metadata; a check subcommand to quickly inspect if a file has a Snappy header; and a validate subcommand to confirm if a Snappy file is valid.

The snappy command can be used to compress/decompress ANY file, not just CSV/TSV files.

Using the snappy command, we can compress NYC's 311 data (15gb, 28m rows) to 4.95 gb in 5.77 seconds with the multithreaded compress subcommand - 2.58 gb/sec with a 0.33 (3.01:1) compression ratio. With snappy decompress, we can roundtrip decompress the same file in 16.71 seconds - 0.89 gb/sec.

Compare that to zip 3.0, which compressed the same file to 2.9 gb in 248.3 seconds on the same machine - 43x slower at 0.06 gb/sec with a 0.19 (5.17:1) compression ratio - for just an additional 14% (2.45 gb) of saved space. zip also took 4.3x longer to roundtrip decompress the same file in 72 seconds - 0.20 gb/sec.

[!TIP] qsv has additional compression support beyond Snappy:

The sqlp command can:

• Automatically decompress gzip, zstd and zlib compressed input files
• Automatically compress output files when using Arrow, Avro and Parquet formats (via --format and --compression options)

When the polars feature is enabled, qsv can automatically decompress these compressed file formats:

• CSV: .csv.gz, .csv.zst, .csv.zlib
• TSV/TAB: .tsv.gz, .tsv.zst, .tsv.zlib; .tab.gz, .tab.zst, .tab.zlib
• SSV: .ssv.gz, .ssv.zst, .ssv.zlib

.zip archives are auto-decompressed in all builds (including qsvlite — zip needs no polars): the first CSV/TSV/TAB/SSV entry in archive order is used, with the delimiter taken from that entry's extension. A .zip with no CSV/TSV/TAB/SSV entry errors rather than parsing an arbitrary file as CSV.

This automatic decompression applies both to commands that read through qsv's standard CSV reader (e.g. count, stats, frequency) and to reference data loaded as luau, validate & describegpt lookup tables.

Compressed remote/downloaded sources are also auto-decompressed before caching, so the cached copy is plain CSV (correctly delimited, auto-indexed): remote luau/validate/describegpt lookup tables (http/https, dathere://, ckan://) and get/dc: cached resources. .zip/.sz work in every build that has those commands; .gz/.zlib/.zst need the flate2/zstd codecs (bundled in the standard qsv & qsvmcp builds), otherwise an actionable error names the missing build feature.

Commands with Extended and Limited Extended Input support handle .zip differently: they expand a .zip's contents as input files (all entries for Extended Input commands like cat/sqlp/to; only the first entry for Limited Extended Input commands like /).

RFC 4180 CSV Standard

qsv follows the RFC 4180 CSV standard. However, in real life, CSV formats vary significantly & qsv is actually not strictly compliant with the specification so it can process "real-world" CSV files. qsv leverages the awesome Rust CSV crate to read/write CSV files.

Click here to find out more about how qsv conforms to the standard using this crate.

When dealing with "atypical" CSV files, you can use the input & fmt commands to normalize them to be RFC 4180-compliant.

UTF-8 Encoding

qsv requires UTF-8 encoded input (of which ASCII is a subset).

Should you need to re-encode CSV/TSV files, you can use the input command to "lossy save" to UTF-8 - replacing invalid UTF-8 sequences with � (U+FFFD REPLACEMENT CHARACTER).

Alternatively, if you want to truly transcode to UTF-8, there are several utilities like iconv that you can use to do so on Linux/macOS & Windows.

Windows Powershell and Windows Excel Usage Note

Unlike other modern operating systems, Microsoft Windows' default encoding is UTF16-LE. This will cause problems when redirecting qsv's output to a CSV file in Powershell & trying to open it with Excel - everything will be in the first column, as the UTF16-LE encoded CSV file will not be properly recognized by Excel.

# the following command will produce a UTF16-LE encoded CSV file on Windows
qsv stats wcp.csv > wcpstats.csv

Which is weird, since you'd think Microsoft's own Excel would properly recognize UTF16-LE encoded CSV files. Regardless, to create a properly UTF-8 encoded file on Windows, use the --output option instead:

# so instead of redirecting stdout to a file on Windows
qsv stats wcp.csv > wcpstats.csv

# do this instead, so it will be properly UTF-8 encoded
qsv stats wcp.csv --output wcpstats.csv

Alternatively, qsv can add a Byte Order Mark (BOM) to the beginning of a CSV to indicate it's UTF-8 encoded. You can do this by setting the QSV_OUTPUT_BOM environment variable to 1.

This will allow Excel on Windows to properly recognize the CSV file as UTF-8 encoded.

Note that this is not a problem with Excel on macOS, as macOS (like most other *nixes) uses UTF-8 as its default encoding.

Nor is it a problem with qsv output files produced on other operating systems, as Excel on Windows can properly recognize UTF-8 encoded CSV files.

Interpreters

For complex data-wrangling tasks, you can use Luau and Python scripts.

Luau is recommended over Python for complex data-wrangling tasks as it is faster, more memory-efficient, has no external dependencies and has several data-wrangling helper functions as qsv's DSL.

See Luau vs Python for more info.

Another "interpreter" included with qsv is MiniJinja, which is used in the template and fetchpost commands.

Memory Management

qsv supports three memory allocators - jemalloc (default), mimalloc and the standard allocator.See Memory Allocator for more info.

It also has Out-of-Memory prevention, with two modes - NORMAL (default) & CONSERVATIVE.See Out-of-Memory Prevention for more info.

Environment Variables & dotenv file support

qsv supports an extensive list of environment variables and supports .env files to set them.

For details, see Environment Variables and the dotenv.template file.

Feature Flags

qsv has several feature flags that can be used to enable/disable optional features.

See Features for more info.

Minimum Supported Rust Version

qsv's MSRV policy is to require the latest stable Rust version that is supported by Homebrew, currently . qsv itself may upgrade its MSRV, but a new qsv release will only be made once Homebrew supports the latest Rust stable.

Goals / Non-Goals

QuickSilver's goals, in priority order, are to be:

• As Fast as Possible - To do so, it has frequent releases, an aggressive MSRV policy, takes advantage of CPU features, employs various caching strategies, uses HTTP/2, and is multithreaded when possible and it makes sense. It also uses the latest dependencies when possible, and will use Cargo patch to get unreleased fixes/features from its dependencies. See Performance for more info.
• Able to Process Very Large Files - Most qsv commands are streaming, using constant memory, and can process arbitrarily large CSV files. For those commands that require loading the entire CSV into memory (denoted by 🤯), qsv has Out-of-Memory prevention, batch processing strategies and "ext"ernal commands that use the disk to process larger than memory files. See Memory Management for more info.
• A Complete Data-Wrangling Toolkit - qsv aims to be a comprehensive data-wrangling toolkit that you can use for quick analysis and investigations, but is also robust enough for production data pipelines. Its many commands are targeted towards common data-wrangling tasks and can be combined/composed into complex data-wrangling scripts with its Luau-based DSL.
  Luau will also serve as the backbone of a whole library of qsv recipes - reusable scripts for common tasks (e.g. street-level geocoding, removing PII, data enrichment, etc.) that prompt for easily modifiable parameters.
• Composable/Interoperable - qsv is designed to be composable, with a focus on interoperability with other common CLI tools like 'awk', 'xargs', 'ripgrep', 'sed', etc., and with well known ETL/ELT tools like Airbyte, Airflow, Pentaho Kettle, etc. Its commands can be combined with other tools via pipes, and it supports other common file formats like JSON/JSONL, Parquet, Arrow IPC, Avro, Excel, ODS, PostgreSQL, SQLite, etc. See File Formats for more info.

QuickSilver's non-goals are to be:

• As Small as Possible - qsv is designed to be small, but not at the expense of performance, features, composability, portability, usability, security or maintainability. However, we do have a qsvlite variant that is ~16% of the size of qsv and a qsvdp variant that is ~16% of the size of qsv. Those variants, however, have reduced functionality. Further, several commands are gated behind feature flags, so you can compile qsv with only the features you need.
• Multi-lingual - qsv's usage text and messages are English-only. There are no plans to support other languages. This does not mean it can only process English input files.
  It can process well-formed CSVs in any language so long as its UTF-8 encoded. Further, it supports alternate delimiters/separators other than comma; the apply whatlang operation detects 87 languages; and its apply thousands, currency and eudex operations supports different languages and country conventions for number, currency and date parsing/formatting.
  Finally, though the default Geonames index of the geocode command is English-only, the index can be rebuilt with the geocode index-update subcommand with the --languages option to return place names in multiple languages (with support for 254 languages).

Testing

qsv has 6,808 tests: 6,071 for the qsv binaries (4,741 integration in the tests directory + 1,330 unit across src/ modules), plus 737 for the MCP server & Cowork plugin (in .claude/skills/tests). Most commands have their own test suite in a separate file with the convention test_<COMMAND>.rs. Apart from preventing regressions, the tests also serve as good illustrative examples, and are often linked from the usage text of each corresponding command.

To test each binary variant:

# to test qsv
cargo test --features all_features

# to test qsvlite
cargo test --features lite
# to test all tests with "stats" in the name with qsvlite
cargo test stats --features lite

# to test qsvmcp
cargo test --features qsvmcp

# to test qsvdp
cargo test --features datapusher_plus

# to test a specific command
# here we test only stats and use the
# t alias for test and the -F shortcut for --features
cargo t stats -F all_features

# to test a specific command with a specific feature
# here we test only luau command with the luau feature
cargo t luau -F feature_capable,luau

# to test the count command with multiple features
# we use "test_count" as we don't want to run other tests
# that have "count" in the testname - e.g. test_geocode_countryinfo
cargo t test_count -F feature_capable,luau,polars

# to test using an alternate allocator
# other than the default jemalloc allocator
cargo t --no-default-features -F all_features,mimalloc

License

MIT

Third-Party Software

qsv vendors and redistributes several open source components — among them plotly.js and (bundled inside it) MapLibre GL JS, which power qsv viz; DataTables, which powers the viz smart data viewer; LuaDate, used by qsv luau; and vendored DCAT-US, DCAT-AP and geoconnex schemas used by qsv profile.

THIRD_PARTY_NOTICES.md records each component's upstream, version or pinned commit, and license, together with the copyright notices and license texts those licenses require us to carry. Sample datasets have their own credits in examples/viz/README.md — note that world_cities.csv is GeoNames-derived and CC BY 4.0, which requires attribution if you reuse it.

If you think something here is attributed incorrectly or not at all, please open an issue — we would rather over-attribute than under-attribute.

Origins

qsv is a fork of the popular xsv utility. Building on this solid foundation, it was forked in Sept 2021 and has since evolved to a general purpose data wrangling toolkit, adding numerous commands and features. See FAQ for more details.

AI Policy

qsv has a very fast development pace with frequent releases. To keep up with this pace, we use the latest AI tools to help with various tasks, including code generation, documentation, testing, bug fixing, code review, etc. To ensure the quality of the work, we maintain more than 6,000 tests; use multi-agent, multi-model, adversarial code reviews with roborev and GitHub Copilot; heavily comment the code to help Agents, Users and Developers understand it; and use tools like Serena and Context7 to give Agents semantic context.

We accredit the Agent/Model used in commit messages and pull request descriptions. Contributors must do the same — see the AI Policy in CONTRIBUTING.md for the full contributor guidelines (attribution, review, licensing).

Sponsor

qsv was made possible by
Standards-based, best-of-breed, open source solutionsto make your Data Useful, Usable & Used.

Naming Collision

This project is unrelated to Intel's Quick Sync Video.

Footnotes

1. see validate_index benchmark ↩

2. The luaufeature is NOT enabled by default on the prebuilt binaries for musl platforms. This is because we cross-compile using GitHub Action Runners using Ubuntu 20.04 LTS with the musl libc toolchain. However, Ubuntu is a glibc-based, not a musl-based distro. We get around this by cross-compiling.
   Unfortunately, this prevents us from cross-compiling binaries with the luau feature enabled as doing so requires statically linking the host OS libc library. If you need the luau feature on musl, you will need to compile from source on your own musl-based Linux Distro (e.g. Alpine, Void, etc.). ↩ ↩²

3. Of course, you'll also need a linker & a C compiler. Linux users should generally install GCC or Clang, according to their distribution’s documentation. For example, if you use Ubuntu, you can install the build-essential package. On macOS, you can get a C compiler by running $ xcode-select --install. For Windows, this means installing Visual Studio 2022. When prompted for workloads, include "Desktop Development with C++", the Windows 10 or 11 SDK & the English language pack, along with any other language packs your require. ↩