qvd2parquet

A fast command-line converter from Qlik QVD files to Parquet.

qvd2parquet input.qvd output.parquet

It reads standard, unencrypted QVD files, preserves useful Parquet types instead of stringifying everything, keeps MONEY and FIX columns as exact Parquet decimals, decodes records in parallel, and streams batches into Parquet row groups so large files never need to be materialized in memory.

Above: 2.4 million rows of an SAP-shaped extract. --inspect reports the type chosen for every column and why, before anything is written. BELNR, HKONT and KOSTL stay text because their values are zero-padded codes that an integer would not preserve; DMBTR and WRBTR become exact decimals rather than floats; BUDAT and CPUDT are read as dates although the QVD declares no type for them. The input is a generated fixture with SAP field names and shapes, not a real extract.

Install

See CHANGELOG.md for what changed in each release. From 1.0.0 the CLI surface and the conversion defaults are stable: a flag will not be removed or change its meaning, and a default will not change what an existing file converts to, outside a major bump. New behaviour arrives behind a new flag or a new value for an existing one.

Prebuilt binaries

Download the archive for your platform from the releases page, unpack it, and put qvd2parquet on your PATH. Verify the download against SHA256SUMS:

shasum -a 256 -c SHA256SUMS --ignore-missing

Binaries are pure Go and statically linked (CGO_ENABLED=0), so they have no runtime dependencies. Supported platforms:

OS	Architectures
Linux	amd64, arm64
Windows	amd64, arm64
macOS	amd64 (Intel), arm64 (Apple silicon)

Every target is pure Go, so another one is a matter of listing it in scripts/build-release.sh and the two workflow matrices; a test keeps the three lists identical.

On Windows, run qvd2parquet.exe from PowerShell or cmd. macOS may quarantine a downloaded binary; clear it with xattr -d com.apple.quarantine qvd2parquet.

From source

Requires Go 1.25 or newer. That floor comes from github.com/apache/arrow-go/v18; the go directive in go.mod is a minimum, not a pin, so the module also builds with 1.26 and 1.27. CI tests against both the 1.25 floor and the newest release, and release binaries are built with the newest toolchain.

go install github.com/ralforion/qvd2parquet/cmd/qvd2parquet@latest

or

go build -o qvd2parquet ./cmd/qvd2parquet

Building all release artifacts

./scripts/build-release.sh                        # version from git describe
VERSION=v1.2.3 ./scripts/build-release.sh          # explicit version
PLATFORMS="linux/amd64 windows/amd64" ./scripts/build-release.sh

This writes .tar.gz (Unix) and .zip (Windows) archives plus SHA256SUMS into dist/. Pushing a v* tag runs the same script in CI and publishes a GitHub release.

Usage

Options take one dash or two -- -workers 4 and --workers 4 are the same flag, as are -workers=4 and --workers=4. --help prints them with one dash; this README writes two. The one place the form matters is a boolean: write --empty-as-null=false, because --empty-as-null false leaves false sitting in the argument list as if it were a file name.

qvd2parquet [options] input.qvd output.parquet
qvd2parquet --inspect [options] input.qvd
qvd2parquet --catalog-scan --catalog-out catalog.parquet <file-or-directory>...

  -out-dir DIR               Convert every input into this directory
  -file-workers 1            Files to convert at once; decode workers are divided
  -recursive                 With --out-dir, descend into subdirectories
  -keep-tree                 With --out-dir, write each output in its input's subfolder
  -include-files 'CE*'       With --out-dir, convert only files matching these patterns
  -exclude-files '*_TMP'     With --out-dir, skip files matching these patterns
  -skip-up-to-date           With --out-dir, leave a file this run already produced
  -delete-older-than 32      With --skip-up-to-date, delete outputs written more than
                             32 days ago and keep skipping them
  -log path.jsonl            Write one JSON Lines record per input, then a summary
  -console-log run.txt       Also write the screen output to this file, as it is printed
  -catalog-out cat.parquet   Write a column-grain catalog of the run: one row per
                             output column, with its comment
  -catalog-scan              With --catalog-out, read the columns out of existing
                             .parquet inputs instead of converting
  -columns name1,name2       Convert only these columns
  -exclude '%*,*_TMP'        Skip fields matching these wildcard patterns
  -field-regex <re>          Rewrite field names with this regexp
  -field-name '${name}'      Template for the new column name
  -field-comment '${comment}'  Template for the column comment
  -mixed error               Mixed-type strategy: error|string|promote|dual-columns
  -dual auto                 Dual strategy: auto|numeric|text|columns
  -duplicate-names error     Two columns resolving to one name: error|suffix
  -infer-dates               Read an untyped column as a date/timestamp when its text says so
  -empty-as-null             Write an empty string symbol as null, as Qlik treats it
  -numeric-promote decimal   Numeric widening: decimal | true (float64) | false
  -mixed-string-fallback     Convert otherwise-invalid mixed columns to string
  -decimal-source auto       Decimal extraction: auto|text|numeric
  -decimal-strict            Fail instead of rounding when a value does not fit its scale
  -compression zstd          Parquet compression: zstd|snappy|gzip|uncompressed
  -encoding 'PAT=ENC,...'    Pin column encodings, or 'auto' to measure per file
  -batch-rows 0              Rows per Arrow batch, 0 sizes it from the column
                             count to hold in-flight memory steady
  -row-group-rows 65536      Rows per Parquet row group, raised for a file whose
                             footer would otherwise pass 8 MiB
  -workers 0                 Decode workers, 0 means one per 2 CPUs (minimum 2)
  -timezone none             none|Local|UTC|IANA timezone name
  -schema path.json          Explicit schema override
  -schema-report path.json   Write the inferred schema/profile report
  -quality-gate full         Validation mode: none|basic|numeric|full|reread
  -quality-report path.json  Write the post-conversion quality report
  -quality-tolerance 1e-9    Relative tolerance for floating-point quality checks
  -quality-abs-tolerance 0   Absolute tolerance for floating-point quality checks
  -progress 1000000          Log every N rows, 0 disables progress
  -stall-timeout 10m         Stop the run when one read or write has not returned
                             after this long, exit 8; 0 never stops
  -inspect                   Read only the header and symbol tables, print the schema, and exit
  -force                     Overwrite an existing output file
  -strict                    Enable strict validation defaults
  -version                   Print the version and exit

Output

The conversion writes the Parquet file; everything else — the identification banner, the per-column schema decisions, progress and the final summary — goes to stderr. stdout stays empty, so qvd2parquet composes safely in pipelines and shell substitutions.

$ qvd2parquet --timezone UTC --quality-gate numeric sales.qvd sales.parquet
qvd2parquet 3.2.0  (c) 2026, RALFORION d.o.o.
qvd2parquet: sales.qvd: table "products", 77 rows, 7 bytes/record, 9 of 9 columns selected
qvd2parquet: read 412 symbols in 1ms; records start at offset 8973
qvd2parquet: schema: Einkaufspreis: REAL with 75 double symbols promoted to decimal(5,2); scale 2 inferred from values
qvd2parquet: schema: Produktname: 77 text symbols, written as utf8
qvd2parquet: schema: Listenpreis: REAL with 25 integer and 35 double symbols promoted to decimal(5,2); scale 2 inferred from values
qvd2parquet: batch: 65536 rows over 9 columns (~0.6M cells per batch), 65536 rows per row group
qvd2parquet: conversion finished in 2ms: 77 rows (47934 rows/s)
qvd2parquet: quality gate numeric finished in 1ms: passed
qvd2parquet: wrote sales.parquet: 77 rows, 9 columns, 4.6 KiB in 16ms overall (4961 rows/s)

--inspect reads only the header and symbol tables, so it answers questions about a large file in seconds rather than minutes. Its table carries each column's value range rendered in the type the column will be written as, which is the point of rendering it at all: a QVD stores a date as a serial day number, so a goods-issue date in the year 3025 arrives as the integer 411241 and sits unremarked among its neighbours.

COLUMN  QVD TYPE  SYMBOLS  NULLS  PARQUET TYPE    RANGE
WADAT   DATE      2071     0      date32          2005-01-13 .. 3025-12-08
BUDAT   DATE      2020     0      date32          2019-12-27 .. 2026-08-31
VV120   REAL      179590   0      decimal(12, 2)  -8115022364.86 .. 8115022364.86
KNDNR   ASCII     27767    0      utf8

Read against BUDAT, the WADAT range is obviously wrong. Text columns have no range worth printing and show none.

A decimal's precision is inferred from the values it holds, so it fits them exactly by construction and never overflows the data it was built from. What varies is how much room is left for a later load, and that is reported separately:

Decimal columns with little room left for larger values:
  VV120 (decimal(12, 2), 81% used)
Pin them with --schema if a later load may exceed the range.

--schema-report carries the same range per column as JSON, alongside the decimal's limit and usedFraction. The --log record does not: it is one line per file and stays that way, so it carries only decimalsNearLimit, the names a scheduled run would want to alert on.

One schema: line is printed per output column, explaining exactly why each type was chosen. That is the first thing to read when a mixed-type column fails; --schema-report writes the same reasoning as JSON.

Each phase reports its own duration, so a slow run can be attributed without guessing: the conversion and the quality gate time themselves separately, and the final line is the whole run, not just the write. On a wide file the gate is routinely the larger half. Those two lines are printed whatever --progress is set to; --progress only controls the running converted N/M rows and verified N/M rows lines in between.

Those running lines carry the share done and an estimate of the time left:

qvd2parquet: converted 5000000/20589661 rows (24%) in 3m21s (24875 rows/s, about 10m26s left)

The row total comes from the QVD header before a single record is decoded, so both are arithmetic on numbers the run already has. The throughput shown is the average since the phase started; the estimate is not, it follows the recent rate, so the two can disagree while a run is speeding up. That is deliberate: a run carries the cost of starting up in its average long after it has found its speed, and projecting from the average stays pessimistic for the rest of the file. The quality gate has its own total and its own speed, so it projects separately rather than continuing the conversion's estimate.

Everything goes to stderr, so 2> run.log captures it all and leaves stdout empty.

Print the version and exit with --version:

$ qvd2parquet --version
qvd2parquet 3.2.0  (c) 2026, RALFORION d.o.o.

Examples

Convert with reproducible timestamps and validated numerics:

qvd2parquet --timezone UTC --quality-gate numeric --quality-report out.quality.json \
  sales.qvd sales.parquet

Convert a subset of a very large QVD with bounded memory:

qvd2parquet --columns CustomerID,OrderDate,Amount --batch-rows 16384 --workers 4 \
  orders.qvd orders.parquet

Strip QlikView's internal key fields and shorten composite SAP field names, keeping the description as a column comment:

qvd2parquet \
  --exclude '%*' \
  --field-regex '^[^-]*-\|\|-(?P<name>[^-]*)-\|\|-(?P<comment>.*)$' \
  A057.qvd a057.parquet

Understand why a column resolved the way it did:

qvd2parquet --schema-report schema.json sales.qvd sales.parquet

Verify the output with an external reader:

duckdb -c "describe select * from read_parquet('sales.parquet')"
duckdb -c "select count(*) as n from read_parquet('sales.parquet')"

Converting a folder

Pass --out-dir with one or more files or directories. Each .qvd becomes a .parquet of the same name:

qvd2parquet --out-dir ./parquet --quality-gate numeric --log run.jsonl ./qvds

qvd2parquet: converting 4 file(s)
qvd2parquet: ok   qvds/products.qvd -> parquet/products.parquet (77 rows, 9 columns, 4.6 KiB)
qvd2parquet: ok   qvds/sales.qvd -> parquet/sales.parquet (1,000 rows, 7 columns, 7.7 KiB)
qvd2parquet: FAIL qvds/truncated.qvd: no XML header terminator (0x00) found: not a QVD file?
qvd2parquet: ok   qvds/stock.qvd -> parquet/stock.parquet (120 rows, 19 columns, 7.3 KiB)
qvd2parquet: retrying 1 file(s) whose first attempt failed to read
qvd2parquet: waiting 10s before retrying qvds/truncated.qvd
qvd2parquet: retry qvds/truncated.qvd (first attempt: no XML header terminator (0x00) found: not a QVD file?)
qvd2parquet: FAIL qvds/truncated.qvd: no XML header terminator (0x00) found: not a QVD file?
converted 3/4 file(s) in 10.034s: 1,197 rows, 19.6 KiB; 1 retried

FAILED (1):
  qvds/truncated.qvd           no XML header terminator (0x00) found: not a QVD file? (failed on retry too)

A failing file does not stop the run. Every input is attempted — including a path that does not exist, which is reported as a failed file rather than aborting the inputs beside it. Failures are listed at the end, and the exit code reports the most actionable one — a schema policy error you can fix outranks a generic read error. Add --recursive to descend into subdirectories.

A file that fails to read is tried once more at the end of the batch. The case this is for is a file another process is still writing, such as a copy job that starts at the same time as the conversion. The bytes read then belong to no single version of the file, and the failure looks like a damaged QVD: a read that ends early (symbol 0 of 1823167: unexpected EOF), a symbol tag that is not one (unknown symbol tag 0x40), or a record that points past its symbol table (decoded symbol id 7166, but symbol table has ...). By the end of the batch the copy has usually finished, and the file converts.

Only failures known to come from reading the input are retried: its header, its symbol tables, or its records. Anything else, such as a schema policy error, an output or a report that could not be written, a schema override that does not apply, a failed quality gate, or a cancelled run, would fail the same way again, and is reported as it is. The retries run one at a time, after every other file, and each waits until 10 seconds have passed since its first attempt ended. In a long batch that time has long passed; in a short one the run says it is waiting.

Neither attempt is hidden. A file that converted on retry is listed under CONVERTED ON RETRY with the reason the first attempt failed, the log has a record for each attempt, and the exit code reflects only the second. A clean retry means something else was writing to the folder while the batch ran, which is worth knowing even when nothing was lost.

converted 2/2 file(s) in 10.033s: 2,000 rows, 16.0 KiB; 1 retried

CONVERTED ON RETRY (1), the first attempt failed:
  qvds/truncated.qvd           no XML header terminator (0x00) found: not a QVD file?

Two inputs whose names would produce the same output file are refused before anything is written, since --force would otherwise silently overwrite the first result.

Keeping the folder layout

Outputs go flat into --out-dir by default. --keep-tree writes each one in the same subfolder its input has under the directory you named, so a folder of table folders converts into a folder of table folders:

qvd2parquet --out-dir parquet-delta --keep-tree --recursive --force --skip-up-to-date qvd-delta

qvd-delta\VBAK\VBAK.qvd        ->  parquet-delta\VBAK\VBAK.parquet
qvd-delta\BSID\BSID_0928.qvd   ->  parquet-delta\BSID\BSID_0928.parquet

• A directory a wildcard matched keeps its own name: qvd-delta\* places the files of qvd-delta\VBAK in parquet-delta\VBAK, without --recursive.
• A file named directly on the command line goes in --out-dir itself.
• Two folders may hold the same file name: A\X.qvd and B\X.qvd become A\X.parquet and B\X.parquet rather than a collision.
• Per-file --schema-report and --quality-report files follow the same subfolders.
• One --skip-up-to-date manifest at the top of --out-dir covers the whole tree. Its entries are keyed by the path under --out-dir, which for a flat run is the file name it always was, so a manifest from an earlier version still skips what it skipped. Turning --keep-tree on for a folder converted flat with --recursive converts each file once more, into its subfolder; the flat outputs from before are left where they are.

Selecting which files convert

A wildcard in the last element of an input path is expanded, so a subset of a folder can be named directly:

qvd2parquet --out-dir ./parquet ./qvds/CE*.qvd

bash and zsh expand that themselves and always did. The expansion here is what makes the same line work in cmd.exe, which expands nothing, and in PowerShell, which does not expand for an external command. It takes .qvd files and directories, so ./qvds/* does not sweep up the .parquet files written beside them, and a wildcard in a directory element is refused rather than half-supported: use --recursive for a tree.

--include-files and --exclude-files filter what a directory contributes, which is what --recursive needs since no path expansion can reach into a tree:

qvd2parquet --out-dir ./parquet --recursive --include-files 'CE*,BSEG' ./qvds

• Patterns are the same wildcards as everywhere else: * and ?, case-insensitive.
• Each is matched against the file name with and without the .qvd extension, so CE*, CE*.qvd and *.qvd all behave as they look.
• --exclude-files wins over --include-files, so a broad include can be narrowed by a specific exclude.
• They filter what a directory offers. A file named on the command line was meant, and is converted whatever the patterns say.

Both the files dropped and a pattern that reached nothing are reported, since a mistyped pattern otherwise looks exactly like a folder that held only what converted:

qvd2parquet: note: --include-files/--exclude-files "ZZ*" matched no file; patterns
are wildcards over the file name, with or without the .qvd extension
qvd2parquet: note: 1 file(s) dropped by --include-files/--exclude-files, 2 left

A selection that leaves nothing is a usage error naming the reason, rather than the empty-folder message it would otherwise share:

qvd2parquet: --include-files/--exclude-files "ZZ*" left none of the 3 .qvd file(s) found in ./qvds

On Windows a pattern starting with % runs into the cmd.exe expansion covered above.

Converting only what changed

Over a folder re-extracted nightly, most inputs have not changed since the last run. --skip-up-to-date leaves those alone:

qvd2parquet --out-dir ./parquet --force --skip-up-to-date ./qvds

qvd2parquet: skip qvds/A057.qvd (up to date)
qvd2parquet: skip qvds/BSEG.qvd (up to date)
qvd2parquet: stale qvds/CE10500.qvd (input modified since, 2026-09-15T02:10:44.51Z, was 2026-09-14T02:09:58.07Z)
qvd2parquet: ok   qvds/CE10500.qvd -> parquet/CE10500.parquet (20,589,661 rows, 213 columns, 1.8 GiB)
converted 1/3 file(s) in 14m22s: 20,589,661 rows, 1.8 GiB; 2 skipped

A file that is not skipped says which check failed before it starts, so a folder that converts itself every night despite the flag can be diagnosed from the run's own output. The reasons are not in the manifest, options changed since it was converted, output was converted from <other path>, input modified since or input size changed, output modified since or output size changed, input cannot be read or output cannot be read, and output footer is <size>, over the 16 MiB Dremio reads for an output an earlier version wrote with too many row groups. The line appears only with --skip-up-to-date.

It is not a timestamp comparison. "Is the .parquet newer than the .qvd" answers the wrong question, in three ways that all end in a stale output nobody is told about:

• It cannot see that the conversion options changed. A run with a different --decimal-source or --encoding would skip a file it has never converted that way.
• A re-extracted QVD copied with its timestamps preserved (rsync -t, robocopy /copy:t) arrives looking older than the output it should replace.
• Two clocks on a network share need not agree.

So the run keeps a record instead, in .qvd2parquet-manifest.json under --out-dir, and a file is skipped only when this exact run already produced it. All five have to hold:

the manifest names the output	it was this tool that wrote it
the entry names this input	two folders can each hold an A.qvd, and both produce A.parquet
the input's size and timestamp match	the source has not been re-extracted
the options fingerprint matches	the run would write the same thing
the output's size and timestamp match	nothing has replaced it since

The input is identified by its resolved absolute path, so running the same job from a different working directory or through a symlinked parent changes nothing. Genuinely moving the folder reconverts it once.

The fingerprint covers every option that can change what is written, including the contents of a --schema override rather than only its path, the transitions of --timezone rather than its name, and the tool's major version.

The timezone is fingerprinted by what it does, because a name is not identity there: time.Local reports an IANA name only when TZ is set, and with TZ unset, which is how a server usually takes its zone from /etc/localtime, it is the bare word Local whether the machine sits in Berlin or Tokyo. Since --timezone Local writes timestamps against the converting machine's zone, a manifest that recorded only the word would be trusted by a machine that would now write different ones.

Sampling instants cannot answer it, however many are taken: America/Boise and America/Denver agree on the first of January and the first of July in every year from 1970 to 2050, and disagree through most of January 1974, when Denver took up the emergency daylight saving three weeks before Boise did. Narrowing to recent years cannot answer it either: Africa/Abidjan and GMT agree from 1970 onwards and differ by sixteen minutes of local mean time in 1900, and a QVD reaches 1900 easily, its own serial epoch being 1899-12-30.

So the fingerprint walks every transition over the whole range a conversion accepts, with nothing in between and nothing outside. It costs almost nothing: the zone table stops at its last real transition rather than projecting a rule forward for ever, so a busy zone is under two hundred steps and UTC is one. A tzdata update that moves a transition reconverts the folder, which is right rather than wasteful; the default --timezone none computes in UTC, which has no transitions at all. Options that cannot change the output are left out, so a folder does not reconvert itself over a different --workers or --progress. The stability promise is what makes the major version enough: a default does not change what an existing file converts to outside a major bump, so 2.2.0 to 2.3.0 keeps the manifest and 3.0.0 retires it.

--skip-up-to-date and --force are different questions, which is what keeps a rerun a one-flag change:

• --force is write permission: may an existing output be overwritten. Any repeat run needs it, with or without this feature.
• --skip-up-to-date is selection: which files to attempt at all.

A nightly job passes both. Dropping --skip-up-to-date reruns the whole folder, and so does deleting the manifest. Nothing is ever skipped by default.

Worth knowing:

• The manifest is written only when the flag is passed, so a run that never asks for this leaves no state behind.
• It is a dotfile, which the Parquet readers that scan a directory ignore along with any other name beginning with a dot or an underscore.
• A file that failed is not recorded, and is attempted again next time. The writer renames a temporary into place, so a failed conversion cannot have replaced the output the manifest describes.
• A manifest that is missing, corrupt or from a newer format converts the folder again rather than failing the run. The worst a lost manifest can do is repeat work.
• It is saved while the run is going, not once at the end: the first file to finish is written out at once, and again at intervals after that, spaced from how long a save takes so a folder of tens of thousands does not spend its time rewriting it. A batch that is stopped or killed halfway therefore resumes on the next run instead of converting the whole folder again. Each save goes through a temporary file and a rename, so an interrupted one leaves the previous manifest intact rather than a truncated one.
• A skipped file is still a record in --log, with "status": "skipped", so a run's log accounts for every input it was given. It carries table as well, taken from the manifest entry rather than by re-reading the input the run just decided not to touch, so a folder can be grouped by table on the runs where almost everything skips. Its counts stay zero, because they measure work this run did.

Deleting old outputs

When the output folder is only a landing zone, outputs nobody needs any more can be deleted without converting them again the next night:

qvd2parquet --out-dir ./parquet --force --skip-up-to-date --delete-older-than 32 ./qvds

qvd2parquet: skip qvds/CE10500_2021.qvd (up to date)
qvd2parquet: skip qvds/CE10500_2022.qvd (up to date; output deleted)
qvd2parquet: ok   qvds/CE10500_2026.qvd -> parquet/CE10500_2026.parquet (2,013,442 rows, 213 columns, 186.3 MiB)
qvd2parquet: deleted parquet/CE10500_2021.parquet (written over 32 days ago)
converted 1/3 file(s) in 1m51s: 2,013,442 rows, 186.3 MiB; 2 skipped, 1 old output(s) deleted

Only an output that is up to date is deleted, and only once it was written more than that many days ago. The deletions happen at the end of the run, after the manifest recording them is saved: if the save fails, nothing is deleted, rather than leaving deleted files the next run would convert back. Each file is checked again just before it is deleted, so an output another job replaced while the run was converting is left alone, and converts on the next run. The manifest keeps its entry, marked as deleted, so later runs skip the file for as long as all of these stay the same:

• the input's size and timestamp. A re-extracted QVD is new data and converts again, and the output then counts its days from that conversion.
• the options and the tool's major version, the same fingerprint as above. A major upgrade therefore converts the deleted files again, once.

A file stays skipped after it is deleted even on a run without --delete-older-than, so leaving the flag out of a manual run does not bring back a folder's worth of conversions. To get one output back, touch its QVD.

The flag needs --skip-up-to-date, because the manifest is the only record of what was deleted. A skipped file whose output is gone has "outputDeleted": true in --log, and the summary record counts the outputs this run deleted in deleted. Before an output is deleted its columns are read from the file and kept in the manifest, so --catalog-out still describes the table, with or without --force. An output whose columns cannot be read is not deleted.

Parallelism

Each decode worker reads its chunks through its own handle on the input. That matters on Windows, where concurrent ReadAt on a single file handle is serialized by the runtime -- it takes the descriptor's read and write locks and moves its shared file pointer -- so one handle would put every worker behind a single mutex.

--file-workers converts several files at once and divides the decode workers between them, so the total stays near the default worker count:

$ qvd2parquet --out-dir out --file-workers 4 ./qvds
qvd2parquet: converting 50 file(s), 4 at a time, 2 decode worker(s) each

Each file reports its own progress. Converting several at once, every line names the file it belongs to:

qvd2parquet: qvds/CE10500.qvd: converted 1000000/20589661 rows (5%) in 43.493s (23099 rows/s, about 14m11s left)
qvd2parquet: qvds/BSEG.qvd: converted 1000000/8402113 rows (12%) in 41.882s (23876 rows/s, about 5m10s left)

Converting one at a time, nothing can interleave and the prefix is left off.

The default is 1: one file at a time, decoding it with the default worker count. Raise it for many small files, where per-file parallelism beats per-chunk. This is the main reason folder conversion is built in rather than left to a shell loop — four separate processes would each start their own full set of workers, oversubscribing the machine fourfold.

The budget being divided is the automatic worker count, not one per CPU, which is why four files at a time on a 16-CPU machine get two workers each. Raise --workers alongside --file-workers when the machine has headroom for more: --file-workers 4 --workers 16 gives four each.

Querying the converted folder

A converted folder needs no further step to be queried: DuckDB reads many Parquet files as one table through a glob. With --keep-tree, each table's files sit in their own subfolder, so one glob per subfolder is one table:

qvd2parquet --out-dir out --keep-tree --recursive --skip-up-to-date qvds

-- every monthly extract of one table, with the file each row came from
select parse_filename(filename, true) as extract, count(*) as trips
from read_parquet('out/taxi/*.parquet', filename = true)
group by all order by 1;

-- a view, so the glob is written once
create view taxi as select * from read_parquet('out/taxi/*.parquet');

Add union_by_name = true when the extracts do not all have the same columns, for example after a field was added to the load script partway through the year. Columns are then matched by name, and a column a file lacks reads as null instead of failing the query.

Converting once and querying the Parquet beats reading the QVDs at query time: each file is decoded a single time, --skip-up-to-date keeps the folder current by converting only what changed, and every query after that gets Parquet's column pruning and row-group skipping. The same folder reads in Spark, Trino, Dremio, pyarrow or polars without converting again.

A glob takes every .parquet it matches, so keep --catalog-out outside the folder you query, or narrow the glob: out/*.parquet over a folder holding catalog.parquet unions the catalog's rows into the table. The --skip-up-to-date manifest is a .json file and never matches.

The log

--log writes JSON Lines: one record per file, then a summary. That format is chosen so a finished run can be queried rather than read:

qvd2parquet --log run.jsonl input.qvd output.parquet

A single-file conversion writes exactly two records: its file record and the summary. Folder conversion writes one file record per input before the summary.

Each record is written the moment its file is finished, so a batch that is stopped or killed leaves the lines for the files it did convert rather than an empty file. Two things follow. The lines are in completion order rather than input order, since files convert concurrently; every record carries time and input, so sort or group by those. And a log with no summary line is a run that did not finish, which is worth knowing rather than a defect: a query that wants the totals should select the summary record rather than assume the last line is one.

A file the batch retried has two records: the failed first attempt with "attempt": 1, and the retry with "attempt": 2. Every other record has "attempt": 1, and the summary counts the retried files in retried. The outcome of each file is its record with the highest attempt:

select * from read_json_auto('run.jsonl')
where type = 'file'
qualify row_number() over (partition by input order by attempt desc) = 1;

The path is yours to name. It is created if its directory does not exist, so --log logs/run-$(date +%F).jsonl works in a scheduled job, and a run that would otherwise leave nothing behind names its own log per day.

The screen output

The screen output is a separate thing: prose on stderr, meant to be read. --console-log copies it to a file as it is printed.

qvd2parquet --out-dir ./parquet --log run.jsonl --console-log run.txt ./qvds

It is a copy, not a redirect: the same lines still reach the screen. The file opens with the banner, so it says which build wrote it, and nothing buffers once it is open, so a run that is stopped or killed keeps every line it had printed, down to the progress of the file that was still converting. The file itself cannot be created until the path guards below have run, so the lines printed before that -- the banner, and any note about which inputs were selected -- are held and written into it when it opens. It works in every mode, including --inspect and --catalog-scan, and it takes the same path guards as --log: it must not name an input, an output, a report, the manifest, or either of the other two logs, since it is created by truncating.

The shell does the same job where there is a shell to do it with:

qvd2parquet --out-dir ./parquet ./qvds 2> run.txt            # to a file
qvd2parquet --out-dir ./parquet ./qvds 2>&1 | tee run.txt    # and on screen

--console-log is for the scheduled job that runs the binary directly, through systemd or a Windows task, where there is not one. Use both logs: the JSON log is what a query reads, and the screen log is what someone reads when a file failed and the record says why but not what led up to it. A failure that cannot be written is dropped with one note rather than failing the run, since the conversion is what the run is for and the screen still has the output.

A run that looks stuck on Windows

A conversion that stops printing, its last progress line frozen on screen, is usually not stuck: the console is paused. Windows consoles have QuickEdit on by default, and one click in the window enters selection mode, which blocks every write to the console. The next progress line blocks with it, and because progress is printed from the goroutine that writes rows, the conversion stops with it. The title bar says Select or Mark while it lasts. Press Esc or Enter in the window and the run carries on from where it was, with nothing lost.

On a long run, turn QuickEdit off in the console properties (right-click the title bar, Properties, uncheck QuickEdit Mode), or use Windows Terminal, where selecting text does not stall the writer, or keep the prose off the console altogether with one of the forms above.

If it really is stuck, the signs come together: the process using no CPU, the output's .parquet.tmp-* file not growing, and the console not in selection mode. The usual cause is a read or write the operating system never completes, most often on a network share or a drive backed by object storage: Resource Monitor's Associated Handles for qvd2parquet.exe then shows the file it is waiting on. Ctrl-C cannot end that. The first one asks the run to stop at its next step, which it never reaches, and the second hands the process to Windows, which cannot end it until the call is cancelled. Ctrl-Break is the same as Ctrl-C on Windows: Go treats it as an interrupt, not as the request for a stack dump it is on Linux and macOS (Ctrl-\ there).

--stall-timeout (default 10 minutes) is what ends such a run. Every read and write on an input or an output is timed on its own, so a conversion that keeps making progress is never stopped however long it takes, and a phase that only computes never counts. When one call has not returned in time:

qvd2parquet: stalled: write to Y:\upload\SAP\daily\ADRC.parquet.tmp-521340 has not returned
in 10m (started 09:33:12); stopping the run. The call is waiting in the operating system and
cannot be interrupted; if this process does not exit, end it with taskkill /F /PID 521340

The file is reported as stalled, the files after it as not converted, the log and the --skip-up-to-date manifest are written, and the run exits 8, so a scheduled job gets control back instead of waiting on the share for as long as it takes to recover. Nothing is retried in that run: the call is still stuck, and the next file would most likely wait on the same drive. The next run picks up from the manifest.

What is left to write after a stall, the log, the manifest and the catalog, can sit on the same share, so each gets 10 seconds, including a write that was already under way when the stall was found; one that has not returned by then is given up on with a note, and the rest of the log or the manifest is skipped rather than waiting again. If the stuck call does return in those seconds and the file then fails some other way, it is still reported as stalled, with the later error after it. Old outputs due for --delete-older-than are left for the next run. A Ctrl-C pressed while a call is stuck does not switch the watch off: the run still stops, with exit 8, once the timeout has passed.

The stuck call itself stays where it is, since nothing in the program can end it. It is cancelled, so if it ever returns it commits nothing: no output is renamed into place, and --force does not delete an existing one. Windows has to cancel it before the process can exit; taskkill /F does the same, and is the fallback the message names. Writing the outputs to a local disk and copying them to the share as a separate step keeps the conversion away from the share altogether.

The log path has to differ from every file the run writes or reads: the inputs, the outputs, --schema, and the schema and quality reports. In batch mode that covers the whole expanded input list, including any input the run could not examine, and the output and per-file report paths derived from it under --out-dir, since none of those are named on the command line. The log is created by truncating, so a collision would otherwise destroy whichever file was written second. Two spellings that differ only by case count as the same path, because the filesystem may well agree.

duckdb -c "select status, count(*), sum(rows) from read_json_auto('run.jsonl')
           where type='file' group by 1"
duckdb -c "select input, error from read_json_auto('run.jsonl') where status='failed'"
duckdb -c "select \"table\", count(*), sum(rows), sum(elapsedMs)
           from read_json_auto('run.jsonl') where type='file' group by 1 order by 4 desc"

Each file record carries table with the QVD's own table name, the row and column counts, output size, elapsed time, throughput, excludeNoMatch with any pattern that dropped nothing from that file, fieldsRenamed and fieldsUnchanged, encodings with any column not written the default way, and the quality gate's verdict with any errors, so a run can be audited without opening every per-file report. table is what a query groups by when a folder holds one table's daily extracts under names that carry a timestamp rather than the table. It is filled in on a skipped file, from the manifest, and on a failed one, by reading that file's header on its own, so a failure past the header is still grouped with the table it belongs to. It is empty only when nothing could read the header at all, which is the case where there is no name to give. --schema-report and --quality-report also work in batch mode; each file gets its own document, named after the input.

The column catalog

A field comment reaches the Parquet file as Arrow field metadata, which lives in the file-level ARROW:schema entry. pyarrow, polars and Arrow-Go decode it. Query engines mostly do not, and Dremio in particular has no column description field at all: DESCRIBE TABLE returns nine columns and none of them is a comment.

So the comment reaches a reader there only as data. --catalog-out writes that data: one Parquet row per output column, for the whole run.

qvd2parquet --out-dir out --catalog-out catalog.parquet \
  --field-regex '^[^-]*-\|\|-(?P<name>[^-]*)-\|\|-(?P<comment>.*)$' \
  extracts/

source  source_table  ordinal  column_name  comment          parquet_type
qvd     A057          1        DATBI        Ende Gültigkeit  int64
qvd     A057          2        KSCHL        Konditionsart    utf8
qvd     A057          3        KBETR        Betrag           decimal(4, 2)

Every row carries run_at, tool_version, source_file, source_table, source_rows, output_file, ordinal, column_name, source_column, comment, qlik_type, parquet_type, nullable, symbols, value_range, strategy and note. The schema is fixed and every field is written on every row, for the same reason the log writes its empty fields: a column that vanishes whenever no row happened to carry it turns where comment <> '' into a binding error on exactly the run where nothing was commented. symbols is the one nullable column, because zero symbols is a real value and a scan that cannot know the count should not claim it.

The catalog is a table, so it can be promoted as a dataset and joined. Against INFORMATION_SCHEMA.COLUMNS it answers which columns are undocumented, and which exist in the engine but not in the last conversion, which is schema drift arriving as a row rather than as a surprise. Point --catalog-out at a dated path under one directory and the runs accumulate into a history you can diff.

The catalog accumulates

A catalog already at the path is merged into, not replaced. A run describes the tables it converted, and a nightly job converts the tables that changed, so a catalog that held only the last run would describe one table on Tuesday and a different one on Wednesday.

Rows are keyed by source_table and column_name:

• a table the run did not touch keeps every row it had, with the run_at of the run that wrote them
• a column the run describes is updated in place, so a changed parquet_type, comment or note replaces what was there
• a column the run produced for the first time is added
• a column a table no longer has is kept, still carrying the run_at of the last run that saw it

Nothing is ever removed, which is what makes drift queryable rather than silent: a row whose run_at is older than the newest run for its table is a column the latest conversion did not produce.

select source_table, column_name, parquet_type, run_at
from catalog c
where run_at < (select max(run_at) from catalog where source_table = c.source_table)

A scan is reconciled with what the catalog already holds for the file it read. A scan sees a Parquet file and nothing else: it takes the table name from the file's own name, and it cannot recover qlik_type, symbols, value_range, strategy or note, which never reached the Parquet. Where the catalog holds a converted row for that file, matched on output_file, the scan adopts the table it was converted under and the column adopts the QVD side the scan cannot see. Without that, a QVD whose header names a different table than its file would arrive as two tables, and a --skip-up-to-date run over an unchanged folder would replace every row's QVD side with the blanks a scan has for it. A column of that file the conversion never wrote joins the table and stays a source='parquet' row, since nothing recorded a QVD side for it.

Two details of that matching are worth knowing. Paths are resolved before they are compared, so a conversion given a relative --out-dir and a scan naming the same directory absolutely are one file rather than two; where a stored path is relative to a working directory the scan cannot reconstruct, the file's name stands in, but only when exactly one table in the catalog wrote a file of that name. A stored path that is absolute never falls back, however the scan was spelled: the scan has just read its file from the directory it is running in, so a path that does not match an absolute stored one is a different file. And where an output has belonged to more than one table over the life of a catalog, which it can because nothing is ever removed, the scan refreshes the table the most recent conversion of that file wrote.

--force still means replace, for the run that wants to start the catalog over. The merge reads the existing file when the writer opens, before anything is converted, so a path holding a Parquet file that is not a catalog fails the run at the start rather than after the folder is done. The new catalog is written to a temporary file and renamed, so the one on disk stays intact until its replacement is complete.

The --skip-up-to-date manifest already works this way, keyed by output file name within --out-dir.

Cataloguing files you already converted

A run that forgot --catalog-out is not lost, because the comments are in the files it wrote. --catalog-scan reads them back:

qvd2parquet --catalog-scan --catalog-out catalog.parquet out/

It reads each file's footer and never a data page, so it costs a seek per file however many rows they hold, and it converts nothing. A file it cannot read is reported and the scan continues, so one bad file in a folder of hundreds does not cost the whole catalog; the run exits non-zero and the summary counts only the files it managed to read. A scan in which every file failed accounted for nothing, so no catalog is written and the run says so rather than naming one. What it cannot recover is the QVD side -- qlik_type, symbols, value_range, strategy, note -- which never reached the Parquet. source_table is taken from the file's own name, which is what --out-dir named it for, so a scan merges into a catalog on the same key a conversion does. Those rows say source='parquet' rather than source='qvd' so a query can tell the difference instead of inferring it from which fields happen to be empty. A file written by something else scans fine and simply has no comment.

--skip-up-to-date uses the same path: a skipped file writes no rows of its own, so its output is scanned instead. A run that skipped the whole folder still writes a catalog describing the whole folder.

A skipped output that cannot be read fails that file rather than dropping it from the catalog with a note. An incomplete catalog beside exit 0 is worse than none, because nothing downstream can tell that a table is missing, and a file the manifest calls current but nothing can read is worth reporting on its own account. Without --catalog-out the same folder skips silently, since nothing asked for the output to be read.

The catalog path has to differ from every file the run reads or writes, on the same terms as --log, and for the same reason: it is written by truncating. That covers the whole expanded input list, including an input the run could not examine, and the outputs and per-file reports derived from it under --out-dir. A run correctly refused for any reason must not destroy a file on its way out, so the checks that can refuse a run all happen before either writer is created, including those concerning neither file, such as two inputs whose names would produce one output.

Ordering alone is not relied on for this, because the next thing able to fail always lands somewhere new. The catalog is written only once at least one input has been converted, skipped or scanned, which is the first moment there is anything to describe. A run that ends before that -- a missing input, an output that cannot be created, another output that fails to open -- leaves whatever is at the catalog path untouched, and reports its failure through the exit code and the log, which is where a failure belongs. A file that contributed no columns, because every one was excluded, still counts as described.

A catalog that cannot be written fails the run. The conversion may well have succeeded, but a job waiting on the catalog has not got what it asked for, so the exit code says so rather than reporting success beside the error.

--catalog-scan converts nothing and so has no conversion records to write, which makes --log meaningless with it; the combination is refused rather than accepted and ignored.

Inspecting a file

--inspect reads the XML header and the symbol tables, prints the schema a conversion would produce, and exits. It never touches the record area, so the cost is independent of row count — on a 29 MiB, 5-million-row file it reads 7.9 KiB and finishes in 0.01s, against 1.58s for the full conversion.

qvd2parquet --inspect products.qvd

File            products.qvd
Table           products
Rows            77
Record size     7 bytes
Symbols read    412 in 2ms (3.2 KiB)
Records skipped 539 B not read
Fields          9 of 9 selected

COLUMN           QVD TYPE  SYMBOLS  NULLS  PARQUET TYPE   NOTES
Einkaufspreis    REAL      75       0      decimal(5, 2)  REAL with 75 double symbols promoted to decimal(5,2); scale 2 inferred from values
KategorieNr      INTEGER   8        0      int64          INTEGER with 8 integer symbols, written as int64
Produktname      UNKNOWN   77       0      utf8           77 text symbols, written as utf8
Listenpreis      REAL      60       0      decimal(5, 2)  REAL with 25 integer and 35 double symbols promoted to decimal(5,2); scale 2 inferred from values

The report is written to stdout, so it can be piped; the banner and any diagnostics stay on stderr. All type policy flags apply, so what you see is what a conversion would write — including --exclude and --field-regex:

qvd2parquet --inspect \
  --exclude '%*,COUNTER' \
  --field-regex '^[^-]*-\|\|-(?P<name>[^-]*)-\|\|-(?P<comment>.*)$' \
  A057.qvd

Fields          3 of 5 selected (2 excluded: %A057_PKEY, %SYS_TS)
Exclude         "COUNTER" matched no field
Field regex     2 of 3 field(s) renamed, 1 unchanged: PlainField

COLUMN                                    QVD TYPE  SYMBOLS  NULLS  PARQUET TYPE  RANGE           NOTES
DATBI (A057-||-DATBI-||-Ende Gültigkeit)  INTEGER   2        0      int64         45000 .. 45001  Ende Gültigkeit
KSCHL (A057-||-KSCHL-||-Konditionsart)    ASCII     2        0      utf8                          Konditionsart
PlainField                                ASCII     2        0      utf8                          2 text symbols, written as utf8

The two middle lines appear only when there is something to say. A pattern that matched no field and a field the expression left alone are both invisible in the conversion itself, and both are usually a mistake worth catching before a run that lasts a quarter of an hour.

When the type policy rejects a column, --inspect prints the reason and falls back to the raw symbol profiles that explain it, then exits 3:

Schema could not be resolved:
  schema/type policy error: mixed type column "CustomerID": symbols contain 1 numeric values and 1 strings; use --mixed=string ...

COLUMN      QVD TYPE  SYMBOLS  NULLS  INTS  FLOATS  STRINGS  DUALS
CustomerID  ASCII     2        0      1     0       1        0

That makes it a cheap pre-flight check in a pipeline: inspect first, and only convert once the schema is what you expect. --schema-report works in inspect mode too, for the same information as JSON.

Column encodings

Every column is written with a dictionary by default, which is right for the columns a QVD is usually full of: a field with twenty-seven thousand distinct values across twenty million rows costs almost nothing that way.

It is worth nothing on a column whose values are nearly all distinct, and the symbol table says so before a row is read. A column so nearly distinct that no row order could make a dictionary pay is written PLAIN from the start, raw bytes with only the compressor working on it, rather than as a dictionary page that fills up, indices for the rows it covered, and plain for the rest in every row group. A Qlik composite primary key is exactly that column: one distinct value per row. A column whose dictionary fits the page and pays whatever the row order keeps it, under every compression. In between, where only the row order could tell, the writer's default stands and --encoding auto is the measurement.

--encoding pins such a column to something better:

qvd2parquet --encoding '%*_PKEY=delta_byte_array' CE10500.qvd ce10500.parquet

qvd2parquet: encoding: %CE10500_PKEY=delta_byte_array

Patterns are wildcards, matched against both the output column name and the original QVD name, so one rule covers a folder of SAP tables whose keys are named per table. A later rule wins over an earlier one, so '*=plain,KEY=delta_byte_array' sets a default and an exception. A pattern that matches no column is reported rather than rejected, for the same reason --exclude does that: one command line is often pointed at tables that do not all carry the same fields.

The encodings on offer, and what they suit:

encoding	for
dictionary	the default: repeated values of any type
plain	values with no structure to exploit
delta_byte_array	text whose consecutive rows share a prefix, such as a sorted composite key
delta_length_byte_array	text of varying length with no shared prefix
delta_binary_packed	integers, dates and timestamps that move in small steps

An encoding the column's type cannot carry is refused before the conversion starts, naming the ones that fit. Pinning a column also turns its dictionary off, since leaving it on would mean the pinned encoding took over only once the dictionary page overflowed.

Percent signs on Windows

cmd.exe expands % before qvd2parquet sees the argument, and quoting does not stop it. Inside a .bat or .cmd file %* is replaced with the batch file's own arguments, so --encoding "%*_PKEY=delta_byte_array" arrives as --encoding "_PKEY=delta_byte_array" and matches nothing. Write %% there instead. At the interactive cmd.exe prompt the opposite holds: %* is passed through as typed, and %% would arrive as two percent signs. That is why a line that works when you type it can stop working once it is pasted into a .bat.

where	write
.bat or .cmd file	--encoding "%%*_PKEY=delta_byte_array"
interactive cmd.exe	--encoding "%*_PKEY=delta_byte_array"
PowerShell	--encoding '%*_PKEY=delta_byte_array'
bash, zsh	--encoding '%*_PKEY=delta_byte_array'

The same holds for any pattern beginning with %, --exclude '%*' included. What matched nothing is reported with the pattern quoted as it arrived, so it names the mangled form rather than the one you wrote:

qvd2parquet: note: --encoding "_PKEY" matched no column; patterns are wildcards
over the output column names and the original QVD names

Measuring instead of guessing

Whether delta_byte_array pays depends on the order the rows arrive in, since it stores each value against the one before it. On a key that arrives in document order it cuts the column to about a third; on the same values shuffled it saves nothing. Nothing in the symbol table reveals that order, so the choice has to be measured rather than reasoned about.

--encoding auto measures it. Sampled rows are written through the real writer twice, once as the run would today and once with each candidate, and the compressed size of the column chunk is compared:

qvd2parquet --inspect --encoding auto CE10500.qvd

Columns that would compress better with a different encoding:
  %CE10500_PKEY  delta_byte_array, measured 31% of current size on 300,000 sampled rows
A conversion with --encoding auto writes them that way; pin them instead with --encoding "%CE10500_PKEY=delta_byte_array".

--schema-report written from that same inspect carries the measured encoding per column, so a scripted preflight reads the same decisions the conversion will make.

On a conversion it also applies what it measured, per file:

qvd2parquet: encoding: measured 1 column(s) on 300,000 sampled rows in 91ms: 1 adopted, 0 left as they are
qvd2parquet: encoding: %CE10500_PKEY  delta_byte_array, measured 31% of current size on 300,000 sampled rows

That run wrote 1.8 MiB where the default wrote 6.2 MiB, which is what the sample predicted before the conversion started.

Details worth knowing:

• Three windows of 100,000 consecutive rows are sampled, at the head, the middle and the tail. At that size the measured ratio lands within about a point of the whole file, and it converges from above, so a sample understates a win rather than overselling it. A file of 300,000 rows or fewer is measured in full, which costs no more than three windows would and leaves no room for a sorted head to answer for a shuffled tail.
• Only columns whose values would overflow the dictionary page are measured. On a 213 column extract that is a handful, and columns are measured in groups so a wide file does not hold every sample at once.
• A candidate has to measure at 80% of the current size or better to be adopted. On a shuffled key nothing is adopted, and the run says so.
• An explicit rule always wins, and a column it names is not measured at all. --encoding 'auto,%*_PKEY=plain' measures everything except that column, and --inspect will not recommend against a rule you have already written.
• --encoding auto is the only thing that makes --inspect read records, and it reads only the sampled windows. Without it, inspect still touches nothing but the header and the symbol tables.
• Over --out-dir every file is measured on its own, which is the point: no pattern can know what each table's key looks like.

Selecting and renaming fields

--columns keeps only the named fields. --exclude drops fields matching shell-style wildcard patterns (* and ?, case-insensitive), which is the usual way to strip QlikView's internal key fields from a SAP extract:

qvd2parquet --exclude '%*' A057.qvd a057.parquet

Patterns match the field's original QVD name, before any renaming, so they describe what you see in the source file. Excluding every column is an error.

A pattern that matches nothing is not an error, since one command line is often pointed at a folder of tables that do not all carry the same fields, but it is reported:

qvd2parquet: note: --exclude "%" matched no field; patterns are wildcards over
the original QVD names, before --field-regex renames anything

Both ways of getting a pattern wrong look exactly like success otherwise. % is not the wildcard %* and matches only a field named %, and a name that --field-regex produces is never what --exclude sees. On Windows there is a third way: cmd.exe eats the % itself, which Percent signs on Windows covers.

QVD field names from SAP extracts are often composite, packing the table, the technical name and a description into one string:

A057-||-DATBI-||-Ende Gültigkeit

--field-regex rewrites them. Name the capture groups name and comment and no other flag is needed:

qvd2parquet \
  --exclude '%*' \
  --field-regex '^[^-]*-\|\|-(?P<name>[^-]*)-\|\|-(?P<comment>.*)$' \
  A057.qvd a057.parquet

qvd2parquet: excluded 2 column(s) by pattern: %A057_PKEY, %SYS_TS
qvd2parquet: schema: A057-||-DATBI-||-Ende Gültigkeit: INTEGER with 2 integer symbols, written as int64; written as "DATBI" with comment "Ende Gültigkeit"
qvd2parquet: schema: A057-||-KBETR-||-Betrag: REAL with 2 double symbols promoted to decimal(4,2); scale 2 inferred from values; written as "KBETR" with comment "Betrag"
qvd2parquet: field-regex: 2 of 3 field(s) renamed, 1 unchanged: PlainField

A field the expression does not match keeps its original name, which is what makes a rule aimed at a subset possible. The closing line says which fields those were, because on a 213 column extract nobody spots the two that stayed behind. At most five are named; --schema-report carries the full list under fieldRegex, and a --log record carries the counts.

The result carries the description as Parquet field metadata, and keeps the original QVD name so nothing is lost:

DATBI  int64             {"comment": "Ende Gültigkeit", "qvd.field": "A057-||-DATBI-||-Ende Gültigkeit"}
KSCHL  string            {"comment": "Konditionsart",   "qvd.field": "A057-||-KSCHL-||-Konditionsart"}
KBETR  decimal128(4, 2)  {"comment": "Betrag",          "qvd.field": "A057-||-KBETR-||-Betrag"}

Rules worth knowing:

• A field the expression does not match keeps its original name and gets no comment, so one rule can target a subset of the fields.
• --field-name and --field-comment accept Go regexp templates ($1, ${name}) when named groups are not enough, e.g. --field-name '${1}_${2}'.
• Two fields that collapse to the same output name are rejected as a schema policy error rather than producing a duplicate Parquet column, unless --duplicate-names=suffix is given: see Two fields under one name.
• A regex with no name group and no explicit --field-name is rejected up front, since it would blank every column name.

Two fields under one name

A rule that keeps only the technical part of a composite name is exactly the kind that collapses two fields onto one: A057-||-DATBI-||-Ende Gültigkeit and B057-||-DATBI-||-Beginn Gültigkeit both become DATBI. A generated ${name}__text companion can land on a real field of that name too, and a --columns selection can bring together fields that never met.

By default that is a schema policy error (exit code 3), because two columns under one name is usually a rule that is coarser than intended and a silent rename would hide it. The message names both source fields and the ways out:

qvd2parquet: schema/type policy error: duplicate output column name "DATBI"
(from source columns 0 and 1); source fields "A057-||-DATBI-||-Ende Gültigkeit"
and "B057-||-DATBI-||-Beginn Gültigkeit"; or pass --duplicate-names=suffix to
keep both, writing the second as "DATBI_2"

--duplicate-names=suffix keeps every column instead. The first field to claim a name keeps it, and each later one is written as ${name}_2, ${name}_3 and so on:

qvd2parquet: schema: A057-||-DATBI-||-Ende Gültigkeit: 1 text symbols, written as utf8; written as "DATBI" with comment "Ende Gültigkeit"
qvd2parquet: schema: B057-||-DATBI-||-Beginn Gültigkeit: 1 text symbols, written as utf8; written as "DATBI_2" with comment "Beginn Gültigkeit"; the name "DATBI" was already taken by an earlier column
qvd2parquet: duplicate-names: 1 duplicate output column name(s) kept under a suffix: "DATBI" -> "DATBI_2"

No data is dropped and no provenance is lost: each column keeps its own comment and its qvd.field metadata, so DATBI_2 still says which QVD field it came from. --schema-report lists every rename under duplicateNames, a --log record carries the count as duplicateNames, and --inspect prints the renames it would make.

Two details are worth knowing:

• A suffix never takes a name the file already uses. With fields A, A and A_2, the second A becomes A_3 and the real A_2 keeps its own name.
• A generated dual companion yields to a real field. If the file has both Qty (a dual) and Qty__text, the column named Qty__text is the QVD field of that name, and the display side of Qty is written as Qty__text_2, whatever order the two appear in.

Exit codes

Code	Meaning
0	success
1	CLI usage error
2	unsupported QVD feature
3	schema/type policy error
4	input read/decode error
5	output/write error
6	quality gate failure
7	cancelled by Ctrl-C or SIGTERM
8	stalled: a read or write did not return within --stall-timeout

Type resolution

The Parquet schema is resolved only after every selected column's symbol table has been read and profiled. That makes mixed-type behaviour explicit instead of producing an unstable schema that depends on which rows were seen first.

QVD field	Parquet / Arrow type
INTEGER with integer symbols	int64
REAL with double symbols	decimal128(p, s), scale inferred from the values; float64 if no exact scale exists
MONEY, FIX	decimal128(p, s) — never float64
DATE	date32 (days since the Unix epoch)
TIMESTAMP	timestamp[us], or timestamp[us, tz=UTC] when --timezone names a zone
TIME	time32[ms] (milliseconds since midnight)
ASCII or text-only symbols	utf8
all-null column	nullable utf8

Every output column is nullable.

What becomes null

• A record whose biased symbol index is negative. This is how QVD encodes a missing value, and it is by far the common case.
• A symbol carrying the 0x00 tag.
• An empty string symbol, since Qlik treats it as absent. Pass --empty-as-null=false to keep "" distinct from null.
• A NaN or infinite value in a column whose type cannot represent one — date, timestamp, time, integer or decimal. A float64 column keeps them. This is defensive: no QVD written by QlikView has been observed to store NaN, but a file produced by another writer might.

Each substitution beyond the first two is counted and reported on stderr, so none of them is silent. A finite value that simply does not fit its column is an error rather than a null, because discarding real data silently is the one thing the converter should not do.

Mixed-type columns

A QVD column can hold several symbol encodings at once. Some combinations are harmless, some are not:

• int + null, float + null — fine.
• int + float — resolved to an exact decimal by default (--numeric-promote=decimal), or widened to float64 with --numeric-promote=true.
• dual numeric + display string — a normal Qlik concept, resolved by --dual.
• number + unrelated text — never silently made numeric.

One shape of the last case resolves on its own, because nothing is actually being decided: when the numeric symbols are integers and every symbol in the column carries its own display string, the file already states the text for every value, so the column is written as utf8 without inventing a rendering for anything. The LEGO parts.qvd is the archetype -- part_num holds 0901 beside the number 901, and 0687b1 beside nothing at all. 0901 is a code, not a quantity, and reading it as 901 would not survive a round trip.

The rule stops there. A decimal beside text is more likely a measurement whose display string is a formatting of it, and a bare number carries no text to reuse, so both still fail and leave the call to you.

--mixed selects what happens to everything else:

Value	Behaviour
error (default)	fail with the counts of each symbol kind and the flag that would fix it
string	write the whole column as UTF-8; the display string wins for duals
promote	keep numerics numeric and pure text as text; still fail on number + text unless --mixed-string-fallback
dual-columns	write the numeric side under the original name and the display side as ${name}__text

A column whose values are integers stored beside their own zero-padded digits is written as utf8 under --dual=auto. The padding is part of the value: SAP keeps its keys as fixed-width codes -- BELNR is CHAR(10), so document 100000001 is stored 0100000001 -- and reading that as 100000001 produces a key that no longer joins back. Padded decimals are formatting rather than codes, and dates are neither, so both keep their own typing. Naming a side with --dual overrides the inference: --dual=numeric still writes the number, padding and all lost.

An untyped column whose display strings render its serial as a date or timestamp is read as one, so it arrives typed rather than as a bare serial with a __text sidecar. Two things bound that: a symbol carrying no value is skipped, since a null placeholder says nothing about the rest of the column, and the serial has to fall between 1600 and 2200. The window reaches back that far because historical series are real -- the Stockholm temperature record starts in 1756. It is a sanity check rather than the real filter: the display string still has to render that particular serial, which is why a month number beside Jan is not a date and keeps both columns.

--dual selects which side of a Qlik dual is written:

Value	Behaviour
auto (default)	keep the display string only when it carries something the number does not
numeric	write the numeric side only
text	write the display string only
columns	always write both

--mixed=dual-columns implies --dual=columns.

What --dual=auto considers redundant

A Qlik dual pairs a number with a display string. Often that string is just the number formatted — 1.234,56 beside 1234.56, or 11/20/2010 beside the Excel-style serial 40502. Writing it would duplicate the numeric column, so auto drops it. When the string carries something else — Open beside 1, or unknown beside -1 — it is kept as ${name}__text and the reason is stated:

schema: Amount: MONEY, written as decimal(6,2); ...; all 3 display strings are formatted renderings of the numeric value, so no text column is written
schema: Status: INTEGER with 3 integer symbols, written as int64; 3 of 3 display strings carry text the number does not (e.g. "Open" beside 1), so they are kept in "Status__text"

A single odd value is enough to keep the column: auto errs towards preserving data, and reports how many strings drove the decision.

A number is compared against the string under the field's declared separators, and where a field declares none the comparison also tries reading the string as grouped digits: 3.449 beside 3449 is a rendering, not information, even when nothing in the header says . groups thousands. An inferred grouping is held to a stricter test than a declared one. The grouping has to be exact — first group one to three digits, every later one three — so 1.5 beside 15 is still kept; a whole number has to match to the digit, so 2.000.000.000 beside 2000000001 is kept; padding is re-tested once the separators are gone, so 0.003.449 is the code 0003449 rather than a rendering of 3449; and a declared decimal separator is believed, which makes 3.449 beside 3449 informative when the field states DecSep=..

Whether a rendered date counts as redundant depends on the type it sits beside. 11/20/2010 next to a date32 column adds nothing, but next to a bare float64 serial it is the only human-readable form, so it is kept.

Identifying dates without a declared type

Qlik writes semantic tags alongside each field. A field declaring no NumberFormat/Type but tagged $date, $timestamp, $time or $interval is resolved to that type:

schema: OrderDate: DATE, written as date32 (days since epoch); no declared type, but tagged $date
schema: Created: TIMESTAMP, written as timestamp[us]; no declared type, but tagged $timestamp

This is the most reliable signal, and the only one that works for a plain numeric field carrying no display strings — a bare Excel serial that no amount of text inspection could identify. A declared type still wins over a tag, since it is the more specific statement.

Inferring dates from display strings

Older files carry no tags at all. --infer-dates (on by default) is the fallback: a column with no declared type is read as a date or timestamp when every display string renders its Excel-style serial value as one:

schema: Date: TIMESTAMP, written as timestamp[us]; no declared type, but all 18 display strings render their value as a timestamp (e.g. "11/20/2010"), so it is read as one

The check is format-agnostic: it converts the serial to a date and requires every number in the string to be one of that date's components, with the day itself present. A neighbouring day is accepted, because the string was rendered in whatever timezone wrote the file and a whole-hour offset can move the calendar date — real QVDs contain exactly this. Serials outside roughly 1900 to 2200 are never read as dates, blank display strings are not evidence, and a column mixing dates with anything else is left alone. Pass --infer-dates=false to disable it.

Output column names must be unique. If a generated ${name}__text column would collide with a real source column of that name, the conversion fails with a schema policy error rather than writing an ambiguous Parquet schema.

The defaults — --mixed=error --numeric-promote=decimal --dual=auto --mixed-string-fallback=false — stop an ETL job on unexpected schema drift while still handling the common int + float and dual-numeric cases.

Numeric widening

--numeric-promote controls how a numeric column that is not already a declared MONEY/FIX is widened:

Value	Behaviour
decimal (default)	resolve any column carrying fractional values to an exact decimal
true	widen int + float to float64
false	refuse to widen; a mixed numeric column is a policy error

--numeric-promote=decimal exists because QlikView often declares a price as plain REAL. In that case the header carries no usable scale — nDec holds a filler value (commonly 14) and the display format has no decimal separator — so the scale is derived from the values themselves: the smallest scale at which every value is exactly representable, up to 9 decimals.

$ qvd2parquet products.qvd out.parquet          # decimal is the default
qvd2parquet: schema: Einkaufspreis: REAL with 75 double symbols promoted to decimal(5,2); scale 2 inferred from values
qvd2parquet: schema: Listenpreis: REAL with 25 integer and 35 double symbols promoted to decimal(5,2); scale 2 inferred from values
qvd2parquet: schema: MengeAufLager: INTEGER with 51 integer symbols, written as int64

Pure-integer columns are left as int64, since decimal(p,0) would gain nothing, and a declared MONEY/FIX keeps using its own nDec rather than a value-inferred scale.

If no scale within 9 decimals represents every value, the column really is floating point and is written as float64. That fallback is silent by default, because the default is a preference rather than a demand and float64 is what the column would have resolved to anyway. Passing --numeric-promote=decimal explicitly turns it into a demand: the conversion then fails with a schema policy error naming the column, so you can pin it with --schema.

Caveat worth knowing. An inferred scale describes the data actually present, so a later extract containing a value with more decimals can resolve the same column to a different scale. For a pipeline that must produce a stable schema across runs, pin the column instead:

{ "columns": { "Listenpreis": { "type": "decimal", "precision": 18, "scale": 2 } } }

Exact decimals

MONEY and FIX are always written as Parquet decimals. Values are carried as scaled integers end to end, so no step of the pipeline rounds through a double.

• Scale comes from NumberFormat/nDec, or is inferred from display strings when nDec is absent. If neither is available the conversion fails and asks for a --schema override.

• Precision is inferred from the scaled values actually present, sign excluded. Exceeding decimal128's 38 digits fails with a clear message.

• --decimal-source=auto (the default) prefers the dual display string, which preserves decimal intent better than the binary double, and falls back to scaling the numeric payload. text and numeric force one source.

• By default a value that does not fit the declared scale is rounded to it the way Qlik's number format displays it: the digits the value reads as are rounded half up, toward positive infinity, whatever the double behind them holds. Measured in Qlik with Num(v, '0.00'):

  value	the double holds	written
  1.005	1.00499999999999989…	1.01
  0.285	0.28499999999999997…	0.29
  -2.345	-2.34500000000000019…	-2.34
  -2.346		-2.35

  Rounding the double itself would write 1.00, 0.28 and -2.35. A display string is rounded by the same rule. Rounding is counted and reported, both on stderr and in --schema-report, so it is never silent:

  qvd2parquet: schema: Amount: MONEY, written as decimal(9,2); ...; 3 value(s) rounded to scale 2 (--decimal-strict=false)
  qvd2parquet: note: 3 decimal value(s) were rounded to their column's scale; pass --decimal-strict to fail instead
  

• --decimal-strict restores the stricter behaviour: the conversion fails naming the column and the offending value. Use it in a pipeline where an unexpected precision change must stop the job. --strict implies it.

• A scale pinned by --schema or a folder's qvd2parquet-schema.json always rounds, whatever --decimal-strict and --strict say: the pin states what the column is. A display string that fits the scale is taken as written, and anything else is rounded as above. The count is reported the same way:

The schema report records the resolved precision and scale and whether the digits came from display strings, numeric payloads, or both.

Schema overrides

{
  "columns": {
    "Amount": { "type": "decimal", "precision": 18, "scale": 4 },
    "CustomerID": { "type": "string" }
  }
}

Supported types: string, int64, float64, date32, timestamp, time, decimal. Overrides are validated against the actual symbols before anything is written, so pinning a column holding doubles to int64, or a text column to date32, fails as a schema policy error (exit code 3) rather than silently truncating or failing part-way through the conversion.

A pinned timestamp is converted with the run's --timezone and typed like any other timestamp: timestamp[us] by default, timestamp[us, tz=UTC] when a zone is named. It reports the same conversion caveats too, so a pin that relocates a wall clock across a DST discontinuity says so rather than doing it quietly.

A schema per folder

One --schema covers a whole run, which does not fit a run over a tree of table folders: BSEG and BKPF need their own pins. So a folder can carry its own, in a file named qvd2parquet-schema.json next to the QVDs, in the same format as --schema. Every QVD in that folder is converted with it.

qvd-delta\BSEG\qvd2parquet-schema.json
qvd-delta\BSEG\BSEG_20260930.qvd
qvd-delta\BSEG\BSEG_20261001.qvd
qvd-delta\BKPF\BKPF_20261001.qvd      (no file: types are inferred)

This is what keeps daily deltas readable as one dataset. Each delta infers its types from its own values, so a day with small amounts writes decimal(5,2) and a day with large ones decimal(9,2), a day of whole numbers writes int64, and a query engine that promotes the folder sees files that disagree. Pinning the columns gives every delta the same schema, and generating the file from the main table's Parquet gives the deltas the main table's:

-- DuckDB: write BSEG's main-table types as a folder schema
copy (
  select json_object('columns', json_group_object(column_name, spec))
  from (
    select column_name,
      case
        when column_type like 'DECIMAL%' then json_object('type', 'decimal',
          'precision', greatest(18, regexp_extract(column_type, '\((\d+),', 1)::int),
          'scale', regexp_extract(column_type, ',(\d+)\)', 1)::int)
        when column_type = 'BIGINT'        then json_object('type', 'int64')
        when column_type = 'DOUBLE'        then json_object('type', 'float64')
        when column_type = 'DATE'          then json_object('type', 'date32')
        when column_type like 'TIMESTAMP%' then json_object('type', 'timestamp')
        when column_type like 'TIME%'      then json_object('type', 'time')
        else json_object('type', 'string')
      end as spec
    from (describe select * from read_parquet('parquet-main/BSEG/*.parquet'))
  )
) to 'qvd-delta/BSEG/qvd2parquet-schema.json' (format csv, header false, quote '', delimiter '\t');

Decimals are widened to at least precision 18, so a delta larger than anything in the main table still fits, and the scale is kept so both sides match.

• The file lives beside the input, not the output: the output folder is the one a query engine reads as a dataset, and a JSON file there is read as data.

• Keys match the QVD field name or, unlike --schema, the name --field-regex gives it. A file generated from Parquet already written carries the renamed names, so it works as it comes.

• With --schema as well, both apply, and a column pinned in both takes the folder's pin, whichever name each of them uses.

• --log, --catalog-out, --console-log, --schema-report and --quality-report refuse the path of any input's folder schema, existing or not: writing there would replace the pins, and a new file there would be read back as pins on the next run. The two reports refuse --schema too.

• A delta that does not fit a pin, such as cents in a column pinned to int64, fails as a schema policy error (exit code 3) instead of writing a file that disagrees with the rest.

• Each folder's file is read once per run, however many QVDs the folder holds.

• A file that cannot be read or parsed fails every QVD of its folder with an error naming it, rather than converting the folder without its pins.

• The conversion log and --inspect name the file a pin came from.

• A column the folder schema leaves unpinned is inferred per file again, so the log and --inspect warn about it by the name it is written as. This is how a field SAP adds shows up on its first day; a pin from --schema counts.

  qvd2parquet: WARNING: 1 column(s) not pinned by qvd-delta\BSEG\qvd2parquet-schema.json, so their types are inferred from this file and may differ from day to day: ZZNEW1
  

  A field SAP removes needs nothing: its pin is ignored for files without it.

• --skip-up-to-date fingerprints the file's contents per folder. Adding, editing or removing one reconverts that folder and no other, and a folder without one keeps the fingerprint it had before this lookup existed.

Dates and times

Qlik stores dates and times as serial day numbers where 25569 is 1970-01-01. A serial names no timezone: it is a bare wall-clock reading, and Qlik's own display string for a dual is a pure formatting of the same number with no zone step. Which zone that wall clock was recorded in is simply not in the file.

--timezone therefore does not change the wall clock -- every mode writes back the same calendar and clock fields, with one exception noted below. What it decides is which instant those fields denote, and so which number lands in the file:

• none (default) writes the wall clock as-is, with no timezone on the column (Parquet isAdjustedToUTC=false). It asserts nothing the QVD does not say, and the output is byte-identical whatever machine converts it. Nothing is converted unless you ask for it.
• Any IANA name (Europe/Berlin) asserts that the wall clocks were recorded in that zone and converts them to true instants. This is the mode that earns its keep for Parquet: the instant is what makes ordering across a DST change, joins against other instant data, and rendering in a consumer's own zone come out right. Losing the name costs nothing, since an instant is unambiguous without it.
• UTC is the same assertion for UTC. It stores the identical bytes as none and differs only in claiming instant semantics.
• Local is UTC's assertion made with whatever zone the converting machine happens to be in, and matches the Java reference reader. It is the one mode whose output depends on where it ran, so it is no longer the default.

The exception is a DST discontinuity. A zoned mode has to place the wall clock on the timeline, and twice a year some wall clocks do not sit there cleanly: 2023-03-26 02:30 never happens in Europe/Berlin, so it is written as 03:30, and 2023-10-29 02:30 happens twice, so one of the two instants is picked. none has no such edge, because it never places the reading on a timeline at all.

Both sides of that discontinuity turn up in real data, so both matter.

Timestamps recorded by a device following the local clock skip the missing hour. The Chicago taxi QVDs show it: on 13 March 2016 the data runs 01:45, then 03:00, with the 02:00 hour absent because the local clock skipped it.

01:45   1035 trips
03:00   1049 trips

That also settles what those values are -- local readings, not UTC, since a UTC series would have no gap.

Plenty of local-time data does contain the missing hour, though. Generated master calendars enumerate every slot whether the zone had it or not, scheduled and derived timestamps are computed in local terms, and extracts from systems that do not observe DST write wall clocks the target zone never had. A zoned conversion silently relocates every one of those, and an ambiguous reading from the repeated hour is resolved by picking one of the two instants.

That is the argument for none being the safe choice. Converting these readings to instants is only correct with the zone the source actually used -- America/Chicago for the taxi data, which nothing in the QVD says, and whose offset is not even constant across a single file. Choosing a zone is a claim about provenance that only the person running the conversion can make, and Local makes it accidentally, using whatever zone the converting machine sits in.

--timezone names the zone the input wall clocks were recorded in. It is not a label for the output: every mode but none writes tz=UTC, because once the readings are on the timeline what is stored is a UTC instant.

--timezone=none              timestamp[us]            2016-03-01 00:00:00
--timezone=UTC               timestamp[us, tz=UTC]    2016-03-01 00:00:00Z
--timezone=America/Chicago   timestamp[us, tz=UTC]    2016-03-01 06:00:00Z
--timezone=Asia/Tokyo        timestamp[us, tz=UTC]    2016-02-29 15:00:00Z

Stamping the source zone instead would split the readership. Parquet cannot record a timezone name at all -- its timestamp type carries only isAdjustedToUTC plus the unit -- so a name survives solely in Arrow's ARROW:schema metadata. pyarrow and polars would recover it and render the values back in the source zone, while DuckDB, Spark, Trino and Dremio, which see the Parquet type alone, would render the instant. Identical bytes would show two different times, and the Arrow rendering would look as though no conversion had happened. Naming UTC keeps every reader agreeing.

Engines differ in what they do with isAdjustedToUTC, so it is worth knowing your target. Measured against dremio/dremio-oss with both files loaded from S3, Dremio renders the stored value verbatim and never shifts it:

ts_naive     (isAdjustedToUTC=false)   TIMESTAMP   2023-03-15 00:00:00.000
ts_chicago   (isAdjustedToUTC=true)    TIMESTAMP   2023-03-15 05:00:00.000

Both arrive as a plain TIMESTAMP, and the same two files render identically on a Dremio whose host clock is Asia/Tokyo -- its CURRENT_TIMESTAMP stays UTC regardless. So Dremio applies no session zone in either direction, which makes both modes stable there, unlike DuckDB, which re-renders an instant in whatever zone the session is set to. The flip side is that Dremio shows no difference between the two: a wall clock and a UTC instant are the same type on arrival, so which one a column holds has to be conveyed by naming or documentation rather than by the schema.

The source zone is therefore not preserved anywhere in the file. It is an input to the conversion, not part of the result; record it in a column comment with --field-comment if the provenance matters downstream.

Timestamps are written as timestamp[us]. Microseconds are the finest unit a Qlik serial carries any signal in: one float64 ulp is about 0.63us at present-day serials. Rounding there removes the encoding noise — measured at up to 210ns on real Qlik output, which is what makes a stored 07:15:00 read back elsewhere as 07:14:59.999999 — without discarding anything the source could have expressed.

date32 and time32[ms] are timezone independent.

Parallel decoding

The record area is fixed-width, so once the symbol tables are read it can be split into contiguous row ranges and decoded concurrently. Each worker owns its Arrow builders, reads its byte range with ReadAt, and emits one Arrow record plus chunk-local quality metrics. A single writer goroutine feeds the Parquet writer, which is not safe for concurrent use.

Row order is preserved. Decoding runs in parallel, but records reach the writer in chunk order, so the Parquet file holds the QVD's rows in their original order regardless of worker count. The output is byte-comparable across worker counts, and row groups hold contiguous ranges of source rows.

That last part is what makes row-group statistics useful. A row group's per-column min/max only bound the rows it actually holds if those rows are contiguous in the source; when they are not, an engine doing row-group skipping over a sorted key cannot rule the group out. On a 500k-row fixture keyed by an ascending DocNo, the spans covered by the row groups summed to exactly 1.0x the row count, against 3.1x and 3.8x on two runs before records were ordered.

A chunk that finishes ahead of its predecessors waits in a small reorder buffer. The feeder issues at most two chunks per worker before waiting for one to be written, which bounds that buffer and is why a slow chunk cannot pull an unbounded number of finished records into memory behind it. Every quality metric is order-independent regardless, so validation is unaffected either way.

On a failure the context is cancelled, in-flight chunks are drained, and the temporary output is removed.

Cancelling a run

Ctrl-C (or SIGTERM) stops the conversion at the next chunk boundary and the quality gate at the next batch, drains what is in flight, removes the temporary output, and exits 7.

Nothing is left behind. The partial Parquet file is deleted, and an existing file at the output path is untouched — including under --force, which licenses replacing that file only on success, since the rename happens last. If the temporary file cannot be deleted, the run says so and names it, rather than leaving an unfinished Parquet file next to the real one with nothing but its .tmp- suffix to mark it. Removal is retried briefly first, because Windows refuses to delete a file while any handle is open and a virus scanner or the search indexer routinely holds one for a moment on a file just written.

One exception, for shares that refuse to overwrite a file. An S3 bucket mounted as a Windows drive lets you delete a file and take its name, but not rename onto it. There, --force falls back to deleting the old output and retrying the rename, and that replacement is not atomic. If the retry fails too, the old output is gone: the new one is kept at its .tmp- path, the error names it, and a batch stops with exit 5 rather than try the same on the next file. The files it did not reach keep their old outputs and are listed as not converted.

Cancelling is reported as cancellation, not as bad data. A stopped run has written fewer rows than the header declares, which is indistinguishable from a truncated input if you only look at the counts, and a stopped gate has verified only part of the file. Neither is an input error or a quality gate failure, and neither exits 4 or 6.

The first signal is graceful; a second one terminates immediately. That matters on a wide file, where the gate can run for minutes after the last converted line:

qvd2parquet: cancelling, finishing the current step; press Ctrl-C again to stop now
qvd2parquet: canceled after 234725 of 1000000 rows

Quality gate

--quality-gate validates the written Parquet file against metrics collected from the values the converter actually produced. Validation always reads the temporary file before the final rename, so a failed gate never leaves a final-looking output behind.

Mode	What it checks
none	nothing
basic	the file opens; row count, column names, and types match the resolved schema; per-column null counts match
numeric	everything in basic plus sum, min, max (and sum of squares for floats) per numeric, decimal, date, timestamp and time column
full (default)	everything in numeric plus order-independent sha256 value fingerprints per column
reread	everything in full plus a second, independent pass over the QVD, compared against the first

Every mode but reread validates the written Parquet against metrics collected from the values the converter produced, and none of them can question those values. A record byte read wrong yields a different symbol index, a different symbol index yields a value that is entirely well formed, and the Parquet then faithfully contains it: there is no syntax to violate and nothing downstream to notice. Only reading the source a second time can tell.

reread reads every range twice as the conversion goes and compares the two straight away: each column's symbol table before a record is decoded from it, then each record chunk before it is converted. The first range that does not match ends the conversion, naming the exact byte:

FAIL X.qvd: the file did not read the same way twice: rows 1966080..2031616:
the byte at offset 154201656 read 0x41 then 0x61. Nothing read from this file
can be trusted and no output was kept

An offset and two byte values are what anyone chasing the layer below can act on. Checking as it goes means a file whose reads do not agree costs the chunks up to the first bad one, not the whole conversion, and no output is kept.

Symbol tables are checked by streaming digest, so a table of any size costs one buffer. Record chunks are compared byte for byte, which is what names the offset; that costs one extra chunk-sized buffer and one extra file handle per decode worker.

What it cannot see is a read that is wrong the same way twice. A second read this soon after the first is likely served from the page cache, so it catches corruption after the read more readily than a bad read from storage. That is a limit of any check inside one process.

--decimal-strict fails instead of rounding, and reports up to three offending values per column with the total:

column "VV120" has 5 such value(s), the first 3 shown
    symbol 0 (float 8115022364.865): stored as 8115022364.864999771, not a multiple of 0.001
    symbol 1 (float 7115022364.865): stored as 7115022364.864999771, not a multiple of 0.001
    symbol 2 (float 6115022364.865): stored as 6115022364.864999771, not a multiple of 0.001

Each value is shown twice on purpose: as its shortest form, and as the double actually holds it. A value reading 8115022364.865 that is stored as 8115022364.864999771 is float64 representation error, and rounding it to the column's scale is more faithful than keeping it. One reading 1234.565 stored as 1234.565000000 is a genuine third decimal, and the column wants pinning with --schema. Pair it with --inspect to get the answer without converting.

--quality-report is written on success and on failure.

The gate defaults to full: a conversion nobody checked is not a conversion anybody can trust, and the fingerprints are what catch a value that survived the type policy but not the round trip.

It is not free, and it costs in two places rather than one. The read-back pass is the visible half: the whole output is read again and every cell digested, split across --workers by row group. The other half is inside the conversion — full is the only mode that fingerprints values, so each decode worker digests every value as it writes it. On a 213-column fixture, holding everything else equal and changing only the mode, conversion ran at 106k rows/s under none, 114k under numeric and 62k under full: the inline digest costs about 40% of decode throughput, while basic and numeric cost nothing there, collecting their metrics from values already being converted.

When that is too much for a run, name a cheaper mode explicitly:

--quality-gate=numeric --quality-report out.quality.json

--quality-gate=none skips validation entirely. Prefer basic over none if throughput matters: it still catches a truncated or mistyped output, and costs a fraction of full.

Performance

Measured on an Apple M3 Max (16 cores) over a 200k-row synthetic fixture with integer, high-cardinality string, decimal, date and nullable double columns.

Decode only (no Parquet writing), the fixture split into ~98 chunks:

Workers	Rows/s
1	4.6M
2	8.3M
4	13.6M
8	18.8M
--workers=0 default, 8 here	18.4M
one per CPU, 16 here	28.6M

Decoding is the parallel half of the pipeline, and it scales: 6x on 16 CPUs.

An earlier version of this table showed it flattening after four workers. That was an artifact. WorkerCount clamps to the number of chunks, and at the default 65536-row batch this fixture is only four chunks, so the 8 and NumCPU rows were both measuring four workers. The benchmark now uses a small batch and reports the worker count each case actually ran, so a clamp cannot hide in the numbers again.

--workers=0 resolves to one per two CPUs: about two thirds of the decode throughput of one per CPU, at half the batches in flight. That trade is what the default is for — on a wide file, in-flight Arrow memory is the binding constraint, not decode (see Memory). On a hyper-threaded machine, where runtime.NumCPU() counts threads, it works out to roughly one worker per physical core. Raise it when the machine has headroom; lower it when it does not. Only decoding is parallel — the Parquet writer is a single goroutine — so the full pipeline scales less steeply than this table.

Full pipeline including Parquet writing:

Compression	Rows/s	Output size
zstd	3.3M	101 KB
snappy	3.1M	235 KB
uncompressed	3.2M	2.4 MB

zstd is both the smallest and the fastest here, which is why it is the default.

Quality gate overhead (100k rows, full pipeline):

Mode	Rows/s	Overhead vs none
none	2.8M	—
basic	2.0M	~1.4x
numeric	2.0M	~1.4x
full	0.48M	~5.8x

basic and numeric cost about the same because both must read the whole Parquet file back. full adds a sha256 digest per cell, which dominates.

The overhead is per cell, not per row, so it grows with width. On a 213-column, 1M-row fixture on the same machine, wall clock for the whole run:

Mode	Wall clock	Overhead vs none
none	10.4s	—
basic	12.7s	~1.2x
numeric	12.7s	~1.2x
full (default)	27.0s	~2.6x

The read-back is split across workers by row group, the same way decoding is, so it scales with --workers: on that fixture the full gate takes 60.9s single-threaded and 9.3s at eight workers.

full is the only mode that pays part of that inside the conversion rather than after it, because only it fingerprints values, and the decode workers digest each value as they write it. Changing only the mode on the same fixture, conversion ran at 106k rows/s under none, 114k under numeric and 62k under full.

Read that against what the mode buys rather than in isolation. On a 213-column, 20.6M-row SAP extract on a 16-core Xeon, 2.0.0 with full still converted slightly faster end to end than 1.0.1 with no gate at all — 23.9k against 23.5k rows/s — because the memory the release gave back mattered more than the digest cost. 1.0.1 peaked at 26.7k early and fell to 23.5k as its resident size climbed to 36 GB; 2.0.0 climbed to 23.9k and held, at 9.3 GB. Comparing the two at any single point mid-run says nothing, since they move in opposite directions.

The gate reports its own progress on the --progress cadence, because reading back a wide file still takes a while even split across workers.

Budget for that on a wide file, or name a cheaper mode.

Batch size and row group size are separate settings, because they size different things. --batch-rows sizes what is held in memory: a batch lives in its worker and again in the queue to the writer, so it costs roughly rows * columns * 16 bytes. --row-group-rows sizes the unit a reader scans and a dictionary is built over, which is what drives output size.

--batch-rows=0, the default, picks a row count that holds about 2M cells, between 4096 and 65536 rows. A narrow file therefore still batches 65536 rows, while a 213-column file batches ~9.4k -- so in-flight memory stays put instead of growing with width. On that 213-column, 1M-row fixture, at the same row group size:

--batch-rows	workers	rows/s	peak RSS
65536	4	52.2k	7.3 GB
65536	16	57.7k	8.5 GB
0 (auto, ~9.4k rows)	4	54.7k	1.8 GB
0 (auto, ~9.4k rows)	16	95.1k	5.5 GB

The memory saving is the direct effect, but the interesting column is throughput. At a fixed 65536-row batch, four times the workers buys 10%, because each one carries a batch of 65536 * 213 cells and the machine spends its time moving memory. Sized by cells, the same workers scale: the batch size was what stopped them.

Lowering --batch-rows by hand used to shrink the row groups with it, which cost far more in output size than it saved in memory -- on this fixture a 4096-row batch tripled the file, to 486 MiB. Row group size is now --row-group-rows and holds still.

--row-group-rows is a floor rather than a fixed size, because of the footer. A Parquet footer holds one entry per column per row group, about 115 bytes each as this tool writes them, and a reader loads all of it before it reads a single value. Engines cap what they will load. Dremio stops at 16 MiB:

Failed to read parquet footer for file '.../PRCD_ELEMENTS.parquet'. Reason:
Footer size of .../PRCD_ELEMENTS.parquet is 28256268. Max supported footer
size is 16777216.

That was SAP's pricing conditions table, some 140 columns over more than a hundred million rows, which at 65536 rows per row group is around 1,750 row groups and 245,000 footer entries. So a file whose footer would pass 8 MiB, estimated at 160 bytes per entry from the row count in the QVD header and the resolved column count, gets row groups large enough to stay under it, in multiples of 65536, and says so:

qvd2parquet: row groups: 327680 rows each rather than 65536, so the footer of 115000000 rows over 140 columns stays near 8 MiB

Every file that fits, which is nearly all of them, is written exactly as before: 140 columns stay at 65536 rows per row group up to 24 million rows. The estimate is only an estimate, since the footer also carries each column's minimum and maximum per row group and a long text column makes those large, so the footer actually written is checked as well and one over 16 MiB is reported as a warning. With --skip-up-to-date, an output that an earlier version wrote with a footer over 16 MiB counts as stale and converts again, once, so the files affected do not have to be found by hand. The price of larger row groups is coarser row-group skipping and a writer that holds one compressed row group in memory, tens of MB here.

Reproduce:

go test ./internal/convert -run XXX -bench . -benchtime 3x
/usr/bin/time -l ./qvd2parquet input.qvd output.parquet   # macOS
/usr/bin/time -v ./qvd2parquet input.qvd output.parquet   # Linux

Memory

Peak memory is roughly:

• the symbol tables of the selected columns, plus
• workers * batch-rows * columns of Arrow builder memory — on a wide file this dominates, at roughly 16 bytes per in-flight cell — plus
• Parquet writer buffers, plus
• one batch-rows * RecordByteSize scratch buffer per worker.

Nothing scales with the total row count. To reduce peak memory on a large file, use --columns to skip wide string columns, then lower --batch-rows and --workers.

Not supported in v1

• Encrypted QVD files.
• Writing QVD files.
• Nested Parquet output.
• Locale-specific display formatting.
• Disk-backed symbol table caching.

Development

go test ./...                   # unit and integration tests
go test -race ./...             # the parallel decoder is race-tested
go vet ./...
./scripts/gen-notices.sh        # after a dependency change, see below
./scripts/check-action-pins.sh  # after editing a workflow, see below
./scripts/bump-version.sh 2.8.0 # to prepare a release, see below

Releasing

./scripts/bump-version.sh <version> writes the version to the three places a release records it -- defaultVersion in cmd/qvd2parquet/main.go, the two README banner lines a test pins to it, and a heading and compare links in CHANGELOG.md -- and leaves everything else to a person. It refuses a version that is not MAJOR.MINOR.PATCH, one already released, and an empty Unreleased section, since a release whose notes are empty is worse than no release at all.

What it deliberately does not decide is the number. The compatibility promise at the top of CHANGELOG.md is what separates a patch from a minor from a major, and no script can read a changelog and work out which one the entries add up to. Nor does it commit, push, tag or publish: the change goes through a pull request like any other, and pushing the v<version> tag once that is merged is what builds the archives and publishes the release.

THIRD-PARTY-NOTICES.md is generated from the module graph of ./cmd/qvd2parquet, resolving a licence per linked package rather than per module, which is what picks up the differently licensed code some dependencies vendor. Regenerate it whenever a dependency is added, removed or bumped. CI runs ./scripts/gen-notices.sh --check and fails when the committed file no longer matches, and the release workflow refuses to publish on a stale one.

Every Action the workflows use is pinned to a commit SHA rather than a version tag, because a tag is a movable label and its owner can repoint it at new code at any time. A SHA is unreadable, though, so the version comment beside it is the only thing a reviewer actually reads, and nothing makes the two agree. That gap is what ./scripts/check-action-pins.sh closes: it resolves each comment's tag upstream and fails when the SHA pinned in the workflow is not the commit that tag names, so a hash quietly swapped for one from a fork stops looking like a routine Dependabot bump. It also rejects any Action that is not SHA-pinned or whose owner is not in the script's allowlist, and any container action not pinned by digest, since an image tag such as :latest moves just as a git tag does. The comments must name exact patch releases: a major tag such as v7 moves with every release and would fail the check for a pin that is still good. CI runs it in the pins job, and the release workflow runs it too, so a tag cannot publish binaries built by steps whose pins were never checked. --offline skips the upstream lookups when you have no network, checking only the SHA and comment format.

Releasing

1. Add a section to CHANGELOG.md for the new version. A test fails if the version the CLI reports has no section, and the release workflow refuses to publish without one.
2. Bump defaultVersion in cmd/qvd2parquet/main.go.
3. Merge via a pull request; main is protected and rejects direct pushes.
4. Tag and push: git tag -a vX.Y.Z -m "..." && git push origin vX.Y.Z.

The workflow cross-compiles all 16 platforms, builds the release body from that version's changelog section, and publishes the archives with SHA256SUMS.

See testdata/README.md for fixture setup.

Package layout:

Package	Responsibility
internal/qvd	QVD format parsing only — header, symbols, bit unpacking, Qlik time. No Parquet dependency.
internal/convert	Profiling, schema resolution, type policy, decimals, parallel decoding, quality metrics.
internal/parquetwrite	Arrow/Parquet writer, temporary output and atomic rename.
internal/qvdtest	Synthetic QVD builder for tests and benchmarks.
cmd/qvd2parquet	CLI parsing, progress logging, orchestration.

Relationship to the Java reader

The behaviour is modelled on the Java QvdReader in ../qvd-reader, with two deliberate differences:

• Symbol tags 0x05 and 0x06 always carry a display string after the numeric payload, and this reader always consumes it. The Java reader skips the string for INTEGER/REAL fields, which desynchronizes the symbol stream.
• Record bits are extracted per field from BitOffset/BitWidth instead of being accumulated bit by bit with a range scan. The results are identical; the unit tests check the fast path against a direct port of the Java loop.

Symbol tables are read sequentially, using each field's declared Length to advance, exactly as the Java reader does. The per-field Offset in the header is not used for seeking.

The symbol tag layout matches the independent pyqvd description of the format, which likewise documents 0x05/0x06 as a numeric payload followed by a null-terminated string.

TestRealQVDProducts converts a QVD written by QlikView build 11282 and compares every cell against the CSV the Java reader produced from the same file. All 77 rows and 9 columns match exactly.

---

License

Copyright © 2026 RALFORION d.o.o.

Licensed under the Apache License 2.0.

Third-party licences

The binary is statically linked, so it embeds the compiled code of its Go dependencies. Their licence texts, and the NOTICE files Apache-2.0 requires to be carried forward, are reproduced in THIRD-PARTY-NOTICES.md, which ships inside every release archive alongside the binary.

All of them are permissive (Apache-2.0, MIT, BSD). None is copyleft, so nothing there places a condition on the data you convert or on software you build alongside the converter.

Built by Ralfo Becher at RALFORION, the team behind the OrionBelt Semantic Layer. About the rest of a Qlik to lakehouse migration: ralforion.com.

---