RTX PRO 6000 Blackwell LLM Wiki

This repository is a field wiki for running frontier LLMs on NVIDIA RTX PRO 6000 Blackwell / SM120 PCIe systems. It is more than a few launch snippets: it contains reproducible Docker builds, exact vLLM and SGLang runbooks, benchmark tables, KLD quality checks, quantization notes, DCP/MTP/DSpark/DFlash debugging, PCIe topology work, and regression history.

Start Here

For GLM-5.3, Qwen3.8-Flash-Next, DeepSeek V4 text/Vision and V4.1, start with the shared Docker launch guide. One image provides model profiles, speculation, GPU selection and optional LMCache. Each model page has a complete launch command, exact option examples and recorded speeds. Use the recipe archive to reproduce a preceding release without changing the recommended model URLs.

Model family	Start here	Scope
GLM-5.3-Flash	GLM-5.3-Flash	Unified TP4 profile, NVFP4 target, no-spec/MTP3/MXFP8 DFlash2 choices, bounded decode/prefill/Sieve measurements and quality reports.
GLM-5.3-Flash Spark	Two-GPU recipe	Memory-tuned Spark checkpoint preset for two 96-GB GPUs, MTP3 and optional RAM/disk prefix storage.
GLM-5.2	GLM-5.2 Runbook Hub	Fathomless vLLM, NVFP4, online FP8/MXFP8, B12X, DCP, MTP, KLD.
DeepSeek-V4.1-Flash	DeepSeek-V4.1-Flash	Unified TP4 profile, native text/vision, adaptive DSpark K7, RAM/SSD Engram tables, bounded API and throughput evidence.
DeepSeek-V4-Flash / DSpark	DeepSeek V4 text, DeepSeek V4 Vision	Unified TP2 profiles, fixed DSpark K5/K3, B12X, optional LMCache and measured image comparisons.
Kimi	Kimi Runbook Hub	Kimi-K2.7-Code, DFlash, parser/tool-call runtime.
Xiaomi MiMo	MiMo Runbook Hub	MiMo V2.5 Pro FP4-DFlash.
Qwen3.8-Flash-Next	Qwen3.8-Flash-Next	Unified TP1/TP2 profile, MTP3, CPU PLE tables, optional vision and FP8 KV.
Qwen3.8-27B	Qwen3.8-27B on RTX PRO 6000 Blackwell, readable QSRT K5 training result, exact QSRT K5 specification	TP1, TP2, and TP4 throughput evidence plus the QSRT K5 training interpretation, artifact, fidelity, runtime, and source contract.
GLM-5.1	GLM-5.1 Runbook Hub	Historical GLM-5.1, KLD methodology, older B12X/SGLang work.
Legacy / secondary models	Legacy Model Runbooks	DeepSeek-V4-Pro, GLM-4.7, Qwen, MiniMax, older Kimi pages.

Need the complete map of every Markdown page?

Index	Use
Full Wiki Index	Complete generated catalog of every page in this repository.
Glossary And Acronym Guide	Acronym expansions and writing rules for newcomer-friendly docs.
Newcomer Onboarding	How to ask useful questions without lowering the technical signal.

What Is In This Repository?

Need	Where
Shared image launch, speculation and cache settings	Unified Docker guide, then the relevant model profile page. Other model families retain their own runbooks.
Rebuild the unified image	Versioned runtime configuration and assembly sources; historical recipes remain in model release pages and scripts.
Compare backend speed	Model benchmark tables plus Benchmark Results.
Check quantization fidelity	General KLD methodology, GLM-5.2 KLD, and model-specific KLD sections.
Compare model output quality on a real Czech tutoring workload	Czech tutoring quality comparison: DeepSeek-V4-Flash, V4.1-Flash, Qwen3.8-Flash-Next and GLM-5.3-Flash per reasoning mode, with a reproducible runner.
Understand MTP, DSpark, or DFlash	Speculative Decoding, DS4/Kimi/MiMo pages.
Debug topology or PCIe behavior	Topology, PCIe Bandwidth, .

Serving Guides and Reproduction Records

Area	Page	Why it matters
GLM-5.3-Flash serving stack	GLM-5.3-Flash	Profile-based MTP3/DFlash2 launches and bounded wheel-image measurements; no-spec, DCP4 and clock-specific history remain attributed to their release.
GLM-5.2 serving stack	GLM-5.2 Infernal Invocation r18	Source-qualified CUDA 13.3 profiles with sparse-prefill row validation, projection-mixed EXL3 TP4, online MCG K6, and NVFP4 TP8.
GLM-5.2 MXFP4	GLM-5.2 FP8 + MXFP4 Experts	Native MXFP4 expert checkpoint path and A8 serving notes.
DS4 text and Vision serving profiles	DeepSeek V4 text, Vision	One image, separate fixed-K5/K3 profiles, optional external cache, GPU KV capacity and matched throughput tables.
Shared GLM/Qwen/DS4/DS4.1 image	Unified Docker guide	One launch interface and image selection point; model-specific alternatives, cache support and a six-profile measurement table.
DS4 full reference	DS4 DSpark v9	Full DSpark and standard MTP sweep reference.
Kimi-K2.7-Code	Kimi-K2.7-Code v3	Fathomless Kimi DFlash validation.

Older pages are intentionally preserved. Prefer the hub page for each model family unless you are reproducing a specific historical result.

Core Topics

Topic	Page
Docker images and release lines	Docker Images
Unified model profiles, speculation and LMCache	Shared Docker launch guide
PCIe oneshot all-reduce	PCIe oneshot all-reduce
NCCL tuning and empty graph-file failures	NCCL tuning
Speculative decoding	Speculative decoding
NVFP4 quantization	NVFP4 quantization
Hybrid NVFP4 assembly	Hybrid NVFP4 assembly
B12X FP8 / DeepGEMM comparison	B12X dense FP8 GEMM vs DeepGEMM
B12X W4A8 tiny decode	B12X W4A8 MX tiny decode
DSpark upstream consolidation	DSpark upstream consolidation

Benchmarks And Quality

Area	Page
Consolidated throughput	Benchmark Results
Shared beta serving-image comparison	Karmic Kraken model measurements; archived comparisons retain their separate image and clock conditions.
vLLM vs SGLang throughput	Inference throughput
GLM-5.2 KLD and quant quality	GLM-5.2 KLD Evaluation
General KLD methodology	Measuring quantization distribution fidelity in vLLM
MTP quality checks	MTP Quality Evaluation
NVFP4 quantization comparison	NVFP4 Quantization Comparison
GLM-5.3-Flash behavioral fidelity	R30 nucleus-sampling comparison of published NVFP4, QAD, and QAD TV-nucleus
GLM-5.3-Flash QAD step-2,500 fidelity	Qualified temperature-1/top-p-0.95 practical-equivalence result

KLD is a regression and quantization-sanity tool, not a complete quality metric. Use it together with long-context decode, coding probes, acceptance-rate checks, and task-level benchmarks.

Hardware And Topology

Most measurements in this repository target RTX PRO 6000 Blackwell / GB202 / SM120 cards: 96 GB GDDR7 per GPU, PCIe 5.0 x16, no NVLink, usually 4-GPU, 8-GPU, or 16-GPU PCIe-switch systems.

Area	Page
SM120 vs SM100	SM120 vs SM100 Architecture
PCIe topology	Topology
PCIe bandwidth	PCIe Bandwidth
GPU configs	GPU Configurations
ASUS ESC8000A-E13P	ASUS ESC8000A-E13P + Broadcom Switches
ASRockRack Turin 16 GPU	ASRockRack + EPYC Turin + 4x c-payne
ASRock WRX90 16 GPU	ASRock WRX90 + 4x c-payne
Power tuning	Blackwell power limit sweep

Inference Engines

Engine	Page	Role
vLLM	vLLM	Primary runtime for supported GLM-5.2, DS4, Kimi, and MiMo pages.
FlashInfer	FlashInfer	SM120 sparse MLA, CUTLASS MoE, sampler, and kernel integration notes.
SGLang	SGLang	Historical and alternate runtime notes, especially older GLM/MiMo paths.

Acronym Policy

The wiki uses many acronyms: DCP, MTP, DSpark, DFlash, MLA, MoE, KLD, TP, CC, P2P, NVFP4, MXFP8, and more. To make pages readable for newcomers:

• Expand important acronyms on first use: Decode Context Parallelism (DCP).
• Do not expand acronyms inside commands, Docker tags, environment variables, JSON, file paths, or raw logs.
• Use Glossary And Acronym Guide as the source of truth.
• Run python3 scripts/check-acronyms.py before polishing a major page.

Keeping The Community Useful

The wiki should reduce accidental gatekeeping: newcomers should be able to find the right runbook, decode acronyms, and reproduce a known-good launch without needing tribal knowledge. That does not mean every Discord question can be answered from memory.

Use Newcomer Onboarding as the support contract: bring the model page, Docker image, full launch command, GPU layout, TP/DCP/MTP or DSpark settings, client command, and logs. That keeps the server welcoming without turning it into an unstructured support queue.

Common Operational Rules

• Do not launch with NCCL_GRAPH_FILE= set to an empty string. Unset it if no real XML graph file is used.
• Reuse cache directories while debugging; otherwise TileLang, Triton, CuTe, and FlashInfer rebuilds dominate iteration time.
• For quick smoke tests, use small MAX_NUM_SEQS and graph caps. For published tables, use the graph sizes documented in the model page.
• For DFlash and DSpark, confirm backend markers and acceptance rates before trusting throughput numbers.
• For GLM-5.2, keep the exact index_topk_pattern and DCP policy from the relevant runbook; a truncated pattern can silently degrade output.

Maintaining The Wiki

When adding a page:

• Link it from the relevant model hub.
• Start a serving guide with a complete copy-paste launch command and practical hardware requirements. Explain each configurable setting with its exact flag or environment variable, default and example override.
• Keep source commits, checkpoint revisions, full benchmark commands and review status in a linked technical report. A normal setup guide does not need a qualification-status block.
• Identify the model, mode, image, hardware and clocks for every speed table. Preserve preceding release recipes in the archive before replacing a model's recommended instructions.
• Regenerate the full index:

python3 scripts/generate-wiki-index.py > INDEX.md

For performance claims, include both the server launch config and the client command so results can be reproduced on another PCIe-only Blackwell host.

Measurements retain their hardware, source and sampling conditions; model hubs separate reusable deployment instructions from historical release evidence.