TensorRT LLM

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT

翻訳待ち

開発ツールPython · C++GitHub @NVIDIA接続状況の再確認が必要です所有者未確認

プロジェクトを見る GitHub

Have you tried it? One tap is enough. No sign-in needed.

接続状況: 再確認が必要; 30日間の稼働率 100%
コードの最終更新: 11月13日; 更新中のリポジトリ
最新バージョン: v1.4.0rc3; 11月7日
貢献者: 420; フォーク 2,930
ライセンス: 未特定・原文をご確認ください
nomorevibe への掲載日: 9月20日; リポジトリ作成 2023年8月

Languages

Python 74.9%
C++ 17.9%
Cuda 6.1%
Groovy and others 1.1%

Updates3 in the last 30 days

All

11月7日v1.4.0rc3GitHub release ↗
10月29日v1.4.0rc2GitHub release ↗
10月22日v1.4.0rc1GitHub release ↗

Comments

Newest first

···

GitHub issue delivery settings

Checking sign-in status.

Loading comments.

情報

Operator: @NVIDIA ↗GitHub repository owner
Website: nvidia.github.io/TensorRT-LLM ↗
Repository: nvidia/tensorrt-llm ↗公開 · 更新中
Site and repository: Repository links to this site
Category: 開発ツール
利用方法: Website
Technology stack: Python
Information refreshed: 11月13日GitHub

Do you operate this project?

Owners of public repositories under a personal GitHub account can sign in to verify ownership and manage the introduction themselves. No administrator approval is needed.

Organization repository · verify with a repository file

Sign in, generate a verification file, and commit it to this repository’s default branch within 30 minutes. Files on other branches or forks do not count. Then check the file below to manage the project without administrator approval. Keep the file to retain your access.

If needed, you will go to sign-in. After signing in, return here and press your chosen verification button again.

Run /nomorevibe verify in your project folder to verify ownership and manage the introduction yourself.

Verify in one minute

Ask about other verification options

Request an introduction correction or removal

Do you operate this project?

Owners of public repositories under a personal GitHub account can sign in to verify ownership and manage the introduction themselves. No administrator approval is needed.

Organization repository · verify with a repository file

If needed, you will go to sign-in. After signing in, return here and press your chosen verification button again.

Run /nomorevibe verify in your project folder to verify ownership and manage the introduction yourself.

Verify in one minute

Request an introduction correction or removal

Introduction

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

翻訳待ち

README

TensorRT LLM

Architecture | Performance | Examples | Documentation | Roadmap

Tech Blogs

[11/11] Dense DWDP: Extending Distributed Weight Data Parallelism to Dense Weights on Vera Rubin NVL72 ✨ ➡️ link
[11/03] ISER-MoE: In-Switch Expert Replication for Load-Balanced MoE Prefill ✨ ➡️ link
[10/14] KV Cache Compression in TensorRT LLM ✨ ➡️ link
[09/22] GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K ✨ ➡️ link
[09/02] Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention in TensorRT-LLM ✨ ➡️ link
[08/20] Evaluating Agentic Serving with Trace Replay and Job-Level Metrics ✨ ➡️ link
[07/17] DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM ✨ ➡️ link
[07/01] Scaling Video Generation Across NVL72 Rack with TensorRT-LLM ✨ ➡️ link
[05/15] Joint Optimization of Agent Applications and TensorRT-LLM ✨ ➡️ link
[04/03] Tuning CUDA Graph Batch Sizes for Higher Output Throughput ✨ ➡️ link
[04/03] DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72 ✨ ➡️ link
[03/16] Optimizing MoE Communication with One-Sided AlltoAll Over NVLink ✨ ➡️ link
[03/04] Sparse Attention in TensorRT LLM ✨ ➡️ link
[02/06] Accelerating Long-Context Inference with Skip Softmax Attention ✨ ➡️ link
[01/09] Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs ✨ ➡️ link

[09/26] Inference Time Compute Implementation in TensorRT LLM ✨ ➡️ link
[09/19] Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly ✨ ➡️ link
[08/29] ADP Balance Strategy ✨ ➡️ link

This is the beginning of the README. Read the rest at the original. Read the full README ↗

Read the full README ↗

TensorRT LLM

Languages

Updates3 in the last 30 days

Comments

Trending in 開発ツール

TensorRT LLM

Languages

Updates3 in the last 30 days

Comments

Trending in 開発ツール

Introduction

README

TensorRT LLM

Tech Blogs

README

README

TensorRT LLM

Tech Blogs