tensorrt-llm Skill
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling. Published by NousResearch in hermes-agent.
What is tensorrt-llm Skill?
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling. Published by NousResearch in hermes-agent. This profile combines repository metadata with install, compatibility, and usage signals so developers can quickly decide whether it fits their agent workflow before opening the source repository.
Automated repository signals based on public metadata such as recency, license, installation evidence, and adoption. These are not a security audit or endorsement.
Key capabilities
- Includes SKILL.md support
- Reusable instructions support
- Deployment
- Deployment use cases
Technical details
- Install or run with Copy skill directory
When to use tensorrt-llm Skill
- Use it for deployment.
Built with
Editorial notes
Source
- Creator: NousResearch
- Repository: NousResearch/hermes-agent
- Skill file: optional-skills/mlops/tensorrt-llm/SKILL.md
What it does
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
Skill instructions
TensorRT-LLM NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs. When to use TensorRT-LLM Use TensorRT-LLM when: - Deploying on NVIDIA GPUs (A100, H100, GB200) - Need maximum throughput (24,000+ tokens/sec on Llama 3) - Require low latency for real-time applications - Working with quantized models (FP8, INT4, FP4) - Scaling across multiple GPUs or nodes Use vLLM instead when: - Need simpler setup and Python-first API - Want PagedAttention without TensorRT compilation - Working with AMD GPUs or non-NVIDIA hardware Use llama.cpp instead when: - Deploying on CPU or Apple Silicon - Need edge deployment without NVIDIA GPUs - Want simpler GGUF quantization format Quick start Installation bash Docker (recommended) docker pull nvidia/tensorrtllm:latest pip install pip install tensorrtllm==1.2.0rc3 Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12 Basic inference python from tensorrtllm import LLM, SamplingParams Initialize model ll
Explore related resources
Frequently asked questions
What is tensorrt-llm?
tensorrt-llm is a open-source AI agent skill with Copy skill directory. Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency.
Who is tensorrt-llm best for?
tensorrt-llm is best for reusing agent instructions, scripts, and references, deployment workflows.
How do I install tensorrt-llm?
Install or run tensorrt-llm using Copy skill directory. Check tensorrt-llm for the latest setup command.
Is tensorrt-llm actively maintained?
tensorrt-llm may need a closer maintenance check before production use.
Auto-fetched from GitHub.
