Inference & Serving
High-throughput production model serving on GPUs.
4 projects · catalog last checked 2026-10-20. "Highlight" is editorial; status, commit and release come from the weekly evidence refresh.
Comparison
| Project | What it does | Highlight (editorial) | Deployment | Hardware | Licence | Status |
|---|---|---|---|---|---|---|
| vLLM | PagedAttention-based high-throughput server | Throughput | self-hosted | gpu-required | Apache-2.0 | active |
| SGLang | Fast serving with structured generation | Structured outputs | self-hosted | gpu-required | Apache-2.0 | active |
| Text Generation Inference | Production LLM server | HF Hub integration | self-hosted | gpu-required | Apache-2.0 | ⚠ archived upstream |
| Triton Inference Server | Multi-framework model server | Mixed model fleets | self-hosted | gpu-required | BSD-3-Clause | active |
Quick start
pip install vllm
vllm serve Qwen/Qwen2.5-1.5B-Instruct --port 8000
Project details
vLLM
PagedAttention-based high-throughput server
| Upstream repository | vLLM on GitHub |
| Setup docs | vLLM setup guide |
| Licence | Apache-2.0 · licence evidence · checked 2026-10-20 |
| Licence detected by GitHub | Apache-2.0 |
| Repository state | active |
| Licence review reason | – |
| Last lookup | ok |
| Last commit | 2026-10-20 |
| Latest release | v0.32.0 (2026-10-18) |
| Deployment | self-hosted |
| Hardware | gpu-required |
| Languages | Python |
| Evidence collected | 2026-10-20 |
SGLang
Fast serving with structured generation
| Upstream repository | SGLang on GitHub |
| Setup docs | SGLang setup guide |
| Licence | Apache-2.0 · licence evidence · checked 2026-10-20 |
| Licence detected by GitHub | Apache-2.0 |
| Repository state | active |
| Licence review reason | – |
| Last lookup | ok |
| Last commit | 2026-10-20 |
| Latest release | v0.5.22 (2026-10-19) |
| Deployment | self-hosted |
| Hardware | gpu-required |
| Languages | Python |
| Evidence collected | 2026-10-20 |
Text Generation Inference
Production LLM server
| Upstream repository | Text Generation Inference on GitHub |
| Setup docs | Text Generation Inference setup guide |
| Licence | Apache-2.0 · licence evidence · checked 2026-10-20 |
| Licence detected by GitHub | Apache-2.0 |
| Repository state | ⚠ archived upstream |
| Licence review reason | – |
| Last lookup | ok |
| Last commit | 2026-03-21 |
| Latest release | v3.3.7 (2025-12-19) |
| Deployment | self-hosted |
| Hardware | gpu-required |
| Languages | Rust, Python |
| Evidence collected | 2026-10-20 |
Triton Inference Server
Multi-framework model server
| Upstream repository | Triton Inference Server on GitHub |
| Setup docs | Triton Inference Server setup guide |
| Licence | BSD-3-Clause · licence evidence · checked 2026-10-20 |
| Licence detected by GitHub | BSD-3-Clause |
| Repository state | active |
| Licence review reason | – |
| Last lookup | ok |
| Last commit | 2026-10-15 |
| Latest release | v2.73.0 (2026-09-30) |
| Deployment | self-hosted |
| Hardware | gpu-required |
| Languages | C++ |
| Evidence collected | 2026-10-20 |