Contents
- Overview
- Self-Hosted Inference Engines
- Managed Cloud Inference Platforms
- Platform Comparison Table
- Cost Analysis
- Selection Guide
- FAQ
- Related Resources
- Sources
Overview
Model serving platforms fall into two categories: self-hosted inference engines you deploy on GPU infrastructure, and managed cloud platforms where you pay per token or per hour with no operational overhead.
Self-hosted engines (vLLM, TensorRT-LLM, SGLang, TGI) deliver the lowest cost per token at scale but require GPU management and DevOps expertise. Managed platforms (Fireworks AI, Together AI, Replicate, Hugging Face Inference Endpoints) are faster to get started but cost more per token.
As of March 2026, the decision hinges on monthly token volume. Below ~500M tokens/month, managed platforms are cost-competitive once engineering time is factored in. Above that threshold, self-hosting becomes materially cheaper.
Self-Hosted Inference Engines
vLLM
The most widely deployed open-source inference engine. vLLM's PagedAttention memory management delivers 50-100% higher throughput than baseline PyTorch inference. Supports 200+ model architectures, OpenAI-compatible API, and multi-GPU tensor parallelism.
Best for: Teams with GPU infrastructure and DevOps capability who want maximum throughput and cost efficiency.
Typical throughput: 2,000-3,500 tokens/sec on H100 for 70B models (batch=32).
Setup time: 30 minutes to production.
See vLLM detailed guide and LLM serving framework comparison.
TensorRT-LLM
NVIDIA's native compilation-based engine. Achieves 2,400+ tokens/sec on H100 through kernel fusion and GPU-specific optimization. Requires compiling models to engine artifacts (30 minutes to 2 hours per model). NVIDIA hardware only.
Best for: High-volume deployments of stable model sets where compilation overhead is amortized and 10-15% throughput gain over vLLM justifies operational complexity.
Setup time: 2-4 hours (compilation).
SGLang
Structured generation specialist with RadixAttention for efficient prefix caching. Outperforms vLLM on multi-turn conversations and RAG workloads where context is shared across requests. Cache hit rates of 75-95% in multi-turn scenarios reduce effective compute costs substantially.
Best for: RAG pipelines, multi-turn chatbots, structured JSON output generation.
Setup time: 30-45 minutes.
Text Generation Inference (TGI)
HuggingFace's production-grade framework. Widest model compatibility (800+ HuggingFace Hub models), simple Docker deployment, built-in token streaming. Throughput trails vLLM by 20-30% but operational simplicity is unmatched.
Best for: Teams already using HuggingFace ecosystem who want the shortest path to production.
Setup time: 5-10 minutes.
Managed Cloud Inference Platforms
Fireworks AI
Token-based pricing for open-source models. Llama 3.1 70B at $0.90/million input tokens, $0.90/million output tokens. Low latency, high availability, no GPU management. Good for teams at early scale.
Together AI
Broad model catalog, competitive pricing. Llama 3.1 70B at $0.88/million tokens. Supports fine-tuned model deployment. Strong for teams needing a mix of models.
Replicate
Per-second billing model. Simple API, focus on ease of use. Higher effective cost than token-based platforms at volume, but excellent for low-traffic or experimental workloads.
Hugging Face Inference Endpoints
Deploy any HuggingFace Hub model to a dedicated endpoint. Per-hour billing on GPU hardware. Good for private models or fine-tuned variants. Uses TGI internally.
Modal
Serverless GPU compute with fast cold starts. Pay per GPU-second. Supports vLLM natively. Good middle ground between fully managed APIs and raw GPU infrastructure.
Platform Comparison Table
| Platform | Type | Llama 70B Cost | Setup | GPU Support | Custom Models |
|---|---|---|---|---|---|
| vLLM (self-hosted) | Self-hosted | ~$3-8/M tokens | 30 min | NVIDIA, AMD | Yes |
| TensorRT-LLM | Self-hosted | ~$2-5/M tokens | 2-4 hr | NVIDIA only | Yes (compile) |
| SGLang | Self-hosted | ~$3-8/M tokens | 30 min | NVIDIA, AMD | Yes |
| TGI | Self-hosted | ~$4-10/M tokens | 10 min | NVIDIA, AMD | Yes |
| Fireworks AI | Managed | $0.90/M tokens | 5 min | Managed | Yes |
| Together AI | Managed | $0.88/M tokens | 5 min | Managed | Yes |
| Replicate | Managed | Variable/sec | 5 min | Managed | Yes |
| HF Endpoints | Managed | $2-4/M tokens | 15 min | NVIDIA | Yes |
Cost estimates for self-hosted assume H100 at ~$2/hr on RunPod with realistic utilization.
Cost Analysis
Break-Even Point
At 100M tokens/month: managed APIs ($88-90) vs self-hosted vLLM on single H100 ($30-40 compute + $500+ engineering allocation) — managed APIs win on total cost.
At 1B tokens/month: managed APIs ($880-900) vs self-hosted ($300-400 compute + ~$1,000 engineering) — roughly break-even depending on team.
At 10B tokens/month: managed APIs ($8,800-9,000) vs self-hosted ($3,000-4,000 compute + $2,000 engineering) — self-hosted saves $3,000-4,000/month.
The crossover point is approximately 500M-1B tokens/month for most teams. Below this, managed platforms win on total cost of ownership. Above it, self-hosting becomes materially cheaper.
GPU Costs for Self-Hosting
For self-hosted inference, GPU rental is the primary cost:
- H100 80GB: $1.99-3.50/hr depending on provider
- A100 80GB: $1.19-2.00/hr
- RTX 4090: $0.30-0.50/hr (adequate for 7B-13B models)
See RunPod GPU Pricing and Lambda GPU Pricing for current rates.
Selection Guide
Start with managed APIs if:
- Monthly token volume below 500M
- Team lacks DevOps capacity for GPU infrastructure
- Time-to-market is the priority
- Workload is variable or unpredictable
Move to self-hosted if:
- Monthly volume exceeds 1B tokens
- Custom or fine-tuned models required
- Data privacy or compliance prevents third-party serving
- Latency requirements need dedicated infrastructure
Choose vLLM for self-hosting in most cases. Broadest model support, active community, OpenAI-compatible API, and reasonable throughput make it the default choice.
Choose TensorRT-LLM only if your team has NVIDIA expertise and serves a fixed set of high-volume models where the 10-15% throughput gain justifies compilation overhead.
Choose SGLang for RAG and multi-turn workloads where RadixAttention's prefix caching provides substantial efficiency gains.
Choose TGI if you're deeply invested in HuggingFace's ecosystem and want the simplest possible deployment path.
FAQ
Is vLLM production-ready? Yes, widely used in production. Major companies trust it for inference. Monitor closely in initial rollout like any new infrastructure.
Can we switch platforms without code changes? Most platforms support OpenAI API, so switching involves configuration change only. TGI and vLLM are drop-in replacements for each other at the API level.
What about proprietary models like GPT-4? Only OpenAI API serves GPT-4. Self-hosting requires open-source model weights (Llama, Mistral, Qwen, etc.). Claude requires Anthropic's API.
How do we handle model updates across platforms? Self-hosted: rolling deployments with Kubernetes. Managed: usually transparent to the user, though version pinning options vary by provider.
Which platform has the best quantization support? TensorRT-LLM for fine-grained NVIDIA-native quantization. vLLM for ease of use with GPTQ, AWQ, and FP8. Both support INT4 and INT8.
Can we benchmark platforms ourselves?
Yes. Use tools like lm-evaluation-harness or custom load-testing scripts. Test on representative workloads with your actual request distribution — throughput numbers from vendor benchmarks rarely match production patterns exactly.
Related Resources
- GPU Memory Requirements for LLMs
- Self-Hosting LLM Options
- Fine-Tune LLM for Chatbot
- LLM Serving Framework Comparison
- Best LLM Inference Engines
- Lambda GPU Pricing
- RunPod GPU Pricing
Sources
- vLLM Documentation (github.com/vllm-project/vllm)
- NVIDIA TensorRT-LLM (github.com/NVIDIA/TensorRT-LLM)
- Hugging Face Inference Endpoints (huggingface.co)
- Fireworks AI Documentation (fireworks.ai)
- Together AI Documentation (together.ai)
- Modal Documentation (modal.com)
- DeployBase GPU Pricing Dashboard (/gpus)