How to Deploy LLMs on RunPod

Direct answer: deploying an LLM on RunPod means renting a GPU cloud instance, uploading or downloading your model weights, choosing a serving framework, and exposing an HTTP endpoint. The entire process takes 10–30 minutes if you already know your model size and quantization target. If you are starting from scratch, budget an hour for GPU selection and cost estimation.

What you will learn in this guide

• How to pick the right GPU and VRAM for your model size and batch size.
• How quantization (GPTQ, AWQ, GGUF) cuts memory usage and what it costs in quality.
• How to serve an LLM with vLLM, Hugging Face TGI, or llama.cpp.
• How to estimate total cost per 1,000 tokens at different load levels.
• How to switch from a single GPU to multi-GPU when you outgrow one card.
• Safety and compliance notes for production APIs.

GPU selection table for common LLMs

Step 1: create a pod and download the model

Log into RunPod and create a pod. For most testing, start with an RTX 3090. If you need FP16 for a 70B model, rent an A100 80GB or a multi-GPU configuration.

Use the PyTorch template or a community template with CUDA 12.1+ and transformers 4.40+. Connect via SSH or Jupyter, then download the model using huggingface-cli or snapshot_download.

huggingface-cli download meta-llama/Meta-Llama-3.1-8B-Instruct \
  --local-dir /workspace/models/llama-3.1-8b \
  --local-dir-use-symlinks False

Step 2: quantize (optional but recommended)

Quantization reduces VRAM usage by storing weights in lower precision. The tradeoff is a small accuracy drop that is usually invisible for chat and generation tasks but can matter for exact reasoning benchmarks.

AWQ (recommended for vLLM)

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Meta-Llama-3.1-8B-Instruct"
quant_path = "/workspace/models/llama-3.1-8b-awq"
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4 }

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

GGUF via llama.cpp (great for single-GPU consumer cards)

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make -j

python convert_hf_to_gguf.py \
  /workspace/models/llama-3.1-8b \
  --outfile /workspace/models/llama-3.1-8b-q4_k_m.gguf \
  --outtype q4_k_m

Step 3: serve with vLLM (highest throughput)

vLLM uses PagedAttention to batch requests efficiently. It is the default choice for production APIs where you want many concurrent users.

pip install vllm

python -m vllm.entrypoints.openai.api_server \
  --model /workspace/models/llama-3.1-8b-awq \
  --quantization awq \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --port 8000

Step 4: test the endpoint

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/workspace/models/llama-3.1-8b-awq",
    "messages": [{"role": "user", "content": "Explain quantization in one sentence."}],
    "max_tokens": 256,
    "temperature": 0.7
  }'

Step 5: expose to the internet

Use RunPod's public HTTP proxy, or run a reverse proxy with HTTPS and an API key. Do not expose vLLM directly to the public internet without authentication.

Cost estimation: what does 1,000 requests cost?

Assume an average prompt of 500 tokens and 300 output tokens, running on an RTX 3090 at $0.44/hr. vLLM at high GPU utilization processes roughly 1,200 tokens/second for an 8B model. A single request takes about 0.67 seconds. One thousand requests take roughly 11 minutes of GPU time.

GPU cost: 11 minutes × $0.44/hr = $0.08. Add 10–20% overhead for idle warmup and network. Total: roughly $0.10 per 1,000 mixed-length requests at 8B scale. At 70B FP16 on an A100 80GB ($1.99/hr), the same volume costs about $0.45 per 1,000 requests because generation is slower and the GPU is more expensive.

When to use serverless instead of a persistent pod

If your LLM API has downtime between bursts, RunPod Serverless is cheaper. If you need low-latency warm workers, persistent pods are better. A common pattern is to prototype on a pod, then switch to Serverless once the endpoint shape is stable.

Multi-GPU deployment

For models larger than a single GPU, use tensor parallelism in vLLM:

python -m vllm.entrypoints.openai.api_server \
  --model /workspace/models/llama-3.1-70b-awq \
  --tensor-parallel-size 2 \
  --quantization awq \
  --port 8000

Ensure the pod has 2+ GPUs visible. RunPod supports multi-GPU templates. Confirm NVLink or PCIe bandwidth expectations before choosing a template.

Production checklist

• Pin your model version with a commit hash or explicit revision.
• Log every request with token counts, latency, and error codes.
• Set a max GPU runtime and alert if a pod stays alive unexpectedly.
• Implement rate limiting and user authentication before exposing to customers.
• Monitor VRAM usage; out-of-memory kills are the most common failure mode.
• Test your model with adversarial prompts before public launch.

Related guides

• RunPod Serverless tutorial
• Best GPU for Stable Diffusion
• RunPod vs Vast.ai
• RunPod review

FAQs

Do I need to quantize my LLM?

Not if you have enough VRAM. FP16 is the most accurate. Quantize when you need to fit a model onto a cheaper GPU or increase batch size.

Which serving framework is fastest?

vLLM generally has the highest throughput for batched requests. TGI is simpler to configure. llama.cpp is best for single-user, low-latency scenarios on consumer hardware.

Can I run OpenAI-compatible APIs on RunPod?

Yes. vLLM and TGI both expose an OpenAI-compatible chat completions endpoint. You can drop them into existing apps with a base-URL swap.

Is RunPod cheaper than AWS for LLM inference?

Usually yes for spot-like workloads. AWS g5 instances have better uptime SLAs but cost 2–3× more per GPU hour. Compare total cost including data transfer and storage.

How do I avoid unexpected bills?

Set daily spend limits in the RunPod dashboard. Stop pods you are not using. Use Serverless for intermittent traffic so you do not pay for idle GPU time.