vLLM vs llama.cpp vs Ollama: What Happens When Your Model Doesn't Fit in 24GB VRAM

1Key Takeaways

• TL;DR Benchmarked llama.cpp, Ollama, and vLLM across 5 models (1B to 116.8B params) on one RTX 3090 (24GB) + 128GB RAM home-lab box, priced through HomeLab Monitor .
• Inside 24GB, vLLM's continuous batching scales aggregate throughput 3.9x-5.4x from concurrency 1 to 8 (llama.cpp only manages 1.2x-1.9x , even with -np 8 explicitly set to match).
• Past 24GB — two models deliberately chosen to force RAM-spill — llama.cpp and Ollama both degrade to single-digit tok/s and keep generating.