vLLM vs llama.cpp on a rented RTX 5090

Task

Serve Qwen3.8-27B at 4-bit on vLLM and on llama.cpp on the same rented RTX 5090. Per engine: billed seconds from launch to ready and to first token, stable tokens/s at one request; then raise concurrent requests step by step and record requests/hour, total output tokens/s and median time-to-first-token; mark the load where latency becomes unacceptable (our threshold: median TTFT > 2 s).

Stand

r01 — baseline — llama.cpp, single stream

beat baseline · status done

Checks

setup

llama-server b11209, -m Qwen3.8-27B-Q4_K_M.gguf -ngl 99 -c 32768 -np 1 --port 8001

launch

T0=$(date +%s); ./llama-server ... & until curl -s localhost:8001/health | grep -q '"ok"'; do sleep 1; done; echo $(( $(date +%s)-T0 ))

command

vllm bench serve --backend openai-chat --base-url http://localhost:8001 --endpoint /v1/chat/completions --model q --tokenizer /workspace/models/awq --dataset-name random --random-input-len 1024 --random-output-len 256 --num-prompts 20 --max-concurrency 1 2>&1 | tee llamacpp_c1.log

Results

Files

• llamacpp_c1.log 5.8 KiB
• llamacpp_c1_server.log 20.6 KiB

Notes

vast base image had no vLLM; pip install failed building xformers, installed via `uv pip install vllm --torch-backend=auto`; llama.cpp taken as prebuilt release binary, no compile

r02 — baseline — vLLM, single stream

beat baseline · status done

Checks

setup

vllm serve /workspace/models/awq --served-model-name q --max-model-len 32768 --limit-mm-per-prompt '{"image":0,"video":0}' --max-num-seqs 64 --attention-backend FLASH_ATTN --port 8000; env VLLM_USE_FLASHINFER_SAMPLER=0

launch

T0=$(date +%s); vllm serve ... & for i in $(seq 1 900); do curl -s localhost:8000/health && break; sleep 1; done; echo $(( $(date +%s)-T0 ))

command

vllm bench serve --backend openai-chat --base-url http://localhost:8000 --endpoint /v1/chat/completions --model q --tokenizer /workspace/models/awq --dataset-name random --random-input-len 1024 --random-output-len 256 --num-prompts 20 --max-concurrency 1 2>&1 | tee vllm_c1.log

Results

Files

• vllm_c1.log 5.7 KiB
• vllm_server.log 82.0 KiB

Notes

three failed starts before this one, not counted in load_s: (1) OOM at profiling — multimodal model reserves memory for 255 images per prompt, fixed with --limit-mm-per-prompt; (2) FlashInfer wheel does not recognise Blackwell SM 12.0, fixed with --attention-backend FLASH_ATTN and sampler off; (3) torchaudio CUDA mismatch after reinstall, torchaudio uninstalled

r03 — subjects — llama.cpp, concurrency ladder

beat subjects · status done

Checks

setup

llama-server -ngl 99, -c 65536 -np 8 for steps 4 and 8; -c 131072 -np 16 for step 16 (context is split across slots)

command

for C in 4 8 16; do vllm bench serve --backend openai-chat --base-url http://localhost:8001 --endpoint /v1/chat/completions --model q --tokenizer /workspace/models/awq --dataset-name random --random-input-len 1024 --random-output-len 256 --num-prompts 100 --max-concurrency $C 2>&1 | tee llamacpp_c$C.log; done

ladder

Files

• llamacpp_c4.log 6.1 KiB
• llamacpp_c8.log 6.8 KiB
• llamacpp_c16.log 8.2 KiB
• llamacpp_np8_server.log 198.5 KiB
• llamacpp_np16_server.log 114.2 KiB

Notes

before step 16 the server was restarted with `-c 131072 -np 16`

r04 — subjects — vLLM, concurrency ladder

beat subjects · status done

Checks

setup

same vllm serve as r02

command

for C in 4 8 16 32 64; do vllm bench serve --backend openai-chat --base-url http://localhost:8000 --endpoint /v1/chat/completions --model q --tokenizer /workspace/models/awq --dataset-name random --random-input-len 1024 --random-output-len 256 --num-prompts 100 --max-concurrency $C 2>&1 | tee vllm_c$C.log; done

ladder

Files

• vllm_c4.log 6.6 KiB
• vllm_c8.log 8.1 KiB
• vllm_c16.log 7.7 KiB
• vllm_c32.log 7.6 KiB
• vllm_c64.log 7.6 KiB
• vllm_server.log 82.0 KiB

Notes

none

Coverage