Generation speed suddenly became very slow, what might be the cause?

[zhuwenfei-wintech]

The models I deployed suddenly became very slow without any indication, generation throughput around 1 token/s.
Here is the detail:
I deployed Qwen2-72B-Instruct-GPTQ-Int4 with vLLM v0.5.1. The deployment was done via docker compose and official docker image v0.5.1.
I deployed the same model with the same settings on two machines. One machine for production with 4x4090 GPU, 256G memory, Xeon 8352V 36 cores. One machine for development with 2xH100 GPU (only one was used for the model), 256G memory, AMD EPYC 9354 32-Core.
Both models on these two machines have been running for over one month without any problem but suddenly they both (not at the same time, but within 24 hours) became very slow. No error log, vRAM was OK, GPU utilization was OK, cpu, memory, everything was OK. I couldn't find a cause.
Here is part of the log, you can see the generation throughput suddenly decrease.

After a clear restart of the containers, the problem seems gone so far.
Any idea what might be the cause?

[jazzisfuture]

I encountered the same problem. At first I thought it was a swap problem, so I set swap-space=0, but I still encountered this problem. This problem occurs when serving online and reasoning about large amounts of data offline.

[MLKoz2]

I encounter this problem as well

[alesalloum]

I've noticed that the whole "warming up" (before the llm generation) part of the offline inference has become way slower compared to 0.8.x version.

[akumaburn]

Different model, but possibly the same issue.

I think its cache thrashing, in the current release build.

Clanker analysis:

1. 0.29.0 keeps every checkpoint for your setup. site-packages/vllm/engine/arg_utils.py:2393 forces prefix_cache_retention_interval=None for any hybrid model with an EAGLE-family drafter, and DFlash counts as one. That means dense checkpointing: every 1648-token boundary's GDN state and drafter sliding-window block gets cached.
2. Your model uses about 15 KV cache groups. The drafter's 5 sliding-window layers set the group size to 5, giving 4 full-attention groups, 10 GDN groups and 1 drafter group.
3. Each prefill chunk churns ~11 cached blocks through the pool. Every chunk allocates a new block in each of those 11 non-attention groups and frees an older one. Because the freed block is cached, BlockPool.free_blocks puts it at the most-recently-used end of the free queue (vllm/v1/core/block_pool.py:768).
4. One long prefill sweeps the whole pool. By my estimate a 236k prefill makes about 2,100 allocations against a 1,852-block pool. Queued conversations' prefixes are the least recently used, so they are evicted first, and when admitted they recompute everything.

How main fixes it

• prefix_cache_retention_interval defaults to 0, and the forced-dense override doesn't exist.
• [Kimi K3] Support internal prefix checkpoints with partial prefix caching and spec-decoding #53614 aligns the GDN checkpoints to the boundary the DFlash drafter's cache hits actually land on, so retention 0 still gets prefix hits.
• The checkpoints that aren't kept are never cached. They go to the reuse-first end of the free queue, so a prefill reuses its own scratch blocks instead of sweeping the pool (reachable_block_mask, single_type_kv_cache_manager.py:1565).

[poojithdevan4D]

FP8 KV cache gave +40–52% throughput — but ONLY when vLLM was already preempting. Raw results + the quality gate that failed.

I benchmarked kv_cache_dtype=fp8 vs bf16 on Qwen2.5 3B/7B/14B + Mistral-7B across A10/A100, single-variable, on rented GPUs. Result is sharper than the "fp8 is free throughput" folklore:

KV cache full + server preempting → fp8 = +40–52% throughput (3B +52%, 7B +40%, 14B +43%, Mistral-7B +42%).
Compute-bound, no preemption → +1.7% (nothing).
GPU utilization was ~100% in both regimes — so "is the GPU busy?" is the wrong signal. The discriminator is preemptions, not GPU%.

Before anyone turns this into folklore: it's a heuristic, not a law. Small, partly post-hoc matrix, mostly single runs, synthetic traffic, and every positive case was already overloaded (TTFT 20–94s) — so this is a goodput-ceiling lever, not a free low-load speedup. Preemption event count is also too crude (in the 14B/Mistral wins it didn't even drop).

Quality: mixed and honest. 0/16 factual-QA regressions and needle-in-haystack 5/5 @14k — but my teacher-forced KL preflight failed (p99 0.39). So: task-safe on small tests, not "lossless."

Everything's open, including the raw result JSON and the external critique that reshaped the project: https://github.com/poojithdevan4D/InferPilot

The most useful reply would be a counterexample: a preempting workload where fp8 doesn't help, or an fp8 win without preemption.

Also — I'm turning this into a local "acceptance test for vLLM config changes" (replay a sanitized trace → SLO-capacity curve → quality gate → rollback). Looking for ~5 teams running self-hosted vLLM to try it on a sanitized trace (token counts + timings, no prompts/text; can run in your VPC). Free; the answer might be "keep your current config."