vLLM generating repeated/duplicate responses

[saraswatmks]

I am trying to serve a LLAMA 70B fine tuned model using vLLM on A100 80GB GPU. The model is fine tuned for chat assistant use case. I notice after 10-15 message conversation, the LLM response start to repeat itself. Below is an example:

User: Hello
Assistant: How are you ?
.
// assume 10 - 15 messages in between
.
.
User: How bad is world economy?
Assistant: World economy is really bad these days due to war.

User: What about US economy?
Assistant: World economy is really bad these days due to war.

User: Why are some people good and bad?
Assistant: World economy is really bad these days due to war.

User: Why do men earn higher than women?
Assistant: World economy is really bad these days due to war.

Currently, I want the server to handle 10 concurrent requests, each having a fixed prompt with 1350 token length. Therefore, I set the following arguments as
--max-model-len 4096, --max-seq-len 10, --max-num-batched-tokens 40960

• Besides, I have tried adding and removing bitsandbytes quantization, I still get the repeated response.
• I am running the vllm server using the docker image: vllm/vllm-openai:latest
• I have tried enabling and disabling the --enable-prefix-caching as well but still no effect
• I have tried different values for repetition_penalty, frequency_penalty, presence_penalty but still not effect.
• Besides llama 70B, I also tried with llama 8B, mistral 22B and get the same repetition in response.

Note: I don't get this problem when I deploy the model using llama.cpp, everything seems to work fine.

How can I fix this repetition issue? Looking for some guidance here.

[setianke]

Facing the same issue - tried the same change or parameters removing --enable-prefix-caching, currently using Mistral Small 3.2 2506 and Qwen 3 32B.

[ashgold]

This is a really serious issue.

Have you tried turning off APC using the following option?
--no-enable-prefix-caching

[helloninglei]

I tried run with disable prefix-caching but doesn't work, however it works for V0

[Mrjaggu]

Any update on how to avoid this issue?

[trevorstr]

I have this same problem with Qwen 0.6B, the default vLLM model. I am serving it with vLLM on an NVIDIA GeForce RTX 3060 12 GB with Docker Compose.

Here is the PowerShell script I use to generate completions:

$Result = Invoke-RestMethod -Uri http://localhost:10200/v1/completions -ContentType application/json -Body (@{ model = 'Qwen/Qwen3-0.6B'; prompt = '/nothink Generate a list of only 10 male first names, and return only the names.'; max_tokens = 2500; chat_template_kwargs = @{ enable_thinking = $false } } | ConvertTo-Json) -Method post

And the output looks like this (I inserted <REPEAT> to represent repetition of the list of names):

 No other info. Please.
Okay, the user wants a list of only 10 male first names. Let me start by recalling some common male names. First, I need to make sure they are correct and not too common. Let me think... "James" comes to mind. Then "Robert" is another. "David" and "Andrew
" are also valid. "Michael" and "Joseph" are good choices. Maybe "Mark" and "Peter" are next. Let me check if there are any other names that might be more appropriate. "Ryan" and "Thomas" could work. Wait, "Michael" is a common first name. Let me count: J
ames, Robert, David, Andrew, Michael, Joseph, Mark, Peter, Ryan, Thomas. That's 10. I need to make sure there are no duplicates and that they are all male names. Yes, that's correct. I should present them in a list with only the names, no other details.
**list** = ["James", "Robert", "David", "Andrew", "Michael", "Joseph", "Mark", "Peter", "Ryan", "Thomas"]


**list** = ["James", "Robert", "David", "Andrew", "Michael", "Joseph", "Mark", "Peter", "Ryan", "Thomas"]


**list** = ["James", "Robert", "David", "Andrew", "Michael", "Joseph", "Mark", "Peter", "Ryan", "Thomas"]


**list** = ["James", "Robert", "David", "Andrew", "Michael", "Joseph", "Mark", "Peter", "Ryan", "Thomas"]

<REPEAT>
<REPEAT>
<REPEAT>
<REPEAT>
<REPEAT>

[xiaomayi-ant]

Have you found a good solution?

[dc740]

After a lot of debugging I checked the params file in hugging face and discovered I forgot to set the stop parameter ( https://docs.vllm.ai/en/v0.8.4/api/inference_params.html ) and this is what caused the repetition in future replies. It worked the same as llama.cpp once I set it.
my current argument list as an example:

vllm serve --dtype float16 --max-model-len 38928 --port 9000 /models/Qwen3Coder --enable-expert-parallel --enable-auto-tool-choice --tool-call-parser qwen3_coder --served-model-name Qwen/qwen-coder-30b --generation-config vllm --override-generation-config '{"repetition_penalty": 1.05, "top_p":0.80, "top_k":20, "min_p":0.0, "temperature":0.7, "stop": ["<|im_start|>", "<|im_end|>"]}'

I hope this helps unblock you.
UPDATE:
it worked for a while, and now I'm getting the same old responses again

[beingPro007]

Same issue i tried every possible params still the same issue my params are

         sampling_params = SamplingParams(
            temperature=0.01,          
            repetition_penalty=1.15,  
            top_p=0.95,               
            max_tokens=max_tokens,
            frequency_penalty=1.0,
            presence_penalty=1.0,
            stop=["]"],
            
        )

i am sending max_tokens as 32000