Different from the output of the HF inference

[xcxhy]

Looking forward to your reply! I set the temperature=0.1, top-k=10, top-p=0.75，I think i infer the same prompt, I will get the same output, through test the hf and vllm inference, HF will get stable output then vllm sometimes get the different output. Is this normal? The parameters of the two inferences are the same and do not bring the same constraints.

[marscc]

I have encountered the same problem. use llama 13B model, set max_tokens=256, frequency_penalty=0.1, temperature=0.1, top-k=50, top-p=0.75，I tested on a set of 40 questions and found that the outputs for 15 questions were different from the outputs obtained using huggingface inference.

[zhuohan123]

The LLM inference process includes sampling, which is a random process. Because the implementation of HF and vLLM are different, it is normal to get different samples. However, if you perform argmax sampling (e.g., temperature=0), then you should be able to see the same results.

[tonystz]

even when I set temperature=0, the result is still not the same. Anyone has some suggestions?

[kaishxu]

same problem. I set temperature=0, and the result is still not the same.

[damandeep-hyprbots]

Same issue.

[GhostofAdam]

I am using greedy search in decoding.
I find when the prompt lenght is less than 7(while 8 as block size), vllm and hf get different output texts, but they are same with longer prompts. I believe a bug is hidden behind this.

[AmazeQiu]

I've meet the same problem.
I use hf with this config:
temperature=0, top_p=1.0 ,top_k=40 ,num_beams=4, max_new_tokens=64
then use vllm with this config :
max_tokens = 64, use_beam_search = True, best_of = 4, temperature = 0 ,top_p = 1.0, top_k = -1
The speed is fast but the accuracy is pretty lower than that using hf. T _ T
I even change the source code to use "model.half().cuda()" to make it same with the way I use hf in my code.

[BaiMoHan]

Has this issue been resolved?

[Jerrrrykun]

I have encountered the same issue for temperature=0. Interestingly, some of outputs are same with HF inference but some are not, leading to decreases in performance.