microsoft

/

phi-4

like 2.28k

Follow

Microsoft 20.8k

Summary

Set generation_config.json eos_token_id back to [100257, 100265], restoring
<|endoftext|> (100257) alongside <|im_end|> (100265) as a stop token. This reverts
only the eos_token_id change from #21 while keeping that PR's other improvements
(chat template fix, pad_token -> <|dummy_85|>).

Problem

With eos_token_id set to 100265 only, the model can fail to stop: it produces a
correct answer and then continues emitting unrelated text until max_tokens is reached
(reported as "random tokens until the max token limit").

Root cause

phi-4 ends many (typically terse) assistant turns with <|endoftext|> (100257) rather
than <|im_end|> (100265). Since #21, generation_config.json only lists 100265, so
when the model emits 100257 no stop criterion fires; decoding runs past the turn
boundary (the following <|im_start|>user<|im_sep|> gets stripped on output, leaving a
stray user) and free-runs until the token cap.

Evidence

• Reproduced with bare HF model.generate() at temperature 0 (greedy). Identical on
  transformers 4.47 and 5.13, so it is not a transformers-version issue.
• Reproduced with vLLM 0.18.0 and 0.24.0 — byte-identical output to HF.
• Next-token distribution right after "The capital of France is Paris.":
  P(<|endoftext|>=100257) = 0.72 vs P(<|im_end|>=100265) = 0.27, so greedy decoding
  deterministically selects the token the current config does not stop on.
• With eos_token_id = [100257, 100265], generation stops cleanly on all three engines.

Scope

Only generation_config.json is changed. config.json and the tokenizer files are left
as-is; this is the minimal change that both transformers and vLLM honor for stopping.