[fix] Fix GlmOcr integration tests: wrong token IDs and image token decode bug (#48650)
* [fix] Fix GlmOcr integration tests: wrong token IDs and image token decode bug
Two distinct issues were causing all 6 integration tests to fail from day one:
1. Wrong expected input_ids: the test was written with Qwen2.5-style token IDs
(151K range) but zai-org/GLM-OCR uses a GLM-4 tokenizer (59K vocab).
Updated to the actual IDs produced by the processor.
2. Image tokens leaking into decoded output: `batch_decode(output, ...)` decodes
the full sequence including the input prompt, which contains hundreds of
`<|image|>` tokens (ID 59280). These tokens are in `all_special_tokens` but
their `added_tokens_decoder` entry has `special=False` in the Rust backend,
so `skip_special_tokens=True` does not filter them. Fixed by decoding only
the newly generated tokens: `output[:, inputs.input_ids.shape[1]:]`.
All 6 tests verified passing on a single-GPU runner.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [style] Fix slice spacing in GlmOcr integration tests (ruff)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>