transformers
4815a0a6 - [fix] Fix GlmOcr integration tests: wrong token IDs and image token decode bug (#48650)

Commit
72 days ago
[fix] Fix GlmOcr integration tests: wrong token IDs and image token decode bug (#48650) * [fix] Fix GlmOcr integration tests: wrong token IDs and image token decode bug Two distinct issues were causing all 6 integration tests to fail from day one: 1. Wrong expected input_ids: the test was written with Qwen2.5-style token IDs (151K range) but zai-org/GLM-OCR uses a GLM-4 tokenizer (59K vocab). Updated to the actual IDs produced by the processor. 2. Image tokens leaking into decoded output: `batch_decode(output, ...)` decodes the full sequence including the input prompt, which contains hundreds of `<|image|>` tokens (ID 59280). These tokens are in `all_special_tokens` but their `added_tokens_decoder` entry has `special=False` in the Rust backend, so `skip_special_tokens=True` does not filter them. Fixed by decoding only the newly generated tokens: `output[:, inputs.input_ids.shape[1]:]`. All 6 tests verified passing on a single-GPU runner. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [style] Fix slice spacing in GlmOcr integration tests (ruff) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Author
Parents
Loading