lighteval
034c23b7 - Nanotron, Multilingual tasks update + misc (#756)

Commit
210 days ago
Nanotron, Multilingual tasks update + misc (#756) ## Nanotron - Logprobs working - Generation works with https://github.com/huggingface/nanotron/tree/nouamane/lighteval-fix, however sometimes there is issue with ordering. - Only relevant part of config are fetched to prevent some dataclass checks which are not needed in nanotron - Smol changes to configs ## Metrics - Probability metrics now works with Char normalization - Token normalization is fixed when used with transformers ## Multlilingual tasks - New squads + few new mcf benchmarks + smol fixes to existing tasks ## Misc - Qa template now only uses unique golds, which speed-upds probs calculation. --------- Co-authored-by: anton <anton@huggingface.co> Co-authored-by: nouamanetazi <nouamane98@gmail.com> Co-authored-by: Hynek Kydlicek <kydlicek.hynek@huggingface.co>
Author
Parents
Loading