Pull Requests huggingface/lighteval

POLLUX LLM-Judge metric

#1210 opened 2026-04-10 13:18 by ulyanaisaeva

catch task has no docs instead of throw

#1207 opened 2026-04-08 12:35 by BuiHoangTu

add multilingual flag to vllm

#1206 opened 2026-04-08 12:30 by BuiHoangTu

fix(vllm): Enhance VLLMModel context size handling for batch inputs

#1205 opened 2026-04-06 19:52 by paulovsantanas

examples: add RAIL Score responsible AI custom task template

#1203 opened 2026-04-02 09:47 by SumitVermakgp

Add --load-tasks-multilingual and fix --custom-tasks for inspect backend

#1199 opened 2026-03-25 12:32 by dzautner

[Bugfix] Check all responses when n>1 instead of only the first one

#1197 opened 2026-03-23 11:07 by eldarkurtic

[Litellm Enhancement] Enable extra sampling args for litellm backend

#1195 opened 2026-03-20 12:34 by eldarkurtic

[Bugfix] presence_penalty is silently dropped from sampling args in litellm backend

#1193 opened 2026-03-18 11:58 by eldarkurtic

[Bugfix] litellm backend should iterate over docs in a split not entire dataset

#1192 opened 2026-03-18 11:45 by eldarkurtic

Remove deprecated prompt_token_ids wrapping in vLLM backend

#1191 opened 2026-03-18 07:50 by sihyeonn

Fix litellm connection pool limiting concurrent_requests

#1190 opened 2026-03-18 07:34 by sihyeonn

feat(utils): show count of evaluated samples in Markdown summary table

#1188 opened 2026-03-13 06:03 by anzzyspeaksgit

Fix typos in math_comparison.py and sample_comparison.py

#1186 opened 2026-03-12 12:25 by joshuaswanson

squad_v2: include unanswerable questions in evaluation

#1185 opened 2026-03-09 11:01 by Matteovanypersele

Update vllm version requirement to 0.17.0

#1183 opened 2026-03-09 10:43 by NathanHB

Fail fast on non-retriable LiteLLM status codes

#1182 opened 2026-03-08 03:58 by yangbaechu

Redact model config credentials in saved and returned results

#1181 opened 2026-03-08 03:03 by yangbaechu

fix(normalizations): guard against index out of range in LogProbToken…

#1180 opened 2026-03-06 11:25 by inakiLakunza

Korean completed and Basque fixed

#1179 opened 2026-03-06 10:46 by inakiLakunza

[LiteLLM] Add cross-provider reasoning_effort support (first step) + token budget fixes

#1178 opened 2026-03-05 15:42 by dyurchenko98

Fix LiteLLM split iteration in greedy_until to avoid duplicate API requests

#1177 opened 2026-03-05 14:56 by dyurchenko98

Fix corpus reference orientation for chrF/chrF++/TER metrics

#1176 opened 2026-03-05 14:40 by dyurchenko98

FIX : handle empty choices in Doc.get_golds() to prevent IndexError

#1174 opened 2026-02-23 21:20 by nandeanie

Fix: pass through custom_tasks and enable multilingual in eval command

#1172 opened 2026-02-19 07:58 by dzautner

Fix IndexError in LogProbTokenNorm when choices_tokens is shorter than choices_logprob

#1171 opened 2026-02-18 16:33 by worksbyfriday

Add jfinqa: Japanese Financial Numerical Reasoning QA

#1169 opened 2026-02-17 17:59 by ajtgjmdjp

fix: restore task list display logic

#1166 opened 2026-02-10 08:49 by s1eeping-king

fix: Transformers Model no template cast stop_sequences to list

#1165 opened 2026-02-07 17:52 by mrsndmn

Fix TypeError in aa_omniscience_prompt

#1161 opened 2026-01-22 19:41 by pjavanrood