Pull Requests huggingface/lighteval

feat: implement loglikelihood and loglikelihood_rolling for LiteLLMClient (closes #1093)

#1244 opened 2026-05-21 19:47 by ALI-AL-MARJANI

Add LEXam legal exam benchmark to Swiss legal evals

#1243 opened 2026-05-21 14:36 by JoelNiklaus

Add Pillow dependency

#1242 opened 2026-05-21 13:00 by JoelNiklaus

Fix callable type hint in parallelism helper

#1239 opened 2026-05-20 04:20 by GoparapukethaN

fix: guard choices[0] and message=None before content access in llm_as_judge

#1238 opened 2026-05-17 23:34 by qizwiz

docs: fix custom model examples

#1237 opened 2026-05-15 15:20 by MukundaKatta

fix: prevent IndexError in Doc.get_golds() for out-of-bounds gold_index

#1236 opened 2026-05-13 02:41 by AmSach

fix typo

#1235 opened 2026-05-08 11:16 by fpetrakov

fix: transpose references before passing to sacrebleu in CorpusLevelTranslationMetric

#1234 opened 2026-05-08 01:04 by jaydenC88

Popotest patch 1

#1231 opened 2026-05-05 15:11 by popotest

test: style-bot trigger

#1221 opened 2026-05-04 12:51 by paulinebm

Add Bayes@N metric

#1219 opened 2026-04-29 00:05 by mohsenhariri

Log per-sample details as trackio.Trace in push_to_wandb

#1217 opened 2026-04-27 17:26 by abidlabs

Add LICA-Bench: graphic design VLM evaluation (39 tasks, 7 domains)

#1212 opened 2026-04-15 01:24 by purvanshi

POLLUX LLM-Judge metric

#1210 opened 2026-04-10 13:18 by ulyanaisaeva

catch task has no docs instead of throw

#1207 opened 2026-04-08 12:35 by BuiHoangTu

add multilingual flag to vllm

#1206 opened 2026-04-08 12:30 by BuiHoangTu

fix(vllm): Enhance VLLMModel context size handling for batch inputs

#1205 opened 2026-04-06 19:52 by paulovsantanas

examples: add RAIL Score responsible AI custom task template

#1203 opened 2026-04-02 09:47 by SumitVermakgp

Add --load-tasks-multilingual and fix --custom-tasks for inspect backend

#1199 opened 2026-03-25 12:32 by dzautner

[Bugfix] Check all responses when n>1 instead of only the first one

#1197 opened 2026-03-23 11:07 by eldarkurtic

[Litellm Enhancement] Enable extra sampling args for litellm backend

#1195 opened 2026-03-20 12:34 by eldarkurtic

[Bugfix] presence_penalty is silently dropped from sampling args in litellm backend

#1193 opened 2026-03-18 11:58 by eldarkurtic

[Bugfix] litellm backend should iterate over docs in a split not entire dataset

#1192 opened 2026-03-18 11:45 by eldarkurtic

Remove deprecated prompt_token_ids wrapping in vLLM backend

#1191 opened 2026-03-18 07:50 by sihyeonn

Fix litellm connection pool limiting concurrent_requests

#1190 opened 2026-03-18 07:34 by sihyeonn

feat(utils): show count of evaluated samples in Markdown summary table

#1188 opened 2026-03-13 06:03 by anzzyspeaksgit

Fix typos in math_comparison.py and sample_comparison.py

#1186 opened 2026-03-12 12:25 by joshuaswanson

squad_v2: include unanswerable questions in evaluation

#1185 opened 2026-03-09 11:01 by Matteovanypersele

Update vllm version requirement to 0.17.0

#1183 opened 2026-03-09 10:43 by NathanHB