Pull Requests huggingface/lighteval

Fix CorpusLevelF1Score(None) to report positive-class F1, not max-per-class

#1275 opened 2026-06-25 20:05 by iamsharduld

Fix MajAtN IndexError when the gold is not the first choice

#1274 opened 2026-06-25 20:03 by iamsharduld

Fix helm_normalizer mis-scoring numbers (homogeneize before remove_punc)

#1273 opened 2026-06-25 20:01 by iamsharduld

Add script-aware language tags for FLORES 200

#1272 opened 2026-06-23 14:44 by goktugozkanmd

Fix sample-cache corruption under accelerate data-parallel

#1271 opened 2026-06-22 14:58 by shipbehaves

fix(mmlu_pro): correct answer letters in prompt and choices

#1269 opened 2026-06-19 19:31 by vineethsaivs

feat: add static linter command for custom tasks

#1268 opened 2026-06-19 18:40 by Kesav2k04

feat: add ID-based few-shot example selection (#634)

#1263 opened 2026-06-16 06:15 by sohammmmm10

Fix KeyError: 'default' for bigbench tasks with nonexistent split

#1262 opened 2026-06-14 18:56 by nuthalapativarun

Document unconditioned_query requirement for PMI normalization

#1261 opened 2026-06-14 18:53 by nuthalapativarun

Fix non-deterministic cache hash for MetricGrouping tasks

#1260 opened 2026-06-14 18:50 by nuthalapativarun

Sync LightevalTaskConfig docstring with current fields

#1259 opened 2026-06-14 18:44 by nuthalapativarun

Fix broken default_tasks.py link in quicktour docs

#1258 opened 2026-06-14 18:42 by nuthalapativarun

Fix LCB grader failing solutions that read sys.stdin.buffer

#1257 opened 2026-06-13 09:37 by vineethsaivs

fix(lcb): expose sys.stdin.buffer so binary-stdin solutions are graded correctly

#1256 opened 2026-06-13 09:22 by avalyset

Add AIME26 task

#1254 opened 2026-06-11 15:01 by Vedant-Agarwal

Enhance model loading and error handling

#1251 opened 2026-06-03 18:54 by nandeanie

docs: fix reference links

#1250 opened 2026-05-31 11:51 by charles-openclaw

Add optional juryeval integration for LLM-as-Judge metrics

#1249 opened 2026-05-31 11:11 by py-ai-dev

Fix flatten_dict crash/wrong key for bare numpy array values

#1247 opened 2026-05-28 21:37 by Kymi808

chore: enable Dependabot weekly GitHub Actions bumps dependabot

#1246 opened 2026-05-26 10:32 by hf-dependantbot-rollout[bot]

Add ArxivRollBench tasks

#1245 opened 2026-05-24 05:34 by liangzid

feat: implement loglikelihood and loglikelihood_rolling for LiteLLMClient (closes #1093)

#1244 opened 2026-05-21 19:47 by ALI-AL-MARJANI

Fix callable type hint in parallelism helper

#1239 opened 2026-05-20 04:20 by GoparapukethaN

fix: guard choices[0] and message=None before content access in llm_as_judge

#1238 opened 2026-05-17 23:34 by qizwiz

docs: fix custom model examples

#1237 opened 2026-05-15 15:20 by MukundaKatta

fix: prevent IndexError in Doc.get_golds() for out-of-bounds gold_index

#1236 opened 2026-05-13 02:41 by AmSach

fix typo

#1235 opened 2026-05-08 11:16 by fpetrakov

fix: transpose references before passing to sacrebleu in CorpusLevelTranslationMetric

#1234 opened 2026-05-08 01:04 by jaydenC88

test: style-bot trigger

#1221 opened 2026-05-04 12:51 by paulinebm