huggingface/lighteval

Pull Requests Commits

Merge branch 'main' into clem_fix_bs2

clefourrier committed 24 days ago

Verified 486ac9d5

removed duplicate code, useless function, added stronger deletion of items, plus updated the logic in generation size to respect what the user asks

clefourrier committed 24 days ago

937f09af

Fix task config metric typing to accept Metric enums (#1018)

emmanuel-ferdman committed 24 days ago

Verified 99162f17

Adding AA Omniscience task (#1066)

NathanHB committed 26 days ago

Verified 5425c337

adding satrred tag for frontend

NathanHB committed 26 days ago

d9ea404f

One file one task definition (#1059)

NathanHB committed 27 days ago

Verified babeec92

use correct hf subset for ifbench multiturn (#1061)

sam-paech-liquid committed 27 days ago

Verified ad58fedd

Update LiteLLM configuration for hosted_vllm provider (#1060)

abhiram1809 committed 30 days ago

Verified af6b5b4f

Fix: correct argument order in MajAtN.compute (#1058)

WeiKangda committed 30 days ago

Verified 391d5b40

Fix set using wrong syntax (#1057)

f14-bertolotti committed 31 days ago

Verified cb97d5c8

add a task dump in registry for better documentation of tasks (#1052)

NathanHB committed 32 days ago

Verified b8ccd20c

Remove suites in task configs example and fix task with hf_filters (#1051)

NathanHB committed 32 days ago

Verified 35babcb1

wrong attribute self.k -> self.n (#1049)

f14-bertolotti committed 32 days ago

Verified 6524c6ad

not forcing use_logits at True (#1050)

f14-bertolotti committed 32 days ago

Verified 2247df76

neater bundle and logdir (#1043)

NathanHB committed 32 days ago

Verified cd91dde7

remove suites from serbian_eval.py (#1044)

S-Y-A-N committed 32 days ago

Verified d04e4f9b

put lower bound on typer to use literal type (#1042)

NathanHB committed 38 days ago

Verified 566a7be3

remove suites and make fewshot optional (#1038)

NathanHB committed 39 days ago

Verified 31433cc9

run all hf providers with `:all` (#1039)

NathanHB committed 39 days ago

Verified 5b7ca62f

fix `lighteval task inspect` command and tiny bench task (#992)

NathanHB committed 39 days ago

Verified b5cbd91a

add openai and inspect ai lower bound (#1035)

NathanHB committed 40 days ago

Verified 5aa09c5a

Update huggingface-cli login to use newer hf auth login (#1034)

xeophon committed 40 days ago

Verified 97303acf

Fix inspect reasoning effrot (#1033)

NathanHB committed 40 days ago

Verified 17e024bc

adds mmlu-pro (#1031)

NathanHB committed 40 days ago

Verified fa4860f9

Adds inspectai (#1022)

NathanHB committed 41 days ago

Verified 880bebef

Move tasks to individual files (#1016)

NathanHB committed 46 days ago

Verified 3cd31fd1

add translation literals indic (#1015)

rpm000 committed 54 days ago

Verified bf8b5471

Fixing naming for sample evals + adding reqs in aime24 (#989)

clefourrier committed 61 days ago

Verified 161d47cc

added fallback for incomplete configs for vlm models launched as llms (#828)

clefourrier committed 61 days ago

Verified e7d885c3

Fix 999: always provide parameters in the metric name to allow using several combinations (#1017)

clefourrier committed 61 days ago

Verified 70acb852

Older