huggingface/lighteval

Pull Requests Commits

removing the assistant prefix to be on the safe side

clefourrier committed 178 days ago

567ca2cf

remove the prefill

clefourrier committed 178 days ago

f42791de

more extraction fixing

clefourrier committed 178 days ago

4449200b

adding assistant prefix option

clefourrier committed 178 days ago

92950e17

reverting prompt

clefourrier committed 178 days ago

5e130659

updated regexes

clefourrier committed 178 days ago

67f36234

adding mmlu pro

clefourrier committed 179 days ago

1fcd1444

Fix system prompt concatenated with instruction in chat template (#830)

clefourrier committed 181 days ago

Verified 0ba8812f

add TUMLU-mini benchmark, solves #577 (#811)

ceferisbarov committed 181 days ago

Verified 936612cb

Add TranslationLiterals for Language.DANISH (#770)

spyysalo committed 181 days ago

Verified b0c092a1

Fix Typos in Documentation and Task Description (#810)

leopardracer committed 181 days ago

Verified a3725503

Catch ROCM/HIP/AMD oom in `should_reduce_batch_size` (#812)

mcleish7 committed 182 days ago

Verified 95ec6691

Flores-200 combinations->permutations (#820)

anton-l committed 182 days ago

Verified dd1af5a9

set default temperature to 0 in generation config (#814)

NathanHB committed 185 days ago

Verified 327071fe

Add org_to_bill parameter to documentation (#781)

tfrere committed 188 days ago

Verified 3d448976

Add Bulgarian and Macedonian literals (#769)

dianaonutu committed 199 days ago

Verified 9771f3a0

Fix: Assign SummaCZS instance to `self.summac` in Faithfulness metric (#795)

sahilds1 committed 199 days ago

Verified 672f6940

[IFEval] Speed up think tag removal (#792)

lewtun committed 201 days ago

Verified 53df8598

add a regex to remove think tags before evaluating ifeval (#791)

NathanHB committed 201 days ago

Verified 8260f590

fix: multiple typos of different value (#782)

DeVikingMark committed 207 days ago

Verified 78871727

Making bootstrap_iters an arg (#697)

pratyushmaini committed 207 days ago

Verified 88050354

Adds GSM-PLUS (#780)

NathanHB committed 208 days ago

Verified 9619194b

Bump dev version to 0.10.1.dev0 (#777)

NathanHB committed 213 days ago

Verified 9dc2e538

Async vllm (#693)

clefourrier committed 214 days ago

Verified c4826ea9

Bump ruff version (#774)

NathanHB committed 214 days ago

Verified c9c19e12

Nanotron, Multilingual tasks update + misc (#756)

hynky1999 committed 214 days ago

Verified 034c23b7

add dependencies to run after pip install (#767)

NathanHB committed 215 days ago

Verified 26517504

fix custom model example (#766)

NathanHB committed 215 days ago

Verified cce0bfca

Adds template for custom path saving results (#755)

NathanHB committed 215 days ago

Verified b6816a87

Allow for model kwargs when loading transformers from pretrained (#754)

NathanHB committed 215 days ago

Verified ce1dbb5e

Older