huggingface/lighteval

Pull Requests Commits

update tests for cachine param

clefourrier committed 156 days ago

e14490ee

Merge branch 'main' into debug_ll_evals

clefourrier committed 156 days ago

Verified ec421e08

added tests to avoid this padding break mess

clefourrier committed 156 days ago

ba2e5a89

added extra params

clefourrier committed 156 days ago

b27d110e

fix in case bs is 0

clefourrier committed 157 days ago

4092548a

padding should be better now, also better batch size management to prevent ooms

clefourrier committed 157 days ago

05168d28

Number of fixes to run accelerate evaluations (#898)

clefourrier committed 157 days ago

Verified 64f93b0e

Update transformers and vllm dependencies (#899)

clefourrier committed 157 days ago

Verified 865335e4

Added post processing (for reasoning tokens) to pipeline (#882)

clefourrier committed 158 days ago

Verified d7beacbc

fix path concatenation (#891)

clefourrier committed 158 days ago

Verified 994e9e07

Fix DP>1 & TP>1 evals with vllm (#841)

NouamaneTazi committed 161 days ago

Verified 24895519

Adds continuous batching (#850)

NathanHB committed 161 days ago

Verified 99bfd9f2

Fix typo: CONCURENT_CALLS -> CONCURRENT_CALLS (#884)

muupan committed 161 days ago

Verified 1404ba1d

Fixed typo in Python API documentation (#862)

dtung8068 committed 161 days ago

Verified 3b1126ad

Automatically infer whether to use a chat template or not instead of using kwargs (#885)

clefourrier committed 161 days ago

Verified e7d70902

Pin transforers version to fix broken test suite due to mismatch with vllm (#886)

clefourrier committed 162 days ago

Verified bc5e450a

Remove datasets 4 from pyproject (#863)

clefourrier committed 176 days ago

Verified d805f9fa

fix: update python api user docs (#784)

ryantzr1 committed 193 days ago

Verified 88bd36ae

Update extractive_match_utils.py for words where `:` is preceded by a space (#831)

clefourrier committed 197 days ago

Verified 126f908a

Complete TranslationLiterals for Language.ESTONIAN (#779)

spyysalo committed 197 days ago

Verified a455539b

Fix GPQA and index extractive metric (#829)

clefourrier committed 197 days ago

Verified 50c78a2e

Icelandic addition (#775)

joenaess committed 198 days ago

Verified 4dc92474

Refacto and remove bloated code (#709)

NathanHB committed 198 days ago

Verified 9288bd84

Fix system prompt concatenated with instruction in chat template (#830)

clefourrier committed 199 days ago

Verified 0ba8812f

add TUMLU-mini benchmark, solves #577 (#811)

ceferisbarov committed 199 days ago

Verified 936612cb

Add TranslationLiterals for Language.DANISH (#770)

spyysalo committed 199 days ago

Verified b0c092a1

Fix Typos in Documentation and Task Description (#810)

leopardracer committed 199 days ago

Verified a3725503

Catch ROCM/HIP/AMD oom in `should_reduce_batch_size` (#812)

mcleish7 committed 200 days ago

Verified 95ec6691

Flores-200 combinations->permutations (#820)

anton-l committed 200 days ago

Verified dd1af5a9

set default temperature to 0 in generation config (#814)

NathanHB committed 203 days ago

Verified 327071fe

Older