lighteval
Adds inspectai
#1022
Merged
Overview
Commits
67
Changes
View On GitHub

Adds inspectai #1022

NathanHB merged 67 commits into main from nathan-move-to-inspectai
NathanHB
NathanHB use inspect-ai to evaluate aime25 and gsm8k
2696a49c
NathanHB revert file
578d5308
NathanHB working for 3 tasks
21fa870d
NathanHB parallel evals of tasks
27b2af11
NathanHB adds gpqa diamond to inspect
b9a610dc
NathanHB move tasks to individual files
25c11285
NathanHB move tasks to individual files
0d42edf4
NathanHB enable extended tasks as well
6cc3c041
NathanHB run precomit hook
4c38951d
NathanHB fix mkqa
d2fd5e1e
NathanHB chaange extended suite to lighteval
2ddb0f94
NathanHB chaange extended suite to lighteval
ee971228
NathanHB add metdata to tasks
e2c8e226
NathanHB add metdata to tasks
c980ddbe
NathanHB remove license notice and put docstring on top of file
57fe3908
NathanHB homogenize tags
ee081f20
NathanHB add docstring for all multilingual tasks
1ed1602a
NathanHB add docstring for all multilingual tasks
f4b0e274
NathanHB add name and dataset to metadata
81d9e4ed
NathanHB use TASKS_TABLE for multilingual tasks
b7345327
NathanHB use TASKS_TABLE for default tasks
c3911fcf
NathanHB use TASKS_TABLE for default tasks
e439f706
NathanHB loads all tasks correclty
6447ee75
NathanHB move community tasks to default tasks and update doc
88754bfa
NathanHB move community tasks to default tasks and update doc
5445f5c0
NathanHB Merge remote-tracking branch 'origin/main' into nathan-reorg-tasks
f53bd76f
NathanHB revert uneeded changes
6a0c615d
NathanHB fix doc build
1435e382
NathanHB fix doc build
15f41f26
NathanHB remove custom tasks and let user decide if loading multilingual tasks
74e5c0f4
NathanHB load-tasks multilingual fix
aad136c1
NathanHB update doc
242bc438
NathanHB remove uneeded file
6806bf88
NathanHB update readme
e94fa590
NathanHB update readme
8800d1ac
NathanHB update readme
970f33bf
NathanHB fix test
b8c26dc2
NathanHB add back the custom tasks
764de725
NathanHB add back the custom tasks
a326ea86
NathanHB fix tasks
81081cde
NathanHB fix tasks
74b40f62
NathanHB fix tasks
083fb1b5
NathanHB fix tests
2dab2bfd
NathanHB fix tests
57ca0e53
NathanHB add inspect-ai
480e40af
NathanHB NathanHB changed the base branch from main to nathan-reorg-tasks 176 days ago
NathanHB NathanHB marked this pull request as draft 176 days ago
NathanHB add tasks
ade29007
NathanHB add gpqa
079ceaf1
NathanHB make model config work
8d007997
NathanHB NathanHB marked this pull request as ready for review 168 days ago
NathanHB NathanHB requested a review from copilot-pull-request-reviewer copilot-pull-request-reviewer 168 days ago
copilot-pull-request-reviewer
copilot-pull-request-reviewer commented on 2025-10-29
NathanHB Update src/lighteval/metrics/metrics.py
cea5e997
NathanHB init
fb47bb78
NathanHB Merge branch 'nathan-move-to-inspectai' of github.com:huggingface/lig…
2736bc9c
HuggingFaceDocBuilderDev
NathanHB NathanHB changed the base branch from nathan-reorg-tasks to main 167 days ago
NathanHB Merge branch 'main' into nathan-move-to-inspectai
d5e6c9fd
NathanHB fix tests
e55a9af9
NathanHB Merge branch 'nathan-move-to-inspectai' of github.com:huggingface/lig…
ba41f1c3
NathanHB fix tests
59c5dcc4
NathanHB fix tests
40254db0
NathanHB fix tests
53275fe9
clefourrier
clefourrier commented on 2025-10-30
NathanHB add correct system prompt for hle
72e5c2b5
NathanHB add correct system prompt for hle
7fc1753d
NathanHB
clefourrier
clefourrier approved these changes on 2025-10-31
NathanHB review suggestions
260d7443
NathanHB add doc
835b7990
NathanHB change buttons
c216a272
NathanHB change buttons
21e6020f
NathanHB change buttons
7e654008
NathanHB move benchmark finder to openeval org
0a4f6bef
NathanHB better help for eval
b661d0d7
NathanHB better help for eval
f142b391
NathanHB NathanHB merged 880bebef into main 162 days ago
NathanHB NathanHB added feature
NathanHB NathanHB added enhancement

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
feature enhancement
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum