PR #276 Probability Metric + New Normalization

Probability Metric + New Normalization #276

NathanHB merged 20 commits into main from prob_metrics_and_more_norms

hynky1999 force pushed from 0d351b26 to 6bcf160c 1 year ago

clefourrier commented on 2024-08-22

hynky1999 requested a review from

clefourrier 1 year ago

NathanHB commented on 2024-08-27

add new acci + prob metrics

53b56c39

Add logic for handli PMI + add tests for requests

c11b4229

formatting + nits

974e0dc0

nicer comments

80b4f012

simplify normalization name, fix bug with ignore space

78e8e7f4

better docs for uncoditioned_query

3c40051b

resolving gh comments

9443be86

nits + fix tests

659907ca

formatting

5cb13dbb

fix metric category separation of requests

49585246

Update src/lighteval/logging/info_loggers.py

8910ad53

Update src/lighteval/metrics/__init__.py

9790c73e

Update src/lighteval/tasks/requests.py

00bd2ecf

Update tests/metrics/test_metric_requests.py

20513e23

hynky1999 force pushed from f3335870 to 20513e23 1 year ago

rework perplexity routing, move probabilities to target_perplexity

0dd7e6c8

fix typo

3f4ae478

hynky1999 requested a review from

NathanHB 1 year ago

NathanHB commented on 2024-09-02

Update src/lighteval/metrics/sample_preparator.py

d98abebd

NathanHB commented on 2024-09-02

Update src/lighteval/tasks/lighteval_task.py

d99b64b6

ruff

9de0a39e

Merge branch 'main' into prob_metrics_and_more_norms

8b35ca79

NathanHB approved these changes on 2024-09-02

NathanHB merged 8c787df2 into main 1 year ago

Reviewers

NathanHB

clefourrier

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

lighteval Probability Metric + New Normalization #276 Merged

Probability Metric + New Normalization #276

lighteval
Probability Metric + New Normalization
#276

Merged