Add docstring docs #413
Add Reference docs with Pipeline docs
There is an incompatibility issue within the lighteval[dev] environment while generating the docs:
- thinc-8.2.4 requires numpy<2
- numpy-2.1.3 is installed
- apparently there is a package that removed the requirement numpy<2
https://github.com/huggingface/lighteval/actions/runs/12084402368/job/33699519327?pr=413
File "thinc/backends/numpy_ops.pyx", line 1, in init thinc.backends.numpy_ops
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
I pin numpy < 2.
Pin numpy<2
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Preliminary docs with docstrings (well, we need to add the docstrings): https://moon-ci-docs.huggingface.co/docs/lighteval/pr_413/en/package_reference/pipeline
Add Tasks docs
Add more Tasks docs
Add Models docs
Fix Models docs
Remove AdapterModel that requires peft
Remove NanotronLightevalModel and VLLMModel that require nanotron andβ¦
Fix markdown comment syntax
Add Metrics docs
Fix typo
Remove Main classes section
Add Datasets docs
Create Main classes section with Pipeline
Add EvaluationTracker docs
Add ModelConfig docs
Add ParallelismManager to Pipeline docs
Add inter-links from using-the-python-api
Fix inter-links
Add more Metrics docs
Comment Metrics enum
Fix typo
Add explanation and GH issue to comment in Metrics enum
Add inter-link to Metrics
Add subsection titles to LightevalTask
Add inter-link to LightevalTaskConfig
Add inter-link to section heading anchor
Add more Metrics docs
Add inter-link to SampleLevelMetric and Grouping
Add inter-link to LightevalTaskConfig
Fix section title with trailing colon
Maybe we should rename the section "API" (with available metrics and tasks) to avoid confusion with the "Reference" section (containing the docstrings of classes/functions).
Ready for review:
- I see this PR as setting up the docs for classes/functions using their docstrings.
- Future PRs could add missing docstrings and improve the existing ones, if needed.
Looking very nice!
Some things (I scrolled the PR but mostly inspected the published docs):
- models does not have a sidebar, so I suggested a split by provider (transformers, tgi/inference endpoints, nanotron, vllm, etc)
- I would put the models as one of the main classes since it's also possible to start an evaluation with Pipeline + Tracker + Model,
- we're missing the doc of the rest of the logging sytem
- do you know where the doc icon is defined? should be a π€οΈ
Not sure about the API name for the lists of possible metrics/tasks - now that we have the doc from the code we should probably use that but let's think about it for another PR
I took a look at the final docs and they are neat (also allows to see very easily where the doc is parsable or messy instead). Tysm for this work!
Add sections to Models docs
Move Models docs to Main classes section
Document you can pass either model or model config to Pipeline
Move Datasets docs to Tasks docs
Add logging docs
- models does not have a sidebar, so I suggested a split by provider (transformers, tgi/inference endpoints, nanotron, vllm, etc)
Done for the models that can be made into docs; other models raise an import error when generating their docs and I have commented out their docs entries; I might open an Issue for this.
- I would put the models as one of the main classes since it's also possible to start an evaluation with Pipeline + Tracker + Model,
Done:
- Put the models into Main classes.
- Update the "Using the Python API" docs to indicate that a model can be passed as well.
- we're missing the doc of the rest of the logging sytem
I added the Logging docs with all the info loggers
- do you know where the doc icon is defined?
Not done: I don't know. I might also open an Issue to address this in an upcoming PR.
Perfect, thanks for your changes, merging :)
Login to write a write a comment.
Add docstring docs.
I see this PR as setting up the docs for classes/functions using their docstrings.
Future PRs could add missing docstrings and improve the existing ones, if needed.