test ci training for text model only (#42597)
* begin test ci training
* add better logging
* add better logging + training loop
* fix sentence + grad_norm assert
* create circlci config fort training
* fix ci to detect training_ci job
* add -s for pytest CI
* add generate assert as well
* make training ci trigger for every model change instead
* set eos_token_id to 0 otherwise it will stop generating too soon
* refactor
* moving logging and metrics to proper files
* update marker in pyproject.toml
* linting
* linting again
* reduce pytest worker
* fix deadlock in test
* add license
* dont show logs during test
* loosen threshold a bit
* Initialize cache state in RecurrentGemmaRecurrentBlock based on batch… (#42627)
* Initialize cache state in RecurrentGemmaRecurrentBlock based on batch size
* loosen a bit the threshold
* Revert "loosen a bit the threshold"
This reverts commit d3d42e1e3b632e62eab016d3660bd1172fb326eb.
* skipping BLT for now (until it get fixed)
