Smart dataset update: don't recompute everything all the time (#2788)
* add smart_set_revision
* add SmartDatasetBackfillPlan
* implement SmartDatasetUpdatePlan
* first tests
* more tests
* add smart_update_dataset + simple out-of-order commits support
* mypy
* mypy
* don't require headSha
* support 'viewer' yaml field
* test with storage_client
* Apply suggestions from code review
Co-authored-by: Sylvain Lesage <sylvain.lesage@huggingface.co>
* style
* sylvain's comments
* unused import
* test cached_revision
* move generate_object_key to StorageClient
* add comment
* typo
* docstring
* sylvain's comments
* style
* raise on parsing error
---------
Co-authored-by: Sylvain Lesage <sylvain.lesage@huggingface.co>