[v4] Introduce ModelRegistry API (#1511)
* added progress_total progress callback status info
* added get_file_metadata helper
* some clean up
* improved get_file_metadata and get_files
* added functions to main export
* removed dynamic import
* restructuring
* refactored the pipeline tasks so I can have a get_pipeline_files function that does not check for tokenizer files or processor files if the task does not use them
* updated doc
* Update packages/transformers/src/utils/core.js
Co-authored-by: Joshua Lochner <admin@xenova.com>
* added is_pipeline_cached and improved return object
* fixes after review
* added ModelRegistry to doc
* added clear_cache and clear_pipeline_cache
* Update packages/transformers/src/utils/cache/clear_cache.js
Co-authored-by: Joshua Lochner <admin@xenova.com>
* small doc fix
* changed delete logic for cache
* fixed examples in cache utilitiy files
* fixed examples
* renamed type
* refactoring get_file_metadata
* moved src/utils/pipeline-tasks.js to src/pipelines/index.js
* fixed doc builder
* fixed doc builder
* created shared getFetchHeaders function
* added case for DecoderOnlyWithoutHead and DecoderOnly
* improved console.warn
* changed to modelType = MODEL_TYPES.EncoderOnly if not foundInMapping
* removed full download from get_file_metadata
* Remove duplicate module tag (already set in ModelRegistry.js)
* Remove test file
* pnpm format
* Cleanup
* use config from_pretrained logic for ensuring config is of correct type
* Reorder file acquisition
* Add example JSDoc to file header
* Add ModelRegistry tests
* FIXME: skip cache clearing tests
breaks simultaneous loading
* Formatting
* Unify model-loader.js, get_model_files.js, and session.js
* console.warn to logger.warn
* Only resolve dtype once in session.js
* Use map + Promise.all
* map -> forEach
* Use env.fetch instead of global fetch
* Add comment to clear_cache for clarity
* Add model_file_name support in cache operations
* Update cache tests
* Fix TOCTOU race condition
* Remove dead code
* cleanup
* Cleanup pipeline import/exports
* renamed folder cache to model_registry
* changed doc title
* fix for unit tests on node 20
* standardize module name
---------
Co-authored-by: Joshua Lochner <admin@xenova.com>
Co-authored-by: Joshua Lochner <26504141+xenova@users.noreply.github.com>
