Add support for multi-chunk external data files (#1212)
* Cache path vs Local path
* Update src/utils/hub.js
* complete the full parameters of option for pipeline api, see: https://github.com/huggingface/transformers.js/issues/1200
Signed-off-by: Thomas Tang <hello@tangkunyin.com>
* Add support for multi-chunk external data files
* Add MAX_EXTERNAL_DATA_CHUNKS check
* Support loading inference sessions from paths (node.js external data)
* Fix types
* Add support for node.js external data & optimize downloading
* Remove debug logs
* Optimize FileResponse body stream for large models
* Expose `use_external_data_format` in pipeline options. TODO: Improve support and allow per-file specification
* Fix post-merge
* Dedent comments
* Default use_external_data_format to null (unset)
* let -> const where possible
* Support setting external data format in config.json
* Add external data model architecture tests
* Simplify tests
* Formatting
---------
Signed-off-by: Thomas Tang <hello@tangkunyin.com>
Co-authored-by: axrati <axmelluzzo@gmail.com>
Co-authored-by: Axm <82290907+axrati@users.noreply.github.com>
Co-authored-by: Thomas Tang <hello@tangkunyin.com>
