dataset-viewer
d1c56d3d - Store transformed values in duckdb index file (#2737)

Commit
1 year ago
Store transformed values in duckdb index file (#2737) * index string lengths * add image widths&lengths + refactor a bit * add check for actul values in result (for string type) * fix lists + keep None values in audio and image * fix processing None values in audio and image * treat cases when all values in column are None * query specific columns in search (instead of data.*) * get unsupported columns once * rename columns __hf_length -> .length, __hf_duration -> .duration and so on * refactor: move stats utils back to worker but outside of job runners --------- Co-authored-by: Sylvain Lesage <sylvain.lesage@huggingface.co>
Author
Polina Kazakova
Parents
Loading