unstructured

961c8d5b - feat: use block matrix to reduce peak memory usage for matmul (#3947)

feat: use block matrix to reduce peak memory usage for matmul (#3947) This PR targets the most memory expensive operation in partition pdf and images: deduplicate pdfminer elements. In large pages the number of elements can be over 10k, which would generate multiple 10k x 10k square double float matrices during deduplication, pushing peak memory usage close to 13Gb  This PR breaks this computation down by computing partial IOU. More precisely it computes IOU for each 2000 elements against all the elements at a time to reduce peak memory usage by about 10x to around 1.6Gb.  The block size is configurable based on user preference for peak memory usage and it is set by changing the env `UNST_MATMUL_MEMORY_CAP_IN_GB`.