unstructured
2d3cd45b - Fix/reduce memory usage (#3629)

Commit
1 year ago
Fix/reduce memory usage (#3629) This PR fixes the high memory usage when computing intersection areas. - it now converts the coordinates into half precision floating point numbers instead of double - removes some intermediate variables to free up memory usage ## test Using a memory profiler like `memory_profiler` in `ipython`: ```ipython ## cell 1 from unstructured.partition.pdf_image.pdfminer_processing import areas_of_boxes_and_intersection_area import numpy as np %load_ext memory_profiler ## cell 2 %%memit coords = np.random.rand(40000).reshape((10000,4)).astype(np.float16) ## cell 3 %%memit inter_area, boxa_area, boxb_area = areas_of_boxes_and_intersection_area(coords, coords) ``` The peak memory and incremental memory from cell 3 should be close to ``` peak memory: 730.55 MiB, increment: 573.22 MiB ``` On main branch the `coords` is double precision and running the same code with ``` coords = np.random.rand(40000).reshape((10000,4)).astype(np.float64) ``` would result in peak memory usage more than 4GiB --------- Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: christinestraub <christinestraub@users.noreply.github.com> Co-authored-by: christinestraub <christinemstraub@gmail.com>
Author
Parents
Loading