unstructured

03be9c07 - enhancement: Speed up function `clean_extra_whitespace_with_index_run` by 68% (#4166)

enhancement: Speed up function `clean_extra_whitespace_with_index_run` by 68% (#4166) <!-- CODEFLASH_OPTIMIZATION: {"function":"clean_extra_whitespace_with_index_run","file":"unstructured/cleaners/core.py","speedup_pct":"68%","speedup_x":"0.68x","original_runtime":"3.74 milliseconds","best_runtime":"2.22 milliseconds","optimization_type":"loop","timestamp":"2025-12-23T05:49:45.872Z","version":"1.0"} --> #### 📄 68% (0.68x) speedup for ***`clean_extra_whitespace_with_index_run` in `unstructured/cleaners/core.py`*** ⏱️ Runtime : **`3.74 milliseconds`** **→** **`2.22 milliseconds`** (best of `19` runs) #### 📝 Explanation and details The optimized code achieves a **68% speedup** through two key changes that eliminate expensive operations in the main loop: ## What Changed 1. **Character replacement optimization**: Replaced `re.sub(r"[\xa0\n]", " ", text)` with `text.translate()` using a translation table. This avoids regex compilation and pattern matching for simple character substitutions. 2. **Main loop optimization**: Eliminated two `re.match()` calls per iteration by: - Pre-computing character comparisons (`c_orig = text_chars[original_index]`) - Using set membership (`c_orig in ws_chars`) instead of regex matching - Direct character comparison (`c_clean == ' '`) instead of regex ## Why It's Faster Looking at the line profiler data, the original code spent **15.4% of total time** (10.8% + 4.6%) on regex matching inside the loop: - `bool(re.match("[\xa0\n]", text[original_index]))` - 7.12ms (10.8%) - `bool(re.match(" ", cleaned_text[cleaned_index]))` - 3.02ms (4.6%) The optimized version replaces these with: - Set membership check: `c_orig in ws_chars` - 1.07ms (1.4%) - Direct comparison: `c_clean == ' '` (included in same line) **Result**: Regex overhead is eliminated, saving ~9ms per 142 invocations in the benchmark. ## Performance Profile The annotated tests show the optimization excels when: - **Large inputs with whitespace**: `test_large_leading_and_trailing_whitespace` shows 291% speedup (203μs → 52.1μs) - **Many consecutive whitespace characters**: `test_large_mixed_whitespace_everywhere` shows 297% speedup (189μs → 47.8μs) - **Mixed whitespace types** (spaces, newlines, nbsp): `test_edge_all_whitespace_between_words` shows 47.9% speedup Small inputs with minimal whitespace see minor regressions (~5-17% slower) due to setup overhead, but these are negligible in absolute terms (< 2μs difference). ## Impact on Production Workloads The function is called in `_process_pdfminer_pages()` during PDF text extraction, processing **every text snippet on every page**. Given that PDFs often contain: - Multiple spaces/tabs between words - Newlines from paragraph breaks - Non-breaking spaces from formatting This optimization will provide substantial cumulative benefits when processing large documents with hundreds of pages, as the per-snippet savings compound across the entire document. ✅ **Correctness verification report:** | Test | Status | | --------------------------- | ----------------- | | ⚙️ Existing Unit Tests | 🔘 **None Found** | | 🌀 Generated Regression Tests | ✅ **45 Passed** | | ⏪ Replay Tests | ✅ **16 Passed** | | 🔎 Concolic Coverage Tests | ✅ **1 Passed** | |📊 Tests Coverage | 100.0% | <details> <summary>🌀 Click to see Generated Regression Tests</summary> ```python from __future__ import annotations # imports from unstructured.cleaners.core import clean_extra_whitespace_with_index_run # unit tests # --- BASIC TEST CASES --- def test_basic_single_spaces(): # No extra whitespace, should remain unchanged text = "Hello world" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 8.95μs -> 9.71μs (7.88% slower) def test_basic_multiple_spaces(): # Multiple spaces between words should be reduced to one text = "Hello world" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 11.0μs -> 10.00μs (10.0% faster) def test_basic_newlines_and_nbsp(): # Newlines and non-breaking spaces replaced with single space text = "Hello\n\xa0world" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 12.8μs -> 10.2μs (25.2% faster) def test_basic_leading_and_trailing_spaces(): # Leading and trailing spaces should be stripped text = " Hello world " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 10.4μs -> 9.88μs (5.62% faster) def test_basic_only_spaces(): # Only spaces should return an empty string text = " " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 6.10μs -> 6.45μs (5.43% slower) def test_basic_only_newlines_and_nbsp(): # Only newlines and non-breaking spaces should return empty string text = "\n\xa0\n\xa0" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 6.47μs -> 6.21μs (4.25% faster) def test_basic_mixed_whitespace_between_words(): # Mixed spaces, newlines, and nbsp between words text = "A\n\n\xa0 B" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 12.9μs -> 9.07μs (41.9% faster) # --- EDGE TEST CASES --- def test_edge_empty_string(): # Empty string should return empty string and empty indices text = "" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 5.53μs -> 5.62μs (1.73% slower) def test_edge_all_whitespace(): # String with only whitespace, newlines, and nbsp text = " \n\xa0 \n\xa0" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 6.91μs -> 7.15μs (3.40% slower) def test_edge_one_character(): # Single non-whitespace character text = "A" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 5.86μs -> 6.33μs (7.52% slower) def test_edge_one_whitespace_character(): # Single whitespace character text = " " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 5.26μs -> 5.96μs (11.8% slower) def test_edge_whitespace_between_every_char(): # Whitespace between every character text = "H E L L O" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 7.13μs -> 8.59μs (17.0% slower) def test_edge_multiple_types_of_whitespace(): # Combination of spaces, newlines, and nbsp between words text = "A \n\xa0 B" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 12.3μs -> 8.56μs (44.1% faster) def test_edge_trailing_newlines_and_nbsp(): # Trailing newlines and nbsp should be stripped text = "Hello world\n\xa0" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 8.36μs -> 9.20μs (9.07% slower) def test_edge_leading_newlines_and_nbsp(): # Leading newlines and nbsp should be stripped text = "\n\xa0Hello world" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 11.3μs -> 9.86μs (14.6% faster) def test_edge_alternating_whitespace(): # Alternating whitespace and characters text = " H E L L O " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 8.30μs -> 8.81μs (5.80% slower) def test_edge_long_run_of_whitespace(): # Long run of whitespace in the middle text = "Hello" + (" " * 50) + "world" cleaned, indices = clean_extra_whitespace_with_index_run(text) # 27.5μs -> 13.4μs (106% faster) # --- LARGE SCALE TEST CASES --- def test_large_no_extra_whitespace(): # Large string with no extra whitespace text = "A" * 1000 cleaned, indices = clean_extra_whitespace_with_index_run(text) # 106μs -> 93.6μs (13.3% faster) def test_large_all_whitespace(): # Large string of only whitespace text = " " * 1000 cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 13.1μs -> 8.95μs (46.6% faster) def test_large_alternating_char_and_whitespace(): # Large string alternating between character and whitespace text = "".join(["A " for _ in range(500)]) # 500 'A ', total length 1000 cleaned, indices = clean_extra_whitespace_with_index_run(text) # 106μs -> 95.5μs (11.5% faster) def test_large_multiple_whitespace_blocks(): # Large string with random blocks of whitespace text = "A" + (" " * 10) + "B" + ("\n" * 10) + "C" + ("\xa0" * 10) + "D" cleaned, indices = clean_extra_whitespace_with_index_run(text) # 28.6μs -> 12.9μs (122% faster) def test_large_leading_and_trailing_whitespace(): # Large leading and trailing whitespace text = (" " * 500) + "Hello world" + (" " * 500) cleaned, indices = clean_extra_whitespace_with_index_run(text) # 203μs -> 52.1μs (291% faster) def test_large_mixed_whitespace_everywhere(): # Large text with mixed whitespace everywhere text = (" " * 100) + "A" + ("\n" * 100) + "B" + ("\xa0" * 100) + "C" + (" " * 100) cleaned, indices = clean_extra_whitespace_with_index_run(text) # 189μs -> 47.8μs (297% faster) # --- FUNCTIONALITY AND INTEGRITY TESTS --- def test_mutation_detection_extra_space(): # If function fails to remove extra spaces, test should fail text = "Test case" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 9.65μs -> 8.87μs (8.84% faster) def test_mutation_detection_strip(): # If function fails to strip leading/trailing whitespace, test should fail text = " Test case " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 9.64μs -> 8.97μs (7.41% faster) def test_mutation_detection_newline_nbsp(): # If function fails to replace newlines or nbsp, test should fail text = "Test\n\xa0case" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 11.7μs -> 9.45μs (23.5% faster) def test_mutation_detection_index_integrity(): # Changing the index logic should break this test text = "A B" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 8.82μs -> 7.73μs (14.2% faster) def test_mutation_detection_empty_output(): # If function fails to return empty string for all whitespace, test should fail text = " \n\xa0 " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 7.79μs -> 8.53μs (8.65% slower) # codeflash_output is used to check that the output of the original code is the same as that of the optimized code. ``` ```python from __future__ import annotations # imports from unstructured.cleaners.core import clean_extra_whitespace_with_index_run # unit tests # 1. Basic Test Cases def test_basic_no_extra_whitespace(): # Text with no extra whitespace should remain unchanged text = "Hello world!" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 10.3μs -> 10.9μs (5.46% slower) def test_basic_multiple_spaces_between_words(): # Multiple spaces between words should be reduced to one text = "Hello world!" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 11.1μs -> 10.2μs (9.12% faster) def test_basic_leading_and_trailing_spaces(): # Leading and trailing spaces should be stripped text = " Hello world! " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 10.5μs -> 9.89μs (6.26% faster) def test_basic_newline_and_nonbreaking_space(): # Newlines and non-breaking spaces should be converted to single spaces text = "Hello\nworld!\xa0Test" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 12.4μs -> 9.69μs (28.0% faster) def test_basic_combined_whitespace_types(): # Combination of spaces, newlines, and non-breaking spaces text = "A \n\xa0 B" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 11.4μs -> 9.02μs (26.3% faster) # 2. Edge Test Cases def test_edge_empty_string(): # Empty string should return empty string and empty indices text = "" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 5.58μs -> 5.64μs (1.01% slower) def test_edge_only_spaces(): # String with only spaces should return empty string text = " " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 5.98μs -> 6.55μs (8.71% slower) def test_edge_only_newlines_and_nbsp(): # String with only newlines and non-breaking spaces text = "\n\xa0\n\xa0" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 6.54μs -> 6.06μs (7.91% faster) def test_edge_single_character(): # Single character should remain unchanged text = "A" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 6.01μs -> 6.45μs (6.78% slower) def test_edge_all_whitespace_between_words(): # All whitespace between words should be reduced to one space text = "A \n\xa0 B" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 13.4μs -> 9.08μs (47.9% faster) def test_edge_whitespace_at_various_positions(): # Whitespace at start, middle, and end text = " A B " cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 9.58μs -> 8.26μs (16.0% faster) def test_edge_multiple_consecutive_whitespace_groups(): # Several groups of consecutive whitespace text = "A \n\n B C" cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 12.9μs -> 9.42μs (37.3% faster) # 3. Large Scale Test Cases def test_large_long_string_with_regular_spacing(): # Large string with regular words and single spaces text = "word " * 200 cleaned, indices = clean_extra_whitespace_with_index_run( text.strip() ) # 107μs -> 95.9μs (12.2% faster) def test_large_long_string_with_extra_spaces(): # Large string with extra spaces between words text = ("word " * 200).strip() cleaned, indices = clean_extra_whitespace_with_index_run(text) # 402μs -> 180μs (123% faster) def test_large_mixed_whitespace(): # Large string with mixed whitespace types words = ["word"] * 500 text = " \n\xa0 ".join(words) cleaned, indices = clean_extra_whitespace_with_index_run(text) # 1.37ms -> 598μs (129% faster) def test_large_leading_and_trailing_whitespace(): # Large string with leading and trailing whitespace text = " " * 100 + "word " * 800 + " " * 100 cleaned, indices = clean_extra_whitespace_with_index_run(text) # 468μs -> 374μs (25.1% faster) def test_large_string_all_whitespace(): # Large string of only whitespace text = " " * 999 cleaned, indices = clean_extra_whitespace_with_index_run( text ) # 13.8μs -> 8.85μs (55.9% faster) # codeflash_output is used to check that the output of the original code is the same as that of the optimized code. ``` ```python from unstructured.cleaners.core import clean_extra_whitespace_with_index_run def test_clean_extra_whitespace_with_index_run(): clean_extra_whitespace_with_index_run("\n\x00") ``` </details> <details> <summary>⏪ Click to see Replay Tests</summary> | Test File::Test Function | Original ⏱️ | Optimized ⏱️ | Speedup | |:--------------------------------------------------------------------------------------------------------------|:--------------|:---------------|:----------| | `test_benchmark1_py__replay_test_0.py::test_unstructured_cleaners_core_clean_extra_whitespace_with_index_run` | 376μs | 347μs | 8.63%✅ | </details> <details> <summary>🔎 Click to see Concolic Coverage Tests</summary> | Test File::Test Function | Original ⏱️ | Optimized ⏱️ | Speedup | |:----------------------------------------------------------------------------------------------------------------|:--------------|:---------------|:----------| | `codeflash_concolic_3yq4ufg_/tmp5dfyu5tu/test_concolic_coverage.py::test_clean_extra_whitespace_with_index_run` | 27.1μs | 17.7μs | 52.7%✅ | </details> To edit these changes `git checkout codeflash/optimize-clean_extra_whitespace_with_index_run-mji60td0` and push. [](https://codeflash.ai)  --------- Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com> Co-authored-by: Aseem Saxena <aseem.bits@gmail.com> Co-authored-by: qued <64741807+qued@users.noreply.github.com> Co-authored-by: Alan Bertl <alan@unstructured.io>