fix(html): keep text inside unrecognized elements (#4497)
### Summary
Fixes #4496. Resolves #3842 (#3708 was closed in its favor).
The v1 HTML parser gives every tag it has no class for a
`DefaultElement`, and `DefaultElement` emits only the element's tail:
its own text and everything inside it are dropped without a trace.
Before the parser rewrite in #3218 (0.15.0) the legacy parser listed
`font` in its `TEXT_TAGS`, so for `<font>` this is also a regression. In
practice this loses:
- `<font>`, still common in email HTML (Outlook and Gmail reply
headers).
- Inline XBRL, required in SEC 10-K/10-Q filings: `<ix:nonFraction>`
wraps the numbers inside sentences, and
`<ix:nonNumeric>`/`<ix:continuation>` wrap whole notes, tables included.
- Custom elements (`<my-card>`, the `<hauptteil_AB>` from #3708),
`<ins>`, `<nobr>`, `<ruby>`, `<picture>`.
Everything routed through `partition_html()` is affected: `.html`,
`.md`, `.eml`/`.msg` HTML bodies, `.epub`, `.rst`, `.org`.
Measured on the repo's fixtures. "Token coverage" is the share of word
tokens from a reference text, built from each fixture's own HTML, that
appear in the output; the reference approximates, but is not, what a
browser displays (method in **Testing**).
| fixture | elements, main → this PR | token coverage, main → this PR |
|---|---|---|
| `example-10k.html` (same for `-utf-16`) | 314 → 466 | 66.0% → 100.0%
of 27,619 tokens |
| `example-10k-1p.html` | 23 → 24 | 97.3% → 100.0% |
| `example-steelJIS-datasheet.html` (same for `-utf-16`) | 12 → 19 |
75.0% → 88.7% |
| `eml/email-no-utf8-2008-07-16.062410.eml`, HTML body | 2 → 22 | 31.5%
→ 93.8% |
On `main` the 10-K loses its cover-page values (form type, fiscal year
end, registrant name, shares outstanding) and all of Notes 1–14. The
email keeps two quoted paragraphs; the replies and their `From:`/`Sent:`
headers are inside `<font>`. The other 40 HTML-routed fixtures in
`example-docs` produce identical output.
### Change
`DefaultElement` drops its two overrides and inherits
`Phrasing.iter_text_segments()` and `Flow.iter_elements()`, so an
unrecognized element is transparent.
On the block-vs-inline question in #3842, no guess is needed. The
element keeps identifying as phrasing, which is how a browser renders an
unknown element (`display: inline`), and the parser already turns each
block nested inside phrasing into its own element, as for a `<p>` inside
a `<span>`. So `<font>` around two words joins the sentence, and
`<ix:nonNumeric>` around a note with headings and tables yields the same
elements as the note without the wrapper. Checked on the real filing:
`example-10k.html` with every `ix:` wrapper stripped (content kept)
partitions to the same 466 elements; 448 are identical, and the other 18
are tables whose text differs only in whitespace from `TableBlock` code
this PR doesn't touch. Text mixed with blocks
(`<x>before<p>para</p>after</x>`) and emphasis and link annotations come
out exactly as they do inside a `<span>`.
Elements whose content is not rendered as text now have to be skipped
explicitly. They are registered as `RemovedPhrasing`, which behaves
exactly like the old `DefaultElement`, so their output does not change:
- metadata: `head`, `title`, and legacy `xml` data islands (such as
Office document properties);
- fallback content: `object`, `iframe`, `audio`, `video`, `canvas`,
`noembed`, `noframes`;
- form-control internals: `select`, `datalist`, `textarea`;
- `dialog`, which is hidden until opened and on static pages is mostly
popup UI;
- `svg`, `math`, and Inline XBRL's hidden `ix:header` (facts, contexts
and units that filings put in a `display:none` `<div>`).
Each entry has a test that leaks text if the entry is removed. Judgment
calls:
- `math` stays skipped so this doesn't pre-empt #4478, which adds a
`Math` class. Whichever PR lands second deletes that line; ruff's F601
flags the duplicate key otherwise.
- `dialog` is skipped whether or not it has `open`, which keeps today's
output for both. An `open`-aware class would be a small follow-up if you
want open dialogs included.
- `embed`, `source` and `track` are left transparent: libxml2 nests the
*following* content inside these void elements (`<p>See <embed src=x>
for details.</p>` puts "for details." inside `<embed>`), so skipping
them would keep losing that text.
- Tags that are already registered (`button`, `label`, `details`, `dl`,
`figure`, `form`, `nav`, `template`, …) are untouched.
`Flow._page_number` now walks ancestors in a loop instead of recursing,
with the same result. `DefaultElement` is a `Flow`, so a block inside N
nested unknown elements now looks up N ancestors while the phrasing
traversal is itself about 2N frames deep; without this, a `<p>` inside
250 nested unknown elements raised `RecursionError`.
The v2 parser (`html_parser_version="v2"`) is not affected; it keeps
unknown-tag text as `UncategorizedText`.
### Before / after
```python
from unstructured.partition.html import partition_html
for html in [
"<p><font face=Arial>Quarterly revenue</font> rose 12%.</p>",
'<p>Revenue was $<ix:nonFraction name="us-gaap:Revenues">4,213</ix:nonFraction> million.</p>',
"<my-card><h2>Pricing</h2><p>Plans start at $10.</p></my-card>",
]:
print([str(e) for e in partition_html(text=html)])
```
Before:
```
['rose 12%.']
['Revenue was $ million.']
[]
```
After:
```
['Quarterly revenue rose 12%.']
['Revenue was $4,213 million.']
['Pricing', 'Plans start at $10.']
```
### Testing
- `test_parser.py`: the three `DescribeDefaultElement` tests that pinned
the old behavior now assert transparency. New tests cover blocks inside
`<foobar>`, `<ix:nonNumeric>` and nested `<my-card><font>` (same
elements and metadata as without the wrapper), text mixed with a block
(same as inside `<span>`), annotations, each newly skipped tag
(parametrized, including open and closed `<dialog>`, plus `head` via a
body-less document), and the page number of a block nested in 250
unknown elements.
- `test_partition.py`: end-to-end tests for `<font>` and for Inline XBRL
(inline fact, block `ix:nonNumeric` note with a table, `ix:header`
content that must not appear).
- Against `main`'s code, all 36 new or changed parser tests and both
end-to-end tests fail.
- `test_unstructured/partition/html/`: 402 passed, 1 failed on this
branch; 367 passed, 1 failed on `main`. The failure is the same test on
both, from my Windows checkout's CRLF fixtures.
- `ruff check .` and `ruff format --check` on the changed files pass.
<details>
<summary>Other suites compared on main and this branch</summary>
Compared test-by-test; every test present on both has the same outcome.
| suite | main | this branch |
|---|---|---|
| `test_md.py` / `test_email.py` | 39 / 72 passed | 39 / 72 passed |
| `test_msg.py` | 49 passed, 1 failed | 49 passed, 1 failed |
| `test_epub.py` + `test_rst.py` + `test_org.py` | 46 passed, 1 failed |
46 passed, 1 failed |
| `chunking/test_title.py` | 46 passed, 7 skipped | 46 passed, 7 skipped
|
The failures are the same tests on both sides and come from my
environment (missing image/PDF extras, CRLF fixtures).
</details>
<details>
<summary>How token coverage was measured</summary>
The reference is each fixture's HTML (for `.eml`, the HTML body
`partition_email()` uses) parsed with lxml's HTML parser, with `<head>`,
`<script>`, `<style>`, `<noscript>`, `<template>` and `hidden`/inline
`display:none` elements removed, split into `\w+` tokens. Coverage is
the share of reference tokens, counted as a multiset, found in the
concatenated element text. It does not apply stylesheets, so it is an
approximation of what a browser shows.
No fixture gains output tokens that aren't in the reference, and I read
every newly emitted element. Every token still missing on this branch
was already missing on `main`: one-character page numbers (skipped by
design), text in a few table cells, and an antivirus footer the email
places after `</html>`.
</details>
### Limitations and not verified
- `ix:header` is matched by its conventional `ix` prefix, because
libxml2's HTML parser has no namespaces. A filing that binds the Inline
XBRL namespace to another prefix would have its hidden header emitted.
The repo's filing fixture uses `ix`.
- Other XML vocabularies embedded in HTML, such as Confluence `ac:`
macros, now contribute their text like any unrecognized element. I have
no such fixture.
- As with `<span>`, adjacent unknown elements with no whitespace between
them are joined: `<x-a>foo</x-a><x-b>bar</x-b>` gives `foobar`, which is
also how a browser renders them.
- The `azure/spring-weather.html` ingest fixture, whose source isn't in
the repo; `test_auto.py` doesn't collect in my environment.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4497?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->
