Search: unify Sphinx and generic parser (#10520)
The current differences from the generic parser and the Sphinx parser are:
- The Sphinx parser doesn't need to find the main content node, since that's given by the `.fjson` file.
- The Sphinx parser removes nodes that have the following classes:
- .toctree-wrapper (toc from the `toctree` directive)
- .contents.local.topic (local toc from the `contents` directive)
How can we unify both?
- I think we handle finding the main node for Sphinx projects correctly with the generic parser.
- Removing elements that have the `.contents.local.topic` class is redundant, since that's inside a `nav` tag (we remove those).
- Elements generated by the `toctree` directive are plain divs with a `.toctree-wrapper` class, so we can't have a generic way of removing them. This is the only impediment for merging both parsers, we have two options
a) Just don't remove those elements, and index them, maybe suggest upstream to enclose them inside a nav tag.
The downside is that search results will have some duplicates (matching the toctree and the actual document), for example, here you can see two pages matching, the actual page and the index pages that has the ToC.
b) Just add that special case to our generic parser (we kind of already do this for `linenos`, `lineno`, and `headerlink`). Maybe suggest upstream to use a nav tag as well, so one day we may be able to stop especial casing that :D
I think I'm fine with option b.
