[Python-Dev] Fixing the XML batteries

Glyph Lefkowitz glyph at twistedmatrix.com
Sat Dec 10 22:32:46 CET 2011

On Dec 10, 2011, at 2:38 AM, Stefan Behnel wrote:

> Note, however, that html5lib is likely way too big to add it to the stdlib, and that BeautifulSoup lacks a parser for non-conforming HTML in Python 3, which would be the target release series for better HTML support. So, whatever library or API you would want to use for HTML processing is currently only the second question as long as Py3 lacks a real-world HTML parser in the stdlib, as well as a robust character detection mechanism. I don't think that can be fixed all that easily.

Here's the problem in a nutshell, I think:

Everybody wants an HTML parser in the stdlib, because it's inconvenient to pull in a dependency for such a "simple" task.
Everybody wants the stdlib to remain small, stable, and simple and not get "overcomplicated".
Parsing arbitrary HTML5 is a monstrously complex problem, for which there exist rapidly-evolving standards and libraries to deal with it.  Parsing 'the web' (which is rapidly growing to include stuff like SVG, MathML etc) is even harder.

My personal opinion is that HTML5Lib gets this problem almost completely right, and so it should be absorbed by the stdlib.  Trying to re-invent this from scratch, or even use something like BeautifulSoup which uses a bunch of heuristics and hacks rather than reference to the laboriously-crafted standard that says exactly how parsing malformed stuff has to go to be "like a browser", seems like it will just give the stdlib solution a reputation for working on the test input but not working in the real world.

(No disrespect to BeautifulSoup: it was a great attempt in the pre-HTML5 world which it was born into, and I've used it numerous times to implement useful things.  But much more effort has been poured into this problem since then, and the problems are better understood now.)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20111210/4f64ce51/attachment.html>

More information about the Python-Dev mailing list