Clif Swiggett, 16.11.2009 17:29:

Thanks for the tips to tighten up my code. Do you have any suggestions for how best to detect whether a source file is html or xml? I'm harvesting arbitrary files off the web ...

You can either pass it into the XML parser and fall back to the HTML parser if that fails (which it most likely will for the large majority of HTML files), or, if you expect more HTML pages than XML pages, you can read the page into a string and search for the XHTML namespace before passing it into the parser. If you want to parse arbitrary web pages, you will likely end up using some kind of parser cascade anyway to make sure you get the best conformance and speed in the normal cases and the best error recovery in the pathological cases. Remember that lxml's parser is very fast, so it's ok to parse things multiple times with different parser configurations. Stefan