Mailman 3 April 2008 - lxml - The Python XML Toolkit

[lxml-dev] Faster parsing!
by Stefan Behnel 23 Apr '08

23 Apr '08

Hi, here is a (pretty ugly, hackish) patch against libxml2 2.6.32 that replaces the hash function of the internal hash table implementation by one that I found on the web: http://www.azillionmonkeys.com/qed/hash.html Remember that cElementTree is still the fastest XML tree parser for Python? By a factor of up to 10 compared to lxml? http://codespeak.net/lxml/performance.html#parsing and-serialising According to lxml's benchmark suite, this patch brings the parser of libxml2/lxml down to 2 times (10x->2x!) the parsing time of cElementTree for larger files (some MB). I find this quite impressive. Here are the numbers (lower is better): lxe: XML (SAXR T1) 39.4800 msec/pass # pretty large tree cET: XML (SAXR T1) 20.0679 msec/pass lxe: XML (SAXR T3) 25.9020 msec/pass cET: XML (SAXR T3) 33.2189 msec/pass lxe: XML (SAXR T4) 0.7598 msec/pass cET: XML (SAXR T4) 0.7181 msec/pass While the benchmark is not a particularly good measure for this exact case as it generates the XML tag names instead of sticking to a (likely smaller) fixed set of language tags, this gives me a factor of 7 (!) in performance improvement for in-memory parsing compared to an unpatched libxml2. I also reran the old testament benchmark for a more realistic benchmark scenario. The speedup there is up to 30%, not that bad either. And lxml's "parse()+iter()" implementation of that benchmark is now as fast as cET's "iterparse()" version. :) I would love to get some feedback from others who want to test this. Just patch your copy of libxml2 and let lxml run against it. I'm eager to hear some numbers to convince Daniel to get a cleaned up version of this patch into mainstream libxml2. Hope you like it, Stefan

2 2

[lxml-dev] (patch) Generating PDF documentation
by Marcin Kasperski 18 Apr '08

18 Apr '08

As this is my first post to this list, let me start from great thanks for this excellent library. Now to the subject. HTML documentation is nice for online reference, but when one starts learning the library, it is nice to have single printable document. I created quick&dirty patch which adds "make pdf" target to the lxml Makefile, generating latex files and then pdflatex-ing them to obtain the PDF book. The result is surely unpolished (some LaTeX presentation tuning could make sense, also the section structure should be imported from common file instead of being copied), but nevertheless may be useful. I diff-ed it against the current subversion tip (rev 53799), but it should apply to other versions too (the patch consists of small Makefile change and two new python files in doc subdirectory) Best regards. -- ---------------------------------------------------------------------- | Marcin Kasperski | If Staff, Scope and Schedule are all fixed, | http://mekk.waw.pl | managers will have no options, other than | | prayer. (Martin) ----------------------------------------------------------------------

2 1

Re: [lxml-dev] etree.parse hangs with a lot of parallel requests
by Dmitri Fedoruk 16 Apr '08

16 Apr '08

Hi! Speaking again about the issue with DTD loading, parsing etc. > >> It seems you only want to parse DTDs locally from disc, so setting > >> "no_network=True" (which is the default in lxml 2.0) should prevent > >> any accidental remote access. > > > > Eventually it turned out that I'm working fine without DTD. So, > > setting no_network = True and load_dtd = False really solved the > > problem. > > Hmm, do you really need to turn off DTD loading or is disabling network > access enough? I wouldn't expect loading the DTD from the disk cache to > take that much time (although, if you can live without it and time is > really critical, then it's obviously better to safe that bit of time > also). I was wrong - I do need DTD to resolve entities correctly. Sometimes I got the html   and things like these. My DTD included all the required entities, but it is referenced by URL. And the only way to deal with this enity is to load the DTD, isn't it? Which options do I have except of switching URL to a local path in SYSTEM definition? Setting up the DTD catalog on every machine that runs the application? The ideal option would be to tell the parser "load the given DTD from a given location(i.e. disk) and use it from now and on for parsing all incoming data", but is it possible? Cheers, Dmitri

2 1

Re: [lxml-dev] lxml 2.0.4 released
by Stefan Behnel 16 Apr '08

16 Apr '08

René 'Necoro' Neumann wrote: > Just me again :) - what's about the GTK-hang-bug? Sorry, that fix went missing. I just re-released a new source distro that has it. Sorry for the inconvenience. Stefan

3 6

[lxml-dev] lxml 2.1beta1 released
by Stefan Behnel 15 Apr '08

15 Apr '08

Hi all, lxml 2.1beta1 is on PyPI. This is the first beta release of the upcoming 2.1 release series. http://codespeak.net/lxml/dev/ http://pypi.python.org/pypi/lxml/2.1beta1 This release contains the bug fixes that were included in 2.0.4 as well as a couple of other important fixes over 2.1alpha1. It also brings some new features, some of which were long awaited - CDATA support was first requested in December 2005, according to the mailing list archive. Another long awaited feature, XSLT extension element support, was already released in 2.1alpha1. Feedback on both will be appreciated. I hope you like it. Have fun, Stefan 2.1beta1 (2008-04-15) Features added * Error logging in Schematron (requires libxml2 2.6.32 or later). * Parser option strip_cdata for normalising or keeping CDATA sections. Defaults to True as before, thus replacing CDATA sections by their text content. * CDATA() factory to wrap string content as CDATA section. Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31. * lxml.etree accepted non well-formed namespace prefix names. Other changes * Major cleanup in internal moveNodeToDocument() function, which takes care of namespace cleanup when moving elements between different namespace contexts. * New Elements created through the makeelement() method of an HTML parser or through lxml.html now end up in a new HTML document (doctype HTML 4.01 Transitional) instead of a generic XML document. This mostly impacts the serialisation and the availability of a DTD context. 2.1alpha1 (2008-03-27) Features added * New event types 'comment' and 'pi' in iterparse(). * XSLTAccessControl instances have a property options that returns a dict of access configuration options. * Constant instances DENY_ALL and DENY_WRITE on XSLTAccessControl class. * Extension elements for XSLT (experimental!) * Element.base property returns the xml:base or HTML base URL of an Element. * docinfo.URL property is writable. Bugs fixed * Default encoding for plain text serialisation was different from that of XML serialisation (UTF-8 instead of ASCII). Other changes * Minor API speed-ups. * The benchmark suite now uses tail text in the trees, which makes the absolute numbers incomparable to previous results. * Generating the HTML documentation now requires Pygments, which is used to enable syntax highlighting for the doctest examples. Most long-time deprecated functions and methods were removed: * etree.clearErrorLog(), use etree.clear_error_log() * etree.useGlobalPythonLog(), use etree.use_global_python_log() * etree.ElementClassLookup.setFallback(), use etree.ElementClassLookup.set_fallback() * etree.getDefaultParser(), use etree.get_default_parser() * etree.setDefaultParser(), use etree.set_default_parser() * etree.setElementClassLookup(), use etree.set_element_class_lookup() Note that parser.setElementClassLookup() has not been removed yet, although parser.set_element_class_lookup() should be used instead. * xpath_evaluator.registerNamespace(), use xpath_evaluator.register_namespace() * xpath_evaluator.registerNamespaces(), use xpath_evaluator.register_namespaces() * objectify.setPytypeAttributeTag, use objectify.set_pytype_attribute_tag * objectify.setDefaultParser(), use objectify.set_default_parser()

1 0

[lxml-dev] Can't load external DTD.
by Nef Asus 15 Apr '08

15 Apr '08

Hello everyone, I've written this little program that refuses to work: from lxml import etree if __name__ == "__main__": xml_input = "C:\Desarrollo\pythontests\lxml\foo.xml" parser = etree.XMLParser(load_dtd = True, dtd_validation = True, attribute_defaults = True) doc = etree.parse(xml_input, parser) Here's the traceback. Traceback (most recent call last): File "C:\Desarrollo\pythontests\lxml\dtd_loader.py", \ line 27, in <module> doc = etree.parse(xml_input, parser) File "lxml.etree.pyx", line 2515, in lxml.etree.parse File "parser.pxi", line 1755, in lxml.etree._parseDocument File "parser.pxi", line 1759, in lxml.etree._parseDocumentFromURL File "parser.pxi", line 1681, in lxml.etree._parseDocFromFile File "parser.pxi", line 826 ,in lxml.etree._BaseParser._parseDocFromFile File "parser.pxi",line 450,in lxml.etree._ParserContext._handleParseResultDoc File "parser.pxi", line 534, in lxml.etree._handleParseResult File "parser.pxi", line 476, in lxml.etree._raiseParseError lxml.etree.XMLSyntaxError: failed to load external entity "NULL", line 9, column 83 This is a snippet of foo.xml : <?xml version="1.0" encoding="iso-8859-1" ?> <!DOCTYPE rem:requirementsProject SYSTEM "C:\Desarrollo\pythontests\lxml\foo.dtd"> ... Then, I tried to write a custom resolver. from lxml import etree class DTDResolver(etree.Resolver): def resolve(self, url, id, context): print("Resolving (url, %s)(id, %s)"% (url,id)) self.resolve_filename("C:\Desarrollo\pythontests\lxml\JENSEN.dtd", \ context) if __name__ == "__main__": parser = etree.XMLParser(load_dtd = True, dtd_validation = True, attribute_defaults = True) parser.resolvers.add(DTDResolver()) xml_input = "C:\Desarrollo\pythontests\lxml\foo.xml" doc = etree.parse(xml_input, parser) But it still fails (same traceback). What am I doing wrong? BTW, lxml version is 2.0.1. Thanks in advance.

4 7

[lxml-dev] trunk build problems + cython version
by Holger Joukl 15 Apr '08

15 Apr '08

Hi, I am trying to build lxml svn trunk to check out the new objectify type conversions and have some difficulty: $ PYTHONPATH=/data/pydev/DOWNLOADS/LXML/cython/versions/Cython-0.9.6.11b/build/lib/ /apps/pydev/bin/python2.4 setup.py build Building lxml version 2.1.alpha1-53751. Building with Cython 0.9.6.11b. Using build configuration of libxslt 1.1.20 Building against libxml2/libxslt in one of the following directories: /apps/prod//lib /data/pydev/DOWNLOADS/LXML/libxml2/libxml2-2.6.27 running build running build_py writing byte-compilation script '/tmp/tmpVmrBsO.py' /apps/pydev/bin/python2.4 -O /tmp/tmpVmrBsO.py removing /tmp/tmpVmrBsO.py running build_ext cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c building 'lxml.etree' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/apps/prod//include -I/apps/prod//include/libxml2 -I/apps/prod/include/libxml2 -I/apps/prod/include -I/apps/pydev/include/python2.4 -c src/lxml/lxml.etree.c -o build/temp.solaris-2.8-sun4u-2.4/src/lxml/lxml.etree.o -w gcc -shared build/temp.solaris-2.8-sun4u-2.4/src/lxml/lxml.etree.o -L/apps/prod//lib -L/data/pydev/DOWNLOADS/LXML/libxml2/libxml2-2.6.27 -L/apps/prod/lib -Wl,-R/apps/prod/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.solaris-2.8-sun4u-2.4/lxml/etree.so cythoning src/lxml/lxml.objectify.pyx to src/lxml/lxml.objectify.c Error converting Pyrex file to C: ------------------------------------------------------------ ... if s is not None: value = __parseBoolAsInt(s) if value == -1: raise ValueError cpdef __parseBool(s): ^ ------------------------------------------------------------ /data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TRUNK/lxml/src/lxml/lxml.objectify.pyx:842:6: Overridable cdef function not allowed here building 'lxml.objectify' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/apps/prod//include -I/apps/prod//include/libxml2 -I/apps/prod/include/libxml2 -I/apps/prod/include -I/apps/pydev/include/python2.4 -c src/lxml/lxml.objectify.c -o build/temp.solaris-2.8-sun4u-2.4/src/lxml/lxml.objectify.o -w gcc: src/lxml/lxml.objectify.c: No such file or directory gcc: No input files error: command 'gcc' failed with exit status 1 Is the Cython version I'm using up to par for this? The dev docs still indicate so: "lxml currently requires Cython 0.9.6.11b or 0.9.6.12, later versions were not tested. " I also tried to build with the latest Cython, but this fails due to generated code not working with my oldtimer compiler (filed a Cython bug report for it). Holger -- Psst! Geheimtipp: Online Games kostenlos spielen bei den GMX Free Games! http://games.entertainment.gmx.net/de/entertainment/games/free

3 4

Re: [lxml-dev] Weird errors in tostring
by Stefan Behnel 15 Apr '08

15 Apr '08

Hi, Bruno Barberi Gnecco wrote: > Hi Stephan, -f- >>> In the other machine all goes well. FYI, the tree (root variable) is >>> being built with root = lxml.html.fromstring(data). I'm parsing data >>> in utf8 and >>> iso-8859-1, and this particular backtrace happened in a HTML document >>> correctly labelled with a meta charset=iso-8859-1. >> >> >> You can ask the document which encoding it was parsed with: >> >> >>> print root.getroottree().docinfo.encoding >> >> It should say "iso-8859-1" if the parser picked up the <meta> tag >> correctly. > > It says 'None', actually. Then that's a clear sign that libxml2 didn't pick up the encoding. > Shouldn't it give the error when *parsing* and creating the tree, > instead of when converting the tree to something else? HTML is parsed with the "recover" option, which lets libxml2 try to work around all sorts of broken page content *without* raising an error. You can still check the error log of the parser to see what happend on the way through the page. > I thought lxml stored the parsed tree in unicode. UTF-8, actually, which is much easier (and faster) to handle in C than any other unicode encoding. > Besides, I'm asking for a unicode string: > > tostring(root, method='xml', encoding=unicode) Which lets lxml serialise the tree to a Python unicode character sequence in XML style. I know, this looks simple, but there's actually work being done here. >> Also, maybe the <meta> tag comes behind the <title> in the document? >> AFAIR, >> libxml2's HTML parser switches encodings when it sees a <meta> >> declaration, >> but it doesn't reparse the document (as most browsers do to work >> around this >> problem). > > It happens with fragments of HTML as well (I'm actually reading HTML > messages). Yet I was having this problem with pages download from the > internet, in which the encoding was incorrectly detected. Which implies most of the time that it was incorrectly specified as well. That is a very common problem in real world HTML pages. Browsers do a great deal of work in their Quirks mode to figure out the page encoding. libxml2's HTML parser works pretty well, but fortune telling wasn't one of its design goals. > Since I had more information > in that case (HTTP headers, with a chardet pass just to be sure) I ended up > forcing the encoding with a 'html.decode(encoding)' step before building > the tree. I think it's weird that it works (since some pages declare one > encoding and use a different one), but it does. You might want to strip <meta> Content Type tags from the string using a regex, that should make sure it works in all cases. Read the function "htmlCheckEncoding()" in libxml2's HTMLparser.c to see what works and what doesn't. For example, there is some code to prevent changing the parser encoding a second time, so that you can override it with the "encoding" parser keyword in lxml. >> If the parser gets the encoding wrong, you can try parsing with >> BeautifulSoup >> (separate install) by using the fromstring() function in >> lxml.html.ElementSoup >> instead. That's quite a bit slower, but it *might* give you better >> results in this case. I wrote a little doc section on that topic: http://codespeak.net/lxml/elementsoup.html#using-soupparser-as-a-fallback > First, why does it work in one of the machines and not in the other, > even with the same data? I installed Python2.5, but with the same results. > Maybe the cause is libxml2 (2.6.30 where it works, 2.6.26 where it > doesn't)? That's almost definitely the reason, yes. > Second, if the tree is created, how to know if the encoding is > wrong? I only convert to string much later. You can serialise immediately, just for testing, that will tell you. Or, you can check the parser error log for encoding errors. Stefan

1 0

[lxml-dev] lxml 2.0.4 released
by Stefan Behnel 13 Apr '08

13 Apr '08

Hi all, lxml 2.0.4 is on PyPI. This is a bug-fix release for the stable 2.0 series. Have fun, Stefan 2.0.4 (2008-04-13) Bugs fixed * Crash bug in iterparse when moving elements into other documents. * HTML elements' .cssselect() method was broken. * ElementTree.find*() didn't accept QName objects.

1 0

[lxml-dev] Weird errors in tostring
by Bruno 13 Apr '08

13 Apr '08

Hi, I'm getting a weird error in lxml.html.tostring; it happens in one machine but not in another, although both are using lxml 2.0.2, but one has python 2.5 (which works all the time) and the other python 2.4 (which doesn't). Here's the relevant backtrace: File "/home/spyder/spyder/core/base.py", line 289, in treetostring return tostring(root, method='xml', encoding=unicode) File "/usr/lib/python2.4/site-packages/lxml-2.0.2-py2.4-linux-i686.egg/lxml/html/ __init__.py", line 1313, in tostring encoding=encoding) File "lxml.etree.pyx", line 2455, in lxml.etree.tostring File "serializer.pxi", line 61, in lxml.etree._tostring File "serializer.pxi", line 126, in lxml.etree._tounicode UnicodeDecodeError: 'utf8' codec can't decode bytes in position 21-24: invalid data In the other machine all goes well. FYI, the tree (root variable) is being built with root = lxml.html.fromstring(data). I'm parsing data in utf8 and iso-8859-1, and this particular backtrace happened in a HTML document correctly labelled with a meta charset=iso-8859-1. If you have any ideas of how to trace what is going wrong?

2 1