Mailman 3 May 2008 - lxml - The Python XML Toolkit

[lxml-dev] Error reporting not clear
by F Wolff 06 May '08

06 May '08

Hallo Stefan and other lxml people. I had a bug report which I traced to an invalid XML file. The error message given by the parser was however not optimally useful. The file is available here (zipped): http://bugs.locamotion.org/attachment.cgi?id=132 and a description of the problem here: http://bugs.locamotion.org/show_bug.cgi?id=384 It might or might not be interesting to improve this error reporting, so I thought I'll mention it. Keep well Friedel

2 1

Re: [lxml-dev] lxml - addition of argument to control namespace declaration serializtion
by Stefan Behnel 05 May '08

05 May '08

Hi, these things are best discussed on the list. Grimes, David wrote: > In 1.3.5-1.3.6 timeframe, there was a patch introduced to > _writeNodeToBuffer() from serialization.pxi which forced namespace > declarations from parent nodes to be serialized onto the sub-tree root > node. In general, and with respect to XML standards, this make a great > deal of sense (so you don't have prefixed elements/attributes without the > corresponding namespace declaration). > > But, the application I've been building essentially takes an XML document > and makes template-string blocks of text out of various sub-trees, to be > later combined back into a full document using __getitem__ substitution in > the form of "%(token)s" string formatting. > > The nsdecl patch of 1.3.5/6 causes interesting behaviour when the sub-tree > being rendered is done in, for example, a loop - one "formatting > operation" per iteration. Also interesting is when many such sub-trees > are combined to form a document which (in my case) we know will have the > declarations on the ultimate root node. What you mean is that we actually make a copy of a non-root node and then copy over the namespace declarations of the ancestors. You say "interesting behaviour". Does that refer to the performance overhead or is there a 'real' problem you see? Looking at the code now, I see some potential optimisations, so if it's just the performance, here's a (trunk) patch that should give a bit of relief. > So ... I've got a patch I'm using in my local build which adds a keyword > argument "nsdecl=True" to tostring(), tounicode() and tofilelike() - these > are all the places which make use of the _writeNodeToBuffer() machinery. > I can spin the patch against any 2.0.x or 2.1.x source tree. > > The argument defaults to True, to maintain backward compatibility, but can > be provided as False to get <= 1.3.4 behaviour. > > Would you consider accepting this patch? At first glance: no. I do not think there is general interest for a serialisation that is not ns well-formed. You seem to have a rather special use case here. I'm not even sure you have to do what you describe based on serialised XML fragments. You might be able to do something like that with subtrees. But that's very close to guessing. Stefan

1 0

[lxml-dev] [Fwd: Re: (no subject)]
by Stefan Behnel 04 May '08

04 May '08

[Forwarding to the list...] From: <mharper3(a)uiuc.edu> Also, adding elem.clear() into the loop still eventually leads to a memory error, just much later. This should be clearing every element, so I'm not quite sure if I understand what clear() actually does. Should I segment the file into smaller pieces so that the tree is unloaded as each piece finishes? I apologize if my questions are trivial. I appreciate your responses greatly. -- Marc

1 1

[lxml-dev] [Fwd: Re: (no subject)]
by Stefan Behnel 04 May '08

04 May '08

[Forwarding to the list ...] From: <mharper3(a)uiuc.edu> Stefan -- Thanks so much for the quick response. I did consider that the tree was being built in memory, but the documentation seems to suggest that is not the case. Specifically the language in the tutorial (http://codespeak.net/lxml/tutorial.html) in both the sections 'incremental parsing' and 'event-driven parsing' seem to suggest using iterparse to access without retaining the tree in memory. I see now that the documentation says otherwise for iterparse, as you pointed out. If you don't mind, why does the iterator retain the tree in memory? I would suspect otherwise from the 'natural' behavior of iterators/generators in general, though that may be an invalid assumption. (i.e. I would parse the entire tree into memory if I thought that I had enough memory to do so; otherwise I would _incrementally_ parse it.) More specifically, I don't want to ignore any parts of the xml file in this specific instance, so a ParserTarget is not the correct solution. Your suggestion to use clear() works for me; maybe it should be made explicit in the tutorial that memory is not cleared unless clear() is called. The only mention in the tutorial is iterparse "also allows to clear() or modify the content of an Element to save memory". My mistake was to assume that the 'used' elements would be freed without an explicit call to do so as the iterator progressed. Again, thank you for your quick reply! -- Marc

1 1

[lxml-dev] (no subject)
by mharper3＠uiuc.edu 04 May '08

04 May '08

Hi lxml-dev: I'm getting glibc/MemoryError/cStringIO crashes/exceptions from the following (minimal reproduction) code: <code> import lxml.etree wiki_xml_filename = 'enwiki-latest-pages-articles.xml' # from http://download.wikimedia.org/enwiki/latest/ context = lxml.etree.iterparse(wiki_xml_filename, events=("end")) for action, elem in context: pass </code> The crash usually occurs about halfway through the file (around <page> 3,000,000) The same code runs on smaller mediawiki xml files (200 mb) without error. I only get this error for this very large xml file (in this case about 13gb uncompressed). I had no trouble parsing the same file with the python standard library sax parser, but it is much slower and I don't like its api. I'm using libxml2-2.6.32 (also used earlier versions), python 2.5.2, python-lxml 2.0.5 (also tried earlier versions), Kubuntu 8.04 with 2.6.24 kernel (also tested on opensuse 10.3 with earlier kernel). Some of the exceptions are MemoryErrors. The machine running the code has 4gb of ram. The kernel does not appear to significantly hit the swap during the run. Here are the errors: ** glibc detected *** python: free(): invalid pointer: 0x08220a15 *** Aborted Also: Traceback (most recent call last): File "minimal.py", line 6, in <module> for action, elem in context: File "iterparse.pxi", line 390, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:65064) File "parser.pxi", line 489, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:47432) lxml.etree.XMLSyntaxError: None Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 37, in apport_excepthook import re, tempfile, traceback File "/usr/lib/python2.5/traceback.py", line 241, in <module> def print_last(limit=None, file=None): MemoryError Original exception was: Traceback (most recent call last): File "minimal.py", line 6, in <module> for action, elem in context: File "iterparse.pxi", line 390, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:65064) File "parser.pxi", line 489, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:47432) lxml.etree.XMLSyntaxError: None ... and also (slightly different) Traceback (most recent call last): File "minimal.py", line 6, in <module> for action, elem in context: File "iterparse.pxi", line 390, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:65064) File "parser.pxi", line 489, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:47432) lxml.etree.XMLSyntaxError: None Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 37, in apport_excepthook import re, tempfile, traceback File "/usr/lib/python2.5/tempfile.py", line 33, in <module> from random import Random as _Random MemoryError Original exception was: Traceback (most recent call last): File "minimal.py", line 6, in <module> for action, elem in context: File "iterparse.pxi", line 390, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:65064) File "parser.pxi", line 489, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:47432) lxml.etree.XMLSyntaxError: None Sometimes I just get 'Segmentation fault' from the shell, and sometimes it just hangs indefinitely. and finally (cStringIO): Traceback (most recent call last): File "minimal.py", line 6, in <module> for action, elem in context: File "iterparse.pxi", line 390, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:65064) File "parser.pxi", line 489, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:47432) lxml.etree.XMLSyntaxError: None Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 36, in apport_excepthook from cStringIO import StringIO ImportError: /usr/lib/python2.5/lib-dynload/cStringIO.so: failed to map segment from shared object: Permission denied Original exception was: Traceback (most recent call last): File "minimal.py", line 6, in <module> for action, elem in context: File "iterparse.pxi", line 390, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:65064) File "parser.pxi", line 489, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:47432) lxml.etree.XMLSyntaxError: None Any direction on tracking down the source is greatly appreciated! -- Marc

2 1

[lxml-dev] lxml 2.1beta2 released
by Stefan Behnel 02 May '08

02 May '08

Hi all, I'm happy to announce the release of lxml 2.1 beta2. It features a couple of enhancements and fixes over the first beta. The main improvement is the much more robust threading support, which makes it a lot easier to move subtrees back and forth between threads. It is described in more detail here: http://permalink.gmane.org/gmane.comp.python.lxml.devel/3571 Please report back on the list (preferably in reply to the above thread) if you notice a difference to lxml 2.0 with your code. Have fun, Stefan 2.1beta2 (2008-05-02) Features added * All parse functions in lxml.html take a parser keyword argument. * lxml.html has a new parser class XHTMLParser and a module attribute xhtml_parser that provide XML parsers that are pre-configured for the lxml.html package. Bugs fixed * Moving a subtree from a document created in one thread into a document of another thread could crash when the rest of the source document is deleted while the subtree is still in use. * Passing an nsmap when creating an Element will no longer strip redundantly defined namespace URIs. This prevented the definition of more than one prefix for a namespace on the same Element. Other changes * If the default namespace is redundantly defined with a prefix on the same Element, the prefix will now be preferred for subelements and attributes. This allows users to work around a problem in libxml2 where attributes from the default namespace could serialise without a prefix even when they appear on an Element with a different namespace (i.e. they would end up in the wrong namespace).

1 0

[lxml-dev] Custom Elements question
by Alex Klizhentas 02 May '08

02 May '08

Hi All, Got a question: I've extended the ElementBase object using the approach described in the tutorial, but SubElement does not work as desired: class NodeBase(etree.ElementBase): def append(self,child): print "aaa" return etree.ElementBase.append(self,child) etree.SubElement(root,"child") #no "aaa" printed OK, but when taking your code to the module: def SubElement(parent, tag, attrib={}, **extra): attrib = attrib.copy() attrib.update(extra) element = parent.makeelement(tag, attrib) parent.append(element) return element SubElement(root,"child") # "aaa" is here! and overriding def makeelement(self, tag, attrib): return Node(tag, attrib) in the NodeBase just does not help, Any advice will be appreciated, Alex

2 6

[lxml-dev] threading fixed :)
by Stefan Behnel 02 May '08

02 May '08

Hi, there has been a long-standing issue in the threading support in lxml, combined with the per-thread string hash table we use for libxml2. Here is a simple example of a sure crasher: ------------------------------- import threading import lxml.etree as et xml = "<root><threadtag/></root>" main_root = et.XML("<root/>") def run_thread(): thread_root = et.XML(xml) main_root.append(thread_root[0]) del thread_root # deletes the document thread = threading.Thread(target=run_thread) thread.start() thread.join() print et.tostring(main_root) ------------------------------- This crashes, because the thread parses the XML fragment into its own dictionary and stores the tag name "threadtag" there. Then it appends the "threadtag" element to a tree in the main program, which uses a different dict. When it deletes the "thread_root", the document will be deleted as well, and the (ref-counted) thread dictionary that contains the string "threadtag" will be freed when the thread terminates. The main program then crashes when it accesses the no longer available tag name in the corrupted document. The solution I came up with today is actually quite simple. We have to traverse the subtree anyway to update the document references and to fix the namespace declarations. So it's only one step more to also fix the name pointers by looking them up in the target dictionary and re-assigning the names. This is only required when we really have two different dicts, which is easy to decide. So there isn't even a performance impact if you only use a single thread or if you do not move subtrees between threads. And the added overhead when you need this is really small. I will release a new beta of 2.1 soon that will have this change, and it would be very helpful if people who currently use threaded code that exchanges (i.e. deep copies) tree fragments between threads could check if this works for them (i.e. if code that crashes under 2.0 if you remove the deep copying works under 2.1). If it proves to fix the problem, I will backport it to 2.0 also. Read: the more feedback I get, the faster this will be fixed in 2.0. :) Stefan

1 0