Mailman 3 April 2010 - lxml - The Python XML Toolkit

[lxml-dev] getting € instead of € from tostring
by Massa, Harald Armin April 22, 2010

April 22, 2010

import lxml.html a=lxml.html.fragment_fromstring("<p>we need 100 € yeah</p>") print lxml.html.tostring(a) <p>we need 100 € yeah</p> What can I do to get "named entitites", as € ä ö etc? (Would even be willing to enumerate and pass the 15 or so I really need) Those are much better human-readable in the unlikely case of debugging. best wishes, Harald -- GHUM Harald Massa persuadere et programmare Harald Armin Massa Spielberger Straße 49 70435 Stuttgart 0173/9409607 no fx, no carrier pigeon - %s is too gigantic of an industry to bend to the whims of reality

3 4

[lxml-dev] bug in HTMLParser with broken html
by Sergio Monteiro Basto April 15, 2010

April 15, 2010

Hi, I found other problem with lxml.html In [1]: from lxml import html In [2]: html.tostring(html.fromstring("""<table> <tr><td><font size=1><a class=menu href="1125.pdf"> 1125. <tr><td><font size=1><a class=menu href="1124.pdf"> 1124. </table>""")) Out[2]: '<table><tr><td><font size="1"><a class="menu" href="1125.pdf"> 1125. <tr><td><font size="1"><a class="menu" href="1124.pdf"> 1124. </a></font></td></tr></a></font></td></tr></table>' the <a href under <a href ?! and <tr><td> <tr><td> </td></tr> </td></tr> ?! firebug result : <table> <tbody><tr><td><font size="1"><a href="1125.pdf" class="menu"> 1125. </a></font></td></tr> <tr><td><font size="1"><a href="1124.pdf" class="menu"> 1124. </a></font></td></tr>< /tbody></table> xmllint --html file also done this wrongly ! Thanks, -- Sérgio M. B.

2 3

[lxml-dev] iterparse segfaults misteriously
by Lawrence Oluyede April 13, 2010

April 13, 2010

Yesterday my colleague and I noticed a series of apparently random deaths of apache processes and we found out it was lxml's fault. We slimmed down the use case to the following lines of code: xml = "<root><el>Test</el></root>" open("/tmp/test.xml", "w").write(xml) from lxml import etree list(etree.iterparse(open("/tmp/test.xml"), events=("start", "end"))) This crashes badly on one of our staging machines but not at all on our development computers. We thought it was due to a slight difference in libxml2/lxml versions but we tried all the configurations and it keeps on crashing with the following exception: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "iterparse.pxi", line 471, in lxml.etree.iterparse.__next__ (src/lxml/lxml.etree.c:91652) TypeError: 'str' object doesn't support slice deletion (we had "TypeError: 'NoneType' object doesn't support slice deletion" on the actual code but the only difference is that it's using StringIO instead of a file) We debugged postmortem and this is what GDB is saying: #0 __pyx_pf_4lxml_5etree_9iterparse___next__ (__pyx_v_self=0x2b78ff1af1d0) at src/lxml/lxml.etree.c:86607 #1 0x00002b78fa892dec in listextend (self=0x2b78ffa15518, b=<value optimized out>) at Objects/listobject.c:823 #2 0x00002b78fa89336a in list_init (self=0x2b78ffa15518, args=<value optimized out>, kw=<value optimized out>) at Objects/listobject.c:2391 #3 0x00002b78fa8b2338 in type_call (type=<value optimized out>, args=0x2b78ffa233d0, kwds=0x0) at Objects/typeobject.c:436 #4 0x00002b78fa869df3 in PyObject_Call (func=0x0, arg=0x307194faf0, kw=0x0) at Objects/abstract.c:1861 #5 0x00002b78fa8e701a in PyEval_EvalFrameEx (f=0x110a2f10, throwflag=<value optimized out>) at Python/ceval.c:3823 #6 0x00002b78fa8ead0b in PyEval_EvalCodeEx (co=0x2b78fac1e0a8, globals=<value optimized out>, locals=<value optimized out>, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2875 #7 0x00002b78fa8ead52 in PyEval_EvalCode (co=0x0, globals=0x307194faf0, locals=0x0) at Python/ceval.c:514 #8 0x00002b78fa90d991 in PyRun_FileExFlags (fp=0x11031010, filename=0x7fffb027d7ee "test_lxml.py", start=<value optimized out>, globals=0x11054490, locals=0x11054490, closeit=1, flags=0x7fffb027cad0) at Python/pythonrun.c:1273 #9 0x00002b78fa90dc18 in PyRun_SimpleFileExFlags (fp=<value optimized out>, filename=0x7fffb027d7ee "test_lxml.py", closeit=1, flags=0x7fffb027cad0) at Python/pythonrun.c:879 #10 0x00002b78fa9167f9 in Py_Main (argc=1, argv=<value optimized out>) at Modules/main.c:532 #11 0x000000307161d8b4 in __libc_start_main () from /lib64/libc.so.6 #12 0x0000000000400629 in _start () Any ideas? We tried with lxml 2.2.0, lxml 2.2.4 and lxml 2.2.6 against libxml2 2.7.2 and libxml2 2.7.7 and libxslt 1.1.4 and libxslt 1.1.26 Python version is 2.5.4 on a Centos 5 64bit -- Lawrence Oluyede [eng] http://oluyede.org - http://twitter.com/lawrenceoluyede [ita] http://www.neropercaso.it [flickr] http://www.flickr.com/photos/rhymes

2 5

[lxml-dev] Bug with HTMLParser()
by Benoit Boissinot April 13, 2010

April 13, 2010

The HTMLParser() seems to strip any whitespace after <sup></sup>, for example: http://paste.pocoo.org/show/201052/ In [25]: etree.tostring(etree.fromstring('<html><body><b>Article 1<sup>er</sup> <i>bis</i> <i>(nouveau)</i></b></body></html>', etree.HTMLParser())) Out[25]: '<html><body><b>Article 1<sup>er</sup><i>bis</i> <i>(nouveau)</i></b></body></html>' In [26]: etree.tostring(etree.fromstring('<html><body><b>Article 1<sup>er</sup> <i>bis</i> <i>(nouveau)</i></b></body></html>')) Out[26]: '<html><body><b>Article 1<sup>er</sup> <i>bis</i> <i>(nouveau)</i></b></body></html>' Is it a bug with lxml? or with libxml2? regards, Benoit

2 2

[lxml-dev] lxml crashes parsing anything on SuSE Linux Enterprise Server 11
by Martin Aspeli April 8, 2010

April 8, 2010

Hi, I'm trying to get lxml to work on SLES11 (x86_64). Much pain and suffering has been had so far to come to the point where lxml is most likely the culprit of all kinds of random crashes. I'm using the system-installed Python 2.6. I'd prefer a non-system Python, but it's nearly impossible to compile one from source that also has ssl and hashlib support, so I've given up. I've got two test scenarios that both produce the same result: - System Python 2.6 using an rpm-installed lxml - Same Python in a virtualenv --no-site-packages with lxml installed via easy_install The system packages we have installed are: $ zypper search libxml2 libxslt libexslt lxml Loading repository data... Reading installed packages... S | Name | Summary | Type --+------------------+----------------------------------------------+----------- | libxml2 | A Library to Manipulate XML Files | srcpackage i | libxml2 | A Library to Manipulate XML Files | package i | libxml2-32bit | A Library to Manipulate XML Files | package i | libxml2-devel | Include Files and Libraries mandatory for -> | package | libxml2-doc | A Library to Manipulate XML Files | package i | libxml2-python | Python Bindings for libxml2 | package | libxml2-python | Python Bindings for libxml2 | srcpackage | libxslt | XSL Transformation Library | srcpackage i | libxslt | XSL Transformation Library | package i | libxslt-32bit | XSL Transformation Library | package i | libxslt-devel | Include Files and Libraries mandatory for -> | package | libxslt-python | Python Bindings for libxslt | package | libxslt-python | Python Bindings for libxslt | srcpackage | perl-XML-LibXSLT | XML::LibXSLT Perl Module | package | perl-XML-LibXSLT | XML::LibXSLT Perl Module | srcpackage i | python-lxml | A Pythonic Binding for the libxml2 and lib-> | package | python-lxml | A Pythonic Binding for the libxml2 and lib-> | srcpackage | python-lxml-doc | Documentation for python-lxml Package | package i | slessp0-libxml2 | Security update for libxml2 | patch The ones with' i' are installed. When trying to parse anything, the result is a segfualt. gdb says: $ gdb ./bin/python GNU gdb (GDB; SUSE Linux Enterprise 11) 6.8.50.20081120-cvs Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". For bug reporting instructions, please see: <http://bugs.opensuse.org/>... (no debugging symbols found) (gdb) run Starting program: /home/osc/tmp/lxml-env/bin/python (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [Thread debugging using libthread_db enabled] (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) Python 2.6 (r26:66714, Feb 21 2009, 02:16:04) [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2 Type "help", "copyright", "credits" or "license" for more information. (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) >>> import lxml.etree >>> lxml.etree.parse('<root />') Program received signal SIGSEGV, Segmentation fault. 0x00007ffff6ef2ab9 in strncmp () from /lib64/libc.so.6 (gdb) bt #0 0x00007ffff6ef2ab9 in strncmp () from /lib64/libc.so.6 #1 0x00007ffff5b04757 in __xmlParserInputBufferCreateFilename () from /usr/lib64/libxml2.so.2 #2 0x00007ffff5b4bd66 in xmlParseCatalogFile () from /usr/lib64/libxml2.so.2 #3 0x00007ffff5b4bff3 in ?? () from /usr/lib64/libxml2.so.2 #4 0x00007ffff5b4c51d in ?? () from /usr/lib64/libxml2.so.2 #5 0x00007ffff5b4d0bf in xmlACatalogResolve () from /usr/lib64/libxml2.so.2 #6 0x00007ffff5b04b43 in ?? () from /usr/lib64/libxml2.so.2 #7 0x00007ffff5b055cf in ?? () from /usr/lib64/libxml2.so.2 #8 0x00007ffff62e557c in __pyx_f_4lxml_5etree__local_resolver ( __pyx_v_c_url=0x806670 "<root />", __pyx_v_c_pubid=0x0, __pyx_v_c_context=0x80c350) at src/lxml/lxml.etree.c:63618 #9 0x00007ffff5b04bef in xmlLoadExternalEntity () from /usr/lib64/libxml2.so.2 #10 0x00007ffff5af30df in xmlCtxtReadFile () from /usr/lib64/libxml2.so.2 #11 0x00007ffff62e85d2 in __pyx_f_4lxml_5etree_11_BaseParser__parseDocFromFile (__pyx_v_self=0x7ffff7e695a8, __pyx_v_c_filename=0x7ffff7f29744 "<root />") at src/lxml/lxml.etree.c:68146 #12 0x00007ffff627d6cb in __pyx_f_4lxml_5etree__parseDocFromFile ( __pyx_v_filename8=0x7ffff7f29720, __pyx_v_parser=0x7ffff7e695a8) at src/lxml/lxml.etree.c:71175 #13 0x00007ffff627d9d9 in __pyx_f_4lxml_5etree__parseDocumentFromURL ( __pyx_v_url=0x7fffffffdc70, __pyx_v_parser=0x700001c4f) at src/lxml/lxml.etree.c:72080 ---Type <return> to continue, or q <return> to quit--- #14 0x00007ffff6311b78 in __pyx_f_4lxml_5etree__parseDocument ( __pyx_v_source=0x7ffff7f29720, __pyx_v_parser=0x7ffff7dac470, __pyx_v_base_url=0x7ffff7dac470) at src/lxml/lxml.etree.c:71797 #15 0x00007ffff6313656 in __pyx_pf_4lxml_5etree_parse ( __pyx_self=<value optimized out>, __pyx_args=<value optimized out>, __pyx_kwds=<value optimized out>) at src/lxml/lxml.etree.c:49958 #16 0x00007ffff7b15f28 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.6.so.1.0 #17 0x00007ffff7b1b6c0 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.6.so.1.0 #18 0x00007ffff7b13822 in PyEval_EvalCode () from /usr/lib64/libpython2.6.so.1.0 #19 0x00007ffff7b34b13 in ?? () from /usr/lib64/libpython2.6.so.1.0 #20 0x00007ffff7b3711c in PyRun_InteractiveOneFlags () from /usr/lib64/libpython2.6.so.1.0 #21 0x00007ffff7b372d6 in PyRun_InteractiveLoopFlags () from /usr/lib64/libpython2.6.so.1.0 #22 0x00007ffff7b357b6 in PyRun_AnyFileExFlags () from /usr/lib64/libpython2.6.so.1.0 #23 0x00007ffff7b40dba in Py_Main () from /usr/lib64/libpython2.6.so.1.0 #24 0x00007ffff6e94586 in __libc_start_main () from /lib64/libc.so.6 #25 0x00000000004006e9 in _start () (gdb) I've tried to compile a custom libxml2, libxslt and zlib and build against those, but not had any luck. I may've been doing that wrong, though. Any help greatly appreciated! Martin

2 2

[lxml-dev] parse timeout
by David Lindquist April 8, 2010

April 8, 2010

Hello, I have to parse a series of URLs, some of which might hang for an unacceptable length of time. I cannot figure out how to add a timeout: import socket from lxml.html import parse socket.setdefaulttimeout(10) doc = parse('http://example.com/hang_for_a_long_time') # this might hang indefinitely Is there some other way to add a timeout, short of recreating the parse function using urllib2? Thanks, David Lindquist

2 1

[lxml-dev] downloads-a-plenty on launchpad page?
by Wichert Akkerman April 6, 2010

April 6, 2010

I just noticed https://launchpad.net/lxml/ really likes you to download the lxml 2.2 release. So much in fact it has that download listed 129 times. I suspect that isn't intentional? :) Wichert.

2 2

[lxml-dev] adding a namespace
by Wichert Akkerman April 5, 2010

April 5, 2010

I am having some problems adding a new namespace to a parsed document. My goal is to take an input file like this: <html xmlns="http://www.w3.org/1999/xhtml"> <body> <div id="one"><p>first paragraph</p></div> <div id="two"><p>second paragraph</p></div> </body> </html> and turn it into this: <html xmlns="http://www.w3.org/1999/xhtml" xmlns:i18n="http://xml.zope.org/namespaces/i18n"> <body> <div id="one"><p i18n:translate="string1">first paragraph</p></div> <div id="two"><p i18n:translate="string2">second paragraph</p></div> </body> </html> the code is fairly simple, and looks like this (simplified from original): NS="http://xml.zope.org/namespaces/i18n" tree=lxml.etree.parse(input) root=tree.getroot() count=1 if "i18n" not in root.nsmap: root.nsmap["i18n"]=NS for el in root.iter(): if "{%s}translate" % NS in el.attrib: continue if hasText(el): el.attrib["{%s}translate" % NS]="string%d" % count count+=1 print lxml.etree.tostring(tree) However the resulting output looks like this: <html xmlns="http://www.w3.org/1999/xhtml"> <body> <div id="one"><p xmlns:ns0="http://xml.zope.org/namespaces/i18n" ns0:translate="string1">first paragraph</p></div> <div id="two"><p xmlns:ns1="http://xml.zope.org/namespaces/i18n" ns1:translate="string2">second paragraph</p></div> </body> </html> while trying to debug this I noticed something odd: lxml allows you to modify the nsmap for an element, but ignores what you do: >>> root.nsmap {None: 'http://www.w3.org/1999/xhtml', 'py': 'http://genshi.edgewall.org/', 'xi': 'http://www.w3.org/2001/XInclude'} >>> root.nsmap["frop"]='http://frip' >>> root.nsmap {None: 'http://www.w3.org/1999/xhtml', 'py': 'http://genshi.edgewall.org/', 'xi': 'http://www.w3.org/2001/XInclude'} I would expect that to either work, or raise an exception telling me I am trying to do something that is not allowed. The current behaviour feels a bit unpythonic. It is possible to specify your own nsmap when creating elements, but I can not find an API to modify the nsmap for a parsed tree. Is that a missing feature, or is there another way to do this? Wichert.

3 5

[lxml-dev] lxml iterparse generator not returning anything
by Joe Sarre April 5, 2010

April 5, 2010

Hi everyone, I'm finding that when using iterparse, the generator always throws StopIteration immediately, without returning any data. I must be doing something wrong, or I must have some kind of setup problem, but I'm struggling to work out what it is. If anybody has any ideas, then that would be greatly appreciated, or if this is a bug, I will raise it on the bug tracker. My version details are: >>> print etree.LXML_VERSION (2, 2, 2, 0) >>> print etree.LIBXML_VERSION (2, 7, 6) >>> print etree.LIBXML_COMPILED_VERSION (2, 7, 3) >>> print etree.LIBXSLT_VERSION (1, 1, 26) >>> print etree.LIBXSLT_COMPILED_VERSION (1, 1, 24) The most striking thing about this is that LIBXML_VERSION != LIBXML_COMPILED_VERSION, and LIBXSLT_VERSION != LIBXSLT_COMPILED_VERSION. If this version discrepancy is the real cause of the problem, then I think this issue is perhaps more appropriate for the Fedora mailing list, and you can ignore the rest of this mail. An example in which I am seeing this ( taken from http://codespeak.net/lxml/parsing.html#iterparse-and-iterwalk ) is: """ >>> from lxml import etree >>> from StringIO import StringIO >>> xml = '''<root> ... <element key='value'>text</element> ... <element>text</element>tail ... <empty-element xmlns="http://testns/" /> ... </root>''' >>> print xml <root> <element key='value'>text</element> <element>text</element>tail <empty-element xmlns="http://testns/" /> </root> >>> context = etree.iterparse(StringIO(xml)) >>> for action, elem in context: ... print("%s: %s" % (action, elem.tag)) end: element end: element end: {http://testns/}empty-element end: root """ if __name__ == '__main__': import doctest doctest.testmod() The result of putting this in a file and running it is that python complains: ********************************************************************** File "test.py", line 20, in __main__ Failed example: for action, elem in context: print("%s: %s" % (action, elem.tag)) Expected: end: element end: element end: {http://testns/}empty-element end: root Got nothing ********************************************************************** 1 items had failures: 1 of 6 in __main__ ***Test Failed*** 1 failures. Thanks in advance for any help, Joe Sarre This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Skyscanner. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.

2 1

[lxml-dev] Tempory data attached to custom subclasses
by Dave Kuhlman April 1, 2010

April 1, 2010

I've been using the custom subclasses capability of lxml. It's slick. I do, however, miss the ability to attach temporary data to the ElementBase subclasses. (see the warnings under "Element initialization" at http://codespeak.net/lxml/element_classes.html) I can, as suggested by the docs, add attributes or children to the underlying etree.Element, but that means that I'd have to strip that temporary data off when I want to serialize the tree. (please stop me if you've already heard this request, or if there is another solution.) I'd have a solution (see below) to this need if I could get a value, say an ID, (1) that is unique to each node and (2) that does not change during the existence of the ElementTree. Note that this "ID" does not have to be meaningful, and does not need to enable me to do anything with the underlying XML object (other than re-identify it). If I could get this opaque ID (or whatever it might be called), then I could use a dictionary and something like the following to store and retrieve temporary data:: Datadict1 = {} def get_temp_data(node, datadict): id = node.get_opaque_id() if id in datadict: return datadict[id] else: data = {} datadict[id] = data return data def test(): doc = lxml.parse('somedoc.xml') root = doc.getroot() node = root[0] data = get_temp_data(node, Datadict1) value1 = 'some temporary data' data['key1'] = value1 o o o data = get_temp_data(node, Datadict1) print data['key1'] test() Looking at lxml-2.2.4/src/lxml/lxml.etree.pyx, it seems like that would be a trivial function to add. (see below) What do you think? It's pretty simple solution. Has it be tried or rejected already? Here is a patch that seems to add the necessary function. This function returns the C pointer to the libxml2 object that is underneath the lxml/etree object. Am I right that this value would be (1) unique and (2) persistent across the lifetime of the lxml/etree ElementTree? Index: lxml.etree.pyx =================================================================== --- lxml.etree.pyx (revision 71999) +++ lxml.etree.pyx (working copy) @@ -1185,6 +1185,21 @@ return None return _elementFactory(self._doc, c_node) + def getopaqueid(self): + u"""getopaqueid(self) + + Returns an opaque ID for the underlying XML C node. This + opaque ID is guaranteed (1) to be unique to each node + and (2) not to change during the existence of the + ElementTree. + """ + cdef xmlNode* c_node + cdef int intnode + c_node = self._c_node + intnode = <int>c_node + opaqueid = intnode + return opaqueid + def getnext(self): u"""getnext(self) - Dave -- Dave Kuhlman http://www.rexx.com/~dkuhlman

2 4