Mailman 3 September 2012 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel April 11, 2023

April 11, 2023

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon April 10, 2023

April 10, 2023

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede Jan. 30, 2023

Jan. 30, 2023

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel Jan. 11, 2023

Jan. 11, 2023

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva Aug. 31, 2022

Aug. 31, 2022

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) May 3, 2013

May 3, 2013

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Apache2 crashes
by Brieuc SCHAFF Feb. 12, 2013

Feb. 12, 2013

Hi, We are using lxml in several of our applications with Python 2.6 and from time to time, the application stops responding after a segmentation fault error, and this kind of backtrace: *Jul 1 15:35:01 server4 httpd: *** glibc detected *** /usr/sbin/apache2: malloc(): smallbin double linked list corrupted: 0x00007f583a03f500 **** *Jul 1 15:35:01 server4 httpd: ======= Backtrace: =========* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f58376c2806]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x7bb39)[0x7f58376c6b39]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(__libc_malloc+0x6e)[0x7f58376c77de]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlInitParserCtxt+0x317)[0x7f58343f2887]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlNewParserCtxt+0x2c)[0x7f58343f2acc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x54eda)[0x7f5818db6eda]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xddcf3)[0x7f5818e3fcf3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xab5ee)[0x7f5818e0d5ee]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb7d8e)[0x7f5818e19d8e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb827e)[0x7f5818e1a27e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f582face030]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5a98)[0x7f582face928]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f582facc96f]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f582fa39cef]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f582fa8e6dc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f582fac8193] * *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f582fe9bfdb]* *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f582fea0eed]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f5837c14eb3]* *Jul 1 15:35:01 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f58379d49ca]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f5837731cdd]* ----------------- *Jul 11 10:44:48 server4 httpd: *** glibc detected *** /usr/sbin/apache2: munmap_chunk(): invalid pointer: 0x00007f3968b8cb68 **** *Jul 11 10:44:48 server4 httpd: ======= Backtrace: =========* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f39789dd806]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlResetError+0x22)[0x7f39755038e2]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlRaiseError+0x237)[0x7f3975504e07]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlSimpleError+0x6b)[0x7f397550543b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputBufferGrow+0x16b)[0x7f397553271b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputGrow+0x4c)[0x7f397550739c]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x3bd52)[0x7f397550bd52]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParseDocument+0x4b8)[0x7f39755204c8]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x505c5)[0x7f39755205c5]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x8fab0)[0x7f3958ad8ab0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2daa0)[0x7f3958a76aa0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2ddc9)[0x7f3958a76dc9]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc26f0)[0x7f3958b0b6f0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc431e)[0x7f3958b0d31e]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f397063b030]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f397063996f]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f39705a6cef]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f39705fb6dc]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f3970635193] * *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f3970a08fdb]* *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f3970a0deed]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f3978f2feb3]* *Jul 11 10:44:48 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f3978cef9ca]* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f3978a4ccdd]* Stopping apache and restarting it fix the issue, but we have no idea why this is happening. We use Unbuntu 10.10's package of lxml, which is version 2.2.4. *>>> print("%-20s: %s" % ('Python', sys.version_info))* *Python : (2, 6, 5, 'final', 0)* *>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))* *lxml.etree : (2, 2, 4, 0)* *>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))* *libxml used : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))* *libxml compiled : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))* *libxslt used : (1, 1, 26)* *>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))* *libxslt compiled : (1, 1, 26)* * * Do you have any idea of what could cause these issues ? I wonder if upgrading to a newer version of the library could change anything. Thanks in advance for any answer you could provide. Regards, B. _____________________ Brieuc SCHAFF +33609898260 _____________________

2 3

ValueError on python3.3
by Andriy Kornatskyy Sept. 30, 2012

Sept. 30, 2012

Here is an error with python3.3 >>> from lxml.etree import HTMLParser Traceback (most recent call last): File "<stdin>", line 1, in <module> File "lxml.etree.pyx", line 48, in init lxml.etree (src/lxml/lxml.etree.c:159821) ValueError: level must be >= 0 Is that something known? Please advise. Thanks. Andriy Kornatskyy

3 6

Re: [lxml] Installing with pip seems to always choose version 3.0.alpha2
by Stefan Behnel Sept. 30, 2012

Sept. 30, 2012

Tal Einat, 29.09.2012 16:25: > On Tue, Sep 25, 2012 at 6:16 PM, Stefan Behnel wrote: >> Anyway, I deleted the version from PyPI now, so it can't be downloaded >> anymore. > > Thanks for the fix! > > It might indeed be a pip issue, but note that (before the fix) I was > telling pip to install specifically lower versions, e.g.: > > pip install lxml==2.3 > pip install lxml==2.3.5 > pip install "lxml<3" > > In all of these cases, pip installed 3.0.alpha2. I simply couldn't get it > to install any other version. > > Should I file a bug report for pip in this regard? Please do. At least bring it up on their mailing list so that they can comment on it. Stefan

1 0

lxml 2.3.6 released
by Stefan Behnel Sept. 28, 2012

Sept. 28, 2012

Hi, I just released lxml 2.3.6. It's a pure bug fix release that closes one bug that was already fixed in 3.0 alpha 2. http://pypi.python.org/pypi/lxml/2.3.6 Bugs fixed * Passing long Unicode strings into the feed() parser interface failed to read the entire string. This release was built with Cython 0.15.1. Have fun, Stefan

1 0

Installation / PyPI issues: 3.0(alpha|beta)
by Tim Diggins Sept. 28, 2012

Sept. 28, 2012

Hi Stefan and co - I've installed lxml 3.0alpha 2 using pip last week (simply: pip install lxml) However, working in a fresh virtualenv I tried again, and It looks like lxml 3.0 alpha 2 has been removed from pypi and lxml 3.0 beta isn't yet on pypi. When I try to install using pip install pip install git+github.com/lxml/lxml@26358d2218a84ad8ee89cce64828f907489c43af I get some compilation warnings and errors, which indicate that either: * src/lxml/lxml.etree.c should be present (is that because I'm using the git version rather than a pypi version?) or * I should install Cython Is that right? But will 3.0beta be out on pypi in a few days? (in which case I'll just wait for that) thanks Tim Tim Diggins http://red56.co.uk

2 1

root.docinfo.externalDTD is None
by Thomas Larsen Wessel Sept. 28, 2012

Sept. 28, 2012

I have an xml document with a doctype pointing to a dtd in the same folder: <!DOCTYPE point SYSTEM "point.dtd"> <point> <address>abc</address> </point> I parse it, and check out the dtd's root = etree.parse(open('point_dtd.xml')) intdtd = root.docinfo.internalDTD # lxml.etree.DTD object extdtd = root.docinfo.externalDTD # None I was expecting externalDTD to return a DTD for the point.dtd schema, and the internalDTD to be None. Furthermore intdtd is "empty", for instance intdtd.elements returns an empty list, while point.dtd does in fact define elements! Shouldn't externalDTD point to dtd for the point.dtd?

2 3

Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 27, 2012

Sept. 27, 2012

Hi Stefan,, I tested by checking sys.path and through strace but couldn't get much out of it. However, a clean install has worked :) Special thanks to you and Simon :) Regards Pankaj Singh From: Pankaj Singh Sent: 26-09-2012 22:58 To: lxml(a)lxml.de Subject: RE: [lxml] PyUnicodeUCS2_DecodeLatin1 error Hi Stefan, Thanks for the inputs. I'll test again with a clean install. Will post the results. Regards Pankaj Singh From: Stefan Behnel Sent: 26-09-2012 21:59 To: lxml(a)lxml.de Subject: Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error Simon Sapin, 26.09.2012 17:18: > Le 26/09/2012 17:10, Pankaj Singh a écrit : >> Is there an option wherein I can specify unicode style while building >> lxml? No. > I *think* it should just pick it up from the current python at > compile-time Yes, that's fully automatic. There is no way it can get that wrong. > Just to be clear: you’re installing/compiling lxml from a source tarball > with a USC4 python, and when importing it in the same python get > "undefined symbol: PyUnicodeUCS2_DecodeLatin1". Is this correct? That can't be the case. I assume there's something wrong with either the import path (PYTHONPATH) or the local installation, potentially both. For diagnosis, run strace -e open,stat python -c 'import lxml.etree' | fgrep etree That will show from where it tries to import the lxml.etree module. In any case, a clean install should fix this. Stefan _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

lxml 3.0 beta1 released
by Stefan Behnel Sept. 26, 2012

Sept. 26, 2012

Hi everyone, I'm proud to announce the first beta release of lxml 3.0. This release fixes a serious memory leak in the last alpha release, so please upgrade if you currently use a pre-release of 3.0. The 3.0 release series features initial support for PyPy and a new tree API for DTDs (by Walter Dörwald), as well as some major cleanups in the code base. The lxml.cssselect package was moved into a separate PyPI project nicely extended and maintained by Simon Sapin. Originally, this release was targeted as lxml 2.4, but some major improvements as well as a couple of potentially incompatible changes make "3.0" appear more appropriate. The final release of 3.0 is scheduled for early October, so now is a good time to report any remaining bugs and to send in your bug fix pull requests for approval. The documentation is here: http://lxml.de/3.0/ Download: http://lxml.de/files/lxml-3.0beta1.tgz Signature: http://lxml.de/files/lxml-3.0beta1.tgz.asc GitHub: https://github.com/lxml/lxml/commit/26358d2218a84ad8ee89cce64828f907489c43af This release was built using Cython 0.17.1. Development on lxml's sources now requires this version. If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.0beta1 (2012-09-26) ===================== Features added -------------- * Python level access to (optional) libxml2 memory debugging features to simplify debugging of memory leaks etc. Bugs fixed ---------- * Fix a memory leak in XPath by switching to Cython 0.17.1. * Some tests were adapted to work with PyPy. Other changes ------------- * The code was adapted to work with the upcoming libxml2 2.9.0 release. 3.0alpha2 (2012-08-23) ====================== Features added -------------- * The .iter() method of elements now accepts tag arguments like "{*}name" to search for elements with a given local name in any namespace. With this addition, all combinations of wildcards now work as expected: "{ns}name", "{}name", "{*}name", "{ns}*", "{}*" and "{*}*". Note that "name" is equivalent to "{}name", but "*" is "{*}*". The same change applies to the .getiterator(), .itersiblings(), .iterancestors(), .iterdescendants(), .iterchildren() and .itertext() methods, the strip_attributes(), strip_elements() and strip_tags() functions as well as the iterparse() function. * C14N allows specifying the inclusive prefixes to be promoted to top-level during exclusive serialisation. Bugs fixed ---------- * Passing long Unicode strings into the feed() parser interface failed to read the entire string. Other changes ------------- 3.0alpha1 (2012-07-31) ====================== Features added -------------- * Initial support for building in PyPy (through cpyext). * DTD objects gained an API that allows read access to their declarations. * xpathgrep.py gained support for parsing line-by-line (e.g. from grep output) and for surrounding the output with a new root tag. * E-factory in lxml.builder accepts subtypes of known data types (such as string subtypes) when building elements around them. * Tree iteration and iterparse() with a selective tag argument supports passing a set of tags. Tree nodes will be returned by the iterators if they match any of the tags. Bugs fixed ---------- * The .find*() methods in lxml.objectify no longer use XPath internally, which makes them faster in many cases (especially when short circuiting after a single or couple of elements) and fixes some behavioural differences compared to lxml.etree. Note that this means that they no longer support arbitrary XPath expressions but only the subset that the ElementPath language supports. The previous implementation was also redundant with the normal XPath support, which can be used as a replacement. * el.find('*') could accidentally return a comment or processing instruction that happened to be in the wrong spot. (Same for the other .find*() methods.) * The error logging is less intrusive and avoids a global setup where possible. * Fixed undefined names in html5lib parser. * xpathgrep.py did not work in Python 3. * Element.attrib.update() did not accept an attrib of another Element as parameter. * For subtypes of ElementBase that make the .text or .tail properties immutable (as in objectify, for example), inserting text when creating Elements through the E-Factory feature of the class constructor would fail with an exception, stating that the text cannot be modified. Other changes -------------- * The code base was overhauled to properly use 'const' where the API of libxml2 and libxslt requests it. This also has an impact on the public C-API of lxml itself, as defined in etreepublic.pxd, as well as the provided declarations in the lxml/includes/ directory. Code that uses these declarations may have to be adapted. On the plus side, this fixes several C compiler warnings, also for user code, thus making it easier to spot real problems again. * The functionality of "lxml.cssselect" was moved into a separate PyPI package called "cssselect". To continue using it, you must install that package separately. The "lxml.cssselect" module is still available and provides the same interface, provided the "cssselect" package can be imported at runtime. * Element attributes passed in as an attrib dict or as keyword arguments are now sorted by (namespaced) name before being created to make their order predictable for serialisation and iteration. Note that adding or deleting attributes afterwards does not take that order into account, i.e. setting a new attribute appends it after the existing ones. * Several classes that are for internal use only were removed from the lxml.etree module dict: _InputDocument, _ResolverRegistry, _ResolverContext, _BaseContext, _ExsltRegExp, _IterparseContext, _TempStore, _ExceptionContext, __ContentOnlyElement, _AttribIterator, _NamespaceRegistry, _ClassNamespaceRegistry, _FunctionNamespaceRegistry, _XPathFunctionNamespaceRegistry, _ParserDictionaryContext, _FileReaderContext, _ParserContext, _PythonSaxParserTarget, _TargetParserContext, _ReadOnlyProxy, _ReadOnlyPIProxy, _ReadOnlyEntityProxy, _ReadOnlyElementProxy, _OpaqueNodeWrapper, _OpaqueDocumentWrapper, _ModifyContentOnlyProxy, _ModifyContentOnlyPIProxy, _ModifyContentOnlyEntityProxy, _AppendOnlyElementProxy, _SaxParserContext, _FilelikeWriter, _ParserSchemaValidationContext, _XPathContext, _XSLTResolverContext, _XSLTContext, _XSLTQuotedStringParam * Several internal classes can no longer be inherited from: _InputDocument, _ResolverRegistry, _ExsltRegExp, _ElementUnicodeResult, _IterparseContext, _TempStore, _AttribIterator, _ClassNamespaceRegistry, _XPathFunctionNamespaceRegistry, _ParserDictionaryContext, _FileReaderContext, _PythonSaxParserTarget, _TargetParserContext, _ReadOnlyPIProxy, _ReadOnlyEntityProxy, _OpaqueDocumentWrapper, _ModifyContentOnlyPIProxy, _ModifyContentOnlyEntityProxy, _AppendOnlyElementProxy, _FilelikeWriter, _ParserSchemaValidationContext, _XPathContext, _XSLTResolverContext, _XSLTContext, _XSLTQuotedStringParam, _XSLTResultTree, _XSLTProcessingInstruction

1 0

Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 26, 2012

Sept. 26, 2012

Hi Stefan, Thanks for the inputs. I'll test again with a clean install. Will post the results. Regards Pankaj Singh From: Stefan Behnel Sent: 26-09-2012 21:59 To: lxml(a)lxml.de Subject: Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error Simon Sapin, 26.09.2012 17:18: > Le 26/09/2012 17:10, Pankaj Singh a écrit : >> Is there an option wherein I can specify unicode style while building >> lxml? No. > I *think* it should just pick it up from the current python at > compile-time Yes, that's fully automatic. There is no way it can get that wrong. > Just to be clear: you’re installing/compiling lxml from a source tarball > with a USC4 python, and when importing it in the same python get > "undefined symbol: PyUnicodeUCS2_DecodeLatin1". Is this correct? That can't be the case. I assume there's something wrong with either the import path (PYTHONPATH) or the local installation, potentially both. For diagnosis, run strace -e open,stat python -c 'import lxml.etree' | fgrep etree That will show from where it tries to import the lxml.etree module. In any case, a clean install should fix this. Stefan _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 26, 2012

Sept. 26, 2012

Hi Simon, Thanks for the quick reply. Lxml was built from the tar using the same python. For safety sake i'll test again after removing all other python versions from the system. Is there an option wherein I can specify unicode style while building lxml? Thanks Pankaj Singh From: Simon Sapin Sent: 26-09-2012 20:24 To: lxml(a)lxml.de Subject: Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error Le 26/09/2012 16:39, Pankaj Singh a écrit : > *ImportError: > /usr/local/Python/lib/python2.6/site-packages/lxml/etree.so: undefined > symbol: PyUnicodeUCS2_DecodeLatin1* lxml can support either UCS4 or UCS2 as the internal Unicode representation, but the switch is made at compile-time. Make sure that the Python that compiled etree.so and the Python that uses it are the same. Same version (at least minor as in 2.7.x, maybe micro as in 2.7.3), same architecture, and in this case same Unicode settings (UCS4 vs UCS2.) The easiest way to do this is to re-install lxml from the source tarball. Do you still have this issue? -- Simon Sapin _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

3 2

Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 26, 2012

Sept. 26, 2012

Yes, that is correct. Pankaj Singh From: Simon Sapin Sent: 26-09-2012 20:48 To: lxml(a)lxml.de Subject: Re: [lxml] PyUnicodeUCS2_DecodeLatin1 error Le 26/09/2012 17:10, Pankaj Singh a écrit : > Is there an option wherein I can specify unicode style while building > lxml? I *think* it should just pick it up from the current python at compile-time, but there might be something buggy. Just to be clear: you’re installing/compiling lxml from a source tarball with a USC4 python, and when importing it in the same python get "undefined symbol: PyUnicodeUCS2_DecodeLatin1". Is this correct? -- Simon Sapin _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

1 0

PyUnicodeUCS2_DecodeLatin1 error
by Pankaj Singh Sept. 26, 2012

Sept. 26, 2012

Hi, Im getting the following error while importing etree *ImportError: /usr/local/Python/lib/python2.6/site-packages/lxml/etree.so: undefined symbol: PyUnicodeUCS2_DecodeLatin1* Environment specs are as follows OS : Red Hat 4.1.2-14 Python : 2.6.4 lxml : 2.3.5 *I needed to install Python with flag --enable-unicode=ucs4* It looks as if lxml is not supporting unicode ucs4 through which python was installed. What are my options if I want to use lxml with above mentioned unicode based python installation? My apologies if this question has been asked before but I ain't quite sure as to how I can search the previous posts without going through posts from every month one by one in the archives :( Thanks and Regards Pankaj Singh

2 1

Installing with pip seems to always choose version 3.0.alpha2
by Tal Einat Sept. 25, 2012

Sept. 25, 2012

pip install lxml==2.3 Running setup.py egg_info for package lxml Building lxml version 3.0.alpha2. ... This is very annoying, I can't seem to get pip to install any version of lxml other than 3.0.alpha2! I don't know if it's an issue with pip or with lxml, but please fix this... Regards, - Tal Einat

3 6

Strange memory leakage problem on Ubuntu
by zzjjzzgggg Sept. 25, 2012

Sept. 25, 2012

Hello, I am new to lxml. I installed lxml on Ubuntu 12.04 via easy_install as suggested easy_install --allow-hosts=lxml.de,*.python.org lxml The version of libxml2 is Version: 2.7.8.dfsg-5.1ubuntu4.1 I use python 3.2. When I use lxml to process some HTML fragments, I find serious memory leakage issues. The test code and test data are attached. When I remove this line ps=html.xpath("//p[@node-type='feed_list_content']") , There isn't memory leakage problem. What's the problem please? Many thanks. I can provide more info if you need. I post a thread in stackoverflow. Somebody suggests it is a bug. http://stackoverflow.com/questions/12383537/lxml-memeory-leakage-with-pytho… Best regards zzjjzzgggg

2 2

lxml eats all memory
by Kevin Leo Sept. 25, 2012

Sept. 25, 2012

Hello, I’m testing the performance of lxml. My test method is: Given a lot of HTML files (About 50000). For each HTML file content, calculate the time spending in lxml.html.fromstring. Finally, calculate the total time. At the beginning, the test process used about 60M physical memory. But after processing thousands of HTML files, the test process will consume all memory quickly when processing a HTML file. “strace -p” shows it do nothing but brk(). First, I thought it’s caused by a strange HTML file. But the result of fromstring that file only is OK. Here is lxml info: 1. sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0) 2. lxml.etree : (2, 3, 5, 0) 3. libxml used : (2, 8, 0) 4. libxslt used : (1, 1, 26) 5. OS: Linux dev-test 2.6.32-gentoo-r29 #1 SMP Fri Jun 1 11:21:29 CST 2012 x86_64 Intel(R) Pentium(R) Dual CPU E2180 @ 2.00GHz GenuineIntel GNU/Linux

2 1

Memory debugging
by Christian Heimes Sept. 19, 2012

Sept. 19, 2012

Hello, I'd like to suggest a trival feature for lxml. libxml2 and libxslt support extended memory debugging when both libraries are compiled with the the '--with-mem-debug' flag [1]. This is useful to debug how many memory is actually used by the XML and XSTL code. In order to show the amount of used memory and memory blocks, the two functions xmlMemBlocks() [2] and xmlMemUsed() [3] needs to be wrapped. Both functions are available in non-debug builds and return 0 in that case. It might be worth to wrap the other debug helpers like xmlMemDisplay(FILE *) [4], too. Regards, Christian Example patch: diff -ru lxml-2.3.4.orig/src/lxml/lxml.etree.pyx lxml-2.3.4/src/lxml/lxml.etree.pyx --- lxml-2.3.4.orig/src/lxml/lxml.etree.pyx 2012-03-26 12:28:58.000000000 +0200 +++ lxml-2.3.4/src/lxml/lxml.etree.pyx 2012-05-03 18:33:00.702130662 +0200 @@ -2956,6 +2956,11 @@ except _TargetParserResult, result_container: return result_container.result +def memoryUsed(): + return tree.xmlMemUsed() + +def memoryBlocks(): + return tree.xmlMemBlocks() ################################################################################ # Include submodules diff -ru lxml-2.3.4.orig/src/lxml/tree.pxd lxml-2.3.4/src/lxml/tree.pxd --- lxml-2.3.4.orig/src/lxml/tree.pxd 2011-09-25 18:58:05.000000000 +0200 +++ lxml-2.3.4/src/lxml/tree.pxd 2012-05-03 17:44:50.323798041 +0200 @@ -322,6 +322,7 @@ cdef extern from "libxml/xmlmemory.h": cdef void* xmlMalloc(size_t size) nogil cdef int xmlMemBlocks() nogil + cdef int xmlMemUsed() nogil cdef extern from "etree_defs.h": cdef bint _isElement(xmlNode* node) nogil [1] http://xmlsoft.org/html/libxml-xmlmemory.html#DEBUG_MEMORY [2] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemBlocks [3] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemUsed [4] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemDisplay

2 2

Re: [lxml] "xmlns: URI is not absolute" mystery/bug?
by Andrew McNamara Sept. 17, 2012

Sept. 17, 2012

>> I don't actually need or want the validation - what I do want is the >> type hints it provides to objectify. Is there a way to pass the schema >> type information to objectify without also performing validation? > >I think you're mistaken: There is no information taken from the schema >for the objectify type annotations. Indeed you are correct - I am mistaken. I did several experiments and thought I saw it casting to the schema-specified types, but I just repeated them and it must just have been too much wishful thinking. >This comes up from time to time but so far no one has made an effort to >implement a generalized solution for lxml. > >See also >https://mailman-mail5.webfaction.com/pipermail/lxml/2008-January/003245.html Paraphrasing (I think): libxml2 doesn't make the schema information readily accessible, which makes it difficult to implement this efficiently. Thanks for your help! -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

1 0

"xmlns: URI is not absolute" mystery/bug?
by Andrew McNamara Sept. 17, 2012

Sept. 17, 2012

I'm trying to parse some XML using a parser with schema. The XML contains a data error, and should raise: lxml.etree.XMLSyntaxError: Element '{xmlapi_1.0}routerId': [facet 'minInclusive'] The value '0' is less than the minimum value allowed ('1'). Instead, I'm getting lxml.etree.XMLSyntaxError: xmlns: URI xmlapi_1.0 is not absolute, line 2, column 17 I suspect what is happening is the "URI xmlapi_1.0 is not absolute" is a warning and is being ignored, but when the later data error is encountered, the "URI is not absolute" is the first message found and it is being erroneously returned instead. As an aside, is it possible to ignore the above data error somehow and continue loading the XML (I don't control the source of data, nor the schema - both are massive and machine generated)? The code below demonstrates the problem: from lxml import objectify, etree schema = etree.XMLSchema(etree.fromstring('''\ <?xml version="1.0" encoding="UTF-8"?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="xmlapi_1.0" xmlns="xmlapi_1.0" elementFormDefault="qualified"> <xsd:element name="container"> <xsd:complexType> <xsd:all> <xsd:element name="routerId" minOccurs="0"> <xsd:simpleType> <xsd:restriction base="xsd:int"> <xsd:minInclusive value="1"/> <xsd:maxInclusive value="10240"/> </xsd:restriction> </xsd:simpleType> </xsd:element> </xsd:all> </xsd:complexType> </xsd:element> </xsd:schema> ''')) parser = objectify.makeparser(schema=schema) fails = '''\ <?xml version="1.0"?> <container xmlns="xmlapi_1.0"> <routerId>0</routerId> </container> ''' result = objectify.fromstring(fails, parser) -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

2 2

lxml and Unicode
by Martin Mueller Sept. 16, 2012

Sept. 16, 2012

Jens Quade wrote: etree.parse expects a filename or a file-like object. try doc = etree.XML(text) - it expects a string containing XML. That works but raises a documentation problem: I didn't see anything to that effect in the lxml documentation, at least not in the tutorial. I don't quite understand the following: doc = etree.parse(text) works when it is part of a separate script. It doesn't work in the combined script, where all of a sudden the 'parse' instruction is unnecessary. But I'm grateful for help that makes things work. M I'm new to lxml and a little wobbly on Unicode in Python 2.7. Here is my problem: 1. I take an SGML text with character entities and convert them to UTF-8 characters 2. I extract note elements from the text with the converted characters. This works if I use two separate Python scripts. The first goes like this: # This Python file uses the following encoding: utf-8 import entityrules filein = open('/users/martin/dropbox/eebo-SGMLfiles/200504/A10675.sgm', 'r') fileout = open('/users/martin/dropbox/lxml/A10675.xml', 'w') text = filein.read() text = entityrules.replaceCharacterEntities(text) print >> fileout, text It takes the SGML file as an input. applies the "replaceCharacterEntities" and writes that output to a new file. The second script goes like this: #encoding = 'utf-8' from lxml import etree filein ='/users/martin/dropbox/lxml/a10675.xml' fileout = open('/users/martin/dropbox/lxml/notesA10675.xml', 'w') doc = etree.parse(filein) for note in doc.getiterator('note'): print >>fileout, etree.tostring(note, with_tail = False, encoding = 'utf-8') Its input is the file generated by the first script. It converts the input file into an element tree and then iterates through the tree, extracting a particular element . Now it should be possible to combine those two scripts in a single script as follows: # This Python file uses the following encoding: utf-8 import entityrules from lxml import etree #converting the SGML character entities to UTF-8 characcters filein = open('/users/martin/dropbox/eebo-SGMLfiles/200504/A10675.sgm', 'r') fileout = open('/users/martin/dropbox/lxml/notesA10675.xml', 'w') text = filein.read() text = entityrules.replaceCharacterEntities(text) print "I finished the first part of this script" #creating an element tree and extracting instances of a particular element doc = etree.parse(text) for note in doc.getiterator('note'): print >>fileout, etree.tostring(note, with_tail = False, encoding = 'utf-8') This script produces the following output: Martin-Muellers-Mac-Pro:lxml martin$ python getNotes.py I finished the first part of this script Traceback (most recent call last): File "getNotes.py", line 14, in <module> doc = etree.parse(text) File "lxml.etree.pyx", line 2954, in lxml.etree.parse (src/lxml/lxml.etree.c:56220) File "parser.pxi", line 1533, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:82303) File "parser.pxi", line 1562, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:82596) File "parser.pxi", line 1462, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:81635) File "parser.pxi", line 1002, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:78544) File "parser.pxi", line 569, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:74488) File "parser.pxi", line 650, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:75379) File "parser.pxi", line 584, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:74650) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1126: ordinal not in range(128) The script does the first part of the job correctly, as is apparent from the first line. And the "text" variable that is passed to the etree function should in principle be identical with the content of the file that is passed to my script from a file. But it isn't, and somewhere along the road Python seems to have forgotten that we 're dealing with Unicode texts. What am I doing wrong? Or is there a bug? I am using the latest Enthought version of Python, which comes bundled with lxml Martin Mueller Professor of English and Classics Northwestern University

2 1