Mailman 3 April 2013 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 10 Apr '23

10 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 09 Apr '23

09 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 30 Aug '22

30 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) 03 May '13

03 May '13

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Namespaced attributes losing their namespace
by Tyler Spivey 01 May '13

01 May '13

In the below test script, I'm not getting the result I expect. Is this a bug in lxml? If not, what am I doing wrong? I'm using lxml 3.1.1 on win32, downloaded from PyPi. What I get: <root xmlns="url1"> <test attrib="value"/> </root> I expect attrib to not lose its namespace, as shown when I re-parse the output: >>> lxml.etree.parse('test.xml').getroot()[0].attrib {'attrib': 'value'} If I swap the append and the attribute setting, things work: <test xmlns:ns0="url1" ns0:attrib="value"/> >>> lxml.etree.parse('test.xml').getroot()[0].attrib {'{url1}attrib': 'value'} If this is intended behaviour, I'll have to see how to fit that into my design, since my functions return elements ready for appending. from lxml import etree nsmap = {None: 'url1'} root = etree.Element('{url1}root', nsmap=nsmap) test = etree.Element('{url1}test') test.attrib['{url1}attrib'] = 'value' root.append(test) print etree.tostring(root, encoding='utf-8', pretty_print=True)

2 6

How to control the processing of newlines in etree.html xpath text() function?
by Olivier de Mirleau 30 Apr '13

30 Apr '13

Hi List, I recently upgraded from linux Fedora 17 to 18, and am facing a change in functionality of the lxml xpath text() function. Here's an example: from io import BytesIO from lxml import etree myHtmlString = \ '<!doctype html public "-//w3c//dtd html 4.0 transitional//en">\r\n'+\ '<html>\r\n'+\ '<head>\r\n'+\ ' <title> a b c </title>\r\n'+\ '</head>\r\n'+\ '<body/>\r\n'+\ '</html>\r\n' myFile = BytesIO(myHtmlString) myTree = etree.parse(myFile, etree.HTMLParser()) myTextElements = myTree.xpath("//text()") myFullText = ''.join([myEl for myEl in myTextElements]) print repr(myFullText) Under F17 that piece of code will write ' a b c ' whereas F18 produces '\r\n\r\n a b c \r\n\r\n\r\n' The version specifications are as follows: f17: Python : sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0) lxml.etree : (2, 3, 5, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) f18: Python : sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0) lxml.etree : (2, 3, 5, 0) libxml used : (2, 9, 1) libxml compiled : (2, 9, 0) libxslt used : (1, 1, 28) libxslt compiled : (1, 1, 26) I.e. neither python nor lxml.etree changed versions and I presume it's therefore due to the underlying libraries having changed versions. I wrote an application under F17 that is now broken under F18. I can imagine three ways to make it work under F18, in increasing order of impact on my own code: 1. Change a flag in an lxml call, recovering the F17 behaviour. 2. Wrap the F18 xpath function, to try and reproduce the F17 xpath 3. Port my downstream code which searches and transforms the output of xpath. Does anybody know whether solution 1 is possible, and if not, does anybody have a suggestion for the implementation of (2)? Bye, Olivier P.S. I initially put up this question on stackoverflow, but no satisfactory answer there yet: http://stackoverflow.com/questions/16123277/how-to-control-newline-processi…

4 6

lxml 3.2.0 released
by Stefan Behnel 28 Apr '13

28 Apr '13

Hi everyone, I just released lxml 3.2.0 as a major bug-fix release. Much of the work (fixing and analysing bugs) that went into this release was done by the participants of the lxml bugathon this weekend. Thanks everyone! You can get it from PyPI: http://pypi.python.org/pypi/lxml/3.2.0 The documentation is here: http://lxml.de/ Download: http://lxml.de/files/lxml-3.2.0.tgz Signature: http://lxml.de/files/lxml-3.2.0.tgz.asc Changelog: http://lxml.de/3.1/changes-3.2.0.html GitHub: https://github.com/lxml/lxml/commit/8fc99f8ef3e2f12e74fb407ae62c5db50cfe7b92 This release was built using Cython 0.19. Note that the build no longer uses Cython, even if it is installed. Recompilation of the sources has to be requested explicitly with the setup.py option "--with-cython". If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.2.0 (2013-04-28) ================== Features added -------------- Bugs fixed ---------- * LP#690319: Leading whitespace could change the behaviour of the string parsing functions in ``lxml.html``. * LP#599318: The string parsing functions in ``lxml.html`` are more robust in the face of uncommon HTML content like framesets or missing body tags. Patch by Stefan Seelmann. * LP#712941: I/O errors while trying to access files with paths that contain non-ASCII characters could raise ``UnicodeDecodeError`` instead of properly reporting the ``IOError``. * LP#673205: Parsing from in-memory strings disabled network access in the default parser and made subsequent attempts to parse from a URL fail. * LP#971754: lxml.html.clean appends 'nofollow' to 'rel' attributes instead of overwriting the current value. * LP#715687: lxml.html.clean no longer discards scripts that are explicitly allowed by the user provided whitelist. Patch by Christine Koppelt. Other changes -------------

1 1

Creating an lxml portable distribution
by Szymon Pyżalski 22 Apr '13

22 Apr '13

Hello! I need to install a current release of lxml into a virtualenv on a VPS. This VPS however has some memory quota that doesn't allow me to build it package there. I tried therefore to build a distribution on my laptop (the same architecture, but other linux distro). I used: $ CFLAGS='-fPIC' python setup.py bdist_egg --static_deps After installing the egg (with easy_install) on the server I get this: >>> import lxml.etree Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/pythonista/www/virtualenv/pc-qa/lib/python3.2/site-packages/lxml-3.1.2-py3.2-linux-x86_64.egg/lxml/etree.py", line 7, in <module> __bootstrap__() File "/home/pythonista/www/virtualenv/pc-qa/lib/python3.2/site-packages/lxml-3.1.2-py3.2-linux-x86_64.egg/lxml/etree.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) ImportError: /home/pythonista/www/virtualenv/pc-qa/lib/python3.2/site-packages/lxml-3.1.2-py3.2-linux-x86_64.egg/lxml/etree.cpython-32mu.so: cannot open shared object file: No such file or directory So apparently my distribution is still dependent on some files from the base machine. If I try to ldd etree.cpython-32mu.so I get a message that it is not a dynamic package, so I don't know what file does it look for. So what is the correct way to create a static, binary distribution for lxml, that I could install on a different machine? Greetings Szymon Pyżalski

5 4