lxml - The Python XML Toolkit February 2013

lxml@python.org

11 participants
13 discussions

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 11 Jan '23

11 Jan '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

4 4

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 21 Sep '22

21 Sep '22

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

3 2

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) 03 May '13

03 May '13

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

lxml 3.1.0 released
by Stefan Behnel 04 Mar '13

04 Mar '13

Hi everyone, I just released lxml 3.1.0. This is a stable feature release that mainly adds a new incremental XML serialisation API, but also fixes a couple of bugs. Upgrading from 3.0.x should be smooth. The complete changelog follows below. Note that this release is called "3.1.0" instead of just "3.1" in order to simplify version sorting. You can get it from PyPI: http://pypi.python.org/pypi/lxml/3.1.0 The documentation is here: http://lxml.de/ Download: http://lxml.de/files/lxml-3.1.0.tgz Signature: http://lxml.de/files/lxml-3.1.0.tgz.asc GitHub: https://github.com/lxml/lxml/commit/f76cca898b7edefe91be584540883d158416e6cb This release was built using Cython 0.18. Note that the build no longer uses Cython, even if it is installed. Recompilation of the sources has to be requested explicitly with the setup.py option "--with-cython". If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.1.0 (2013-02-10) ================== Features added -------------- * GH#89: lxml.html.clean allows overriding the set of attributes that it considers 'safe'. Patch by Francis Devereux. Bugs fixed ---------- * LP#1104370: ``copy.copy(el.attrib)`` raised an exception. It now returns a copy of the attributes as a plain Python dict. * GH#95: When used with namespace prefixes, the ``el.find*()`` methods always used the first namespace mapping that was provided for each path expression instead of using the one that was actually passed in for the current run. * LP#1092521, GH#91: Fix undefined C symbol in Python runtimes compiled without threading support. Patch by Ulrich Seidl. Other changes ------------- 3.1beta1 (2012-12-21) ===================== Features added -------------- * New build-time option ``--with-unicode-strings`` for Python 2 that makes the API always return Unicode strings for names and text instead of byte strings for plain ASCII content. * New incremental XML file writing API ``etree.xmlfile()``. * E factory in lxml.objectify is callable to simplify the creation of tags with non-identifier names without having to resort to getattr(). Bugs fixed ---------- * When starting from a non-namespaced element in lxml.objectify, searching for a child without explicitly specifying a namespace incorrectly found namespaced elements with the requested local name, instead of restricting the search to non-namespaced children. * GH#85: Deprecation warnings were fixed for Python 3.x. * GH#33: lxml.html.fromstring() failed to accept bytes input in Py3. * LP#1080792: Static build of libxml2 2.9.0 failed due to missing file. Other changes ------------- * The externally useless class ``_ObjectifyElementMakerCaller`` was removed from the module API of lxml.objectify. * LP#1075622: lxml.builder is faster for adding text to elements with many children. Patch by Anders Hammarquist.

5 16

benchmarks using an Element freelist
by Stefan Behnel 02 Mar '13

02 Mar '13

Hi, I had some fun during the last days implementing low-level object freelist support in Cython and gave it a try in lxml. It's implemented as a class decorator, so all you have to do is to add a "@cython.freelist(8)" declaration to get an 8 item freelist for the class. Even relatively short freelists tend to bring a lot as they hit typical iteration use cases, for example. For benchmarking, I initially used "Element.attrib", which is a property that instantiates a new object on each request (not cached in order to avoid reference cycles). Most use cases of this class consist of a one-shot operation to read some XML attributes, so it's a classical "create and throw away" thing, the ideal candidate for a freelist. The code that I ran through timeit is this: Setup: 'import lxml.etree as et; el = et.Element("tag", a="1", b="2")' Benchmarked code: 'el.attrib' Initial timings: 10000000 loops, best of 3: 0.0856 usec per loop Replacing "_Attrib(element)" by "_Attrib.__new__(_Attrib, element)": 10000000 loops, best of 3: 0.0781 usec per loop Adding only the freelist decorator to the _Attrib class: 10000000 loops, best of 3: 0.0737 usec per loop Combining both "__new__()" and the freelist: 10000000 loops, best of 3: 0.0608 usec per loop That gives an overall speedup of almost 30%, just by changing two lines of code. The main Element proxy class in lxml is another case where it really makes sense to apply this decorator. It gets instantiated whenever Python access to an XML element is requested, e.g. during iteration. Here are the timings for iterating over a large XML document. Benchmarked code: 'list(islice(tree.iter(), 10000000, 10000001))' Initial timings: 100 loops, best of 3: 15.6 msec per loop Enabling the freelist: 100 loops, best of 3: 13.2 msec per loop That's 15% faster, even all the way through a non-trivial tree iteration operation. And with just one additional declaration in the code. Note that this change will require Cython 0.19, which isn't currently close to release (we just recently released 0.18). So this change may take a bit of time to make it into an lxml release. Stefan

2 6

xpath issue:same structure different result
by python 22 Feb '13

22 Feb '13

I am having issues with the urllib and lxml.html modules. Here is my original code: import urllib import lxml.html down='http://v.163.com/special/visualizingdata/' file=urllib.urlopen(down).read() root=lxml.html.document_fromstring(file) xpath_str="//div[@class='down s-fc3 f-fl']/a" urllist=root.xpath(xpath_str)for url in urllist:print url.get("href") When run, it returns this output: http://mov.bn.netease.com/movieMP4/2012/12/A/7/S8H1TH9A7.mp4 http://mov.bn.netease.com/movieMP4/2012/12/D/9/S8H1ULCD9.mp4 http://mov.bn.netease.com/movieMP4/2012/12/4/P/S8H1UUH4P.mp4 http://mov.bn.netease.com/movieMP4/2012/12/B/V/S8H1V8RBV.mp4 http://mov.bn.netease.com/movieMP4/2012/12/6/E/S8H1VIF6E.mp4 http://mov.bn.netease.com/movieMP4/2012/12/B/G/S8H1VQ2BG.mp4 But, when I change the line xpath_str='//div[@class="down s-fc3 f-fl"]//a' into xpath_str='//div[@class="col f-cb"]//div[@class="down s-fc3 f-fl"]//a' that is to say, urllist=root.xpath('//div[@class="col f-cb"]//div[@class="down s-fc3 f-fl"]//a') I do not receive any output. What is the flaw in this code? it is so strange that the shorter one can work,the longer one can not,they have the same xpath structure!

3 2

Noob question.
by Sandip Shah 18 Feb '13

18 Feb '13

Hi, If I have this as my HTML <html> <head></head> <body> <a> cityMy City stateMy State </a> </body> </html> and I have this as my code s = '<html><head></head><body><a>cityCitystateState</a></body>' s1 = etree.HTML(s) How can I get (find or findall or something else) "My City" and "My State" using lxml? Thanks in advance. Silent

2 3

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

lxml - The Python XML Toolkit February 2013