Mailman 3 March 2013 - lxml - The Python XML Toolkit

Installation on Mac insanely complex
by Kurt Olsen March 10, 2013

March 10, 2013

Aloha, I'm trying to do 3d printing. Do to that I need to use Inkscape, and one it's plugins, which demands lxml. And I cannot get it installed. I'm an end-user. I know nothing of 'ports' or 'macports' of building it or this or that or the million other complex things you folks and google are talking about. All I know is this thing 'lxml' that you folks maintain is stopping me cold because I can't install it on OSX 10.7.5 Python 2.7.1 Inkscape 0.48 Any help would be greatly appreciated because right now LXML is standing between me and $2500 worth of 3d printer! Thanks in advance, Kurt

4 6

Re: [lxml] etree.HTML suddenly eats all RAM and sets all CPU to WAIT
by Knut Ole Sjøli March 8, 2013

March 8, 2013

This is what it looks like in nmon: http://imgur.com/htgmPM3 The Commit_AS and Active under Memory Stats are stable and very low before the "crash." Then after the "crash", they start growing fast and never stops. I'm wondering, could this be an Amazon AWS limit/restriction that is being put into effect? I'm running a small instance, and re-using a lot of sockets (20/second) to fetch HTML, download rate at about 250KB/s. Thanks for any input.

1 0

Re: [lxml] how to iterate over elements ONLY with CDATA in them?
by Tanim Islam March 8, 2013

March 8, 2013

Le 07/03/2013 19:39, Tanim Islam a écrit : > Certainly! I have XML data file that I access through the NPR API. Only > the <title> elements with CDATA in them contain the programs in the > podcast, whereas the other > <title> elements only contain descriptions of the program or other > information I do not need. > > I have attached the XML file that I hope clarifies my issue. Also, I > cannot ask NPR to change the data they send:) Hi, I’ve only skimmed the attached file but can’t you find another criteria for what parts are interesting? Maybe <title> elements that are in a <story> parent? This is easy to express in XPath: title_elements = root.xpath('//story/title') -- Simon Sapin Hi, Thanks, that did the trick! Tanim Islam

1 0

etree.HTML suddenly eats all RAM and sets all CPU to WAIT
by Knut Ole Sjøli March 8, 2013

March 8, 2013

I'm parsing about 20 docs/second with the code below, and it runs well — until suddenly, the RAM is flooded completely. Not incrementally, just all at once, after about five to ten minutes (dependent on doc/second load). Code: def parse_og(self, data): """ lxml parsing """ try: tree = etree.HTML( data ) m = tree.xpath("//meta[@property]") for i in m: y = i.attrib['property'] x = i.attrib['content'] self.rj[y] = x except Exception: print 'lxml error: ', sys.exc_info()[1:3] pass datais first 50kb of a HTML document. Is there some cleanup I should be doing? Or could this be a bug in lxml or the underlying C lib? Thank you. OS : Ubuntu 12.10 (AWS) Python : sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0) lxml.etree : (3, 1, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26)

1 0

Re: [lxml] how to iterate over elements ONLY with CDATA in them?
by Tanim Islam March 7, 2013

March 7, 2013

> How do I iterate, on an lxml generated XML tree, over elements ONLY with > the CDATA keyword in them? Is there a way I can extract the raw text in an > element -- ones with CDATA in them, for instance? Could you explain a bit *why* you would want to do this? Stefan Certainly! I have XML data file that I access through the NPR API. Only the <title> elements with CDATA in them contain the programs in the podcast, whereas the other <title> elements only contain descriptions of the program or other information I do not need. I have attached the XML file that I hope clarifies my issue. Also, I cannot ask NPR to change the data they send :)

3 2

how to iterate over elements ONLY with CDATA in them?
by Tanim Islam March 6, 2013

March 6, 2013

Hello: How do I iterate, on an lxml generated XML tree, over elements ONLY with the CDATA keyword in them? Is there a way I can extract the raw text in an element -- ones with CDATA in them, for instance? Tanim Islam

2 1

lxml 3.1.0 released
by Stefan Behnel March 4, 2013

March 4, 2013

Hi everyone, I just released lxml 3.1.0. This is a stable feature release that mainly adds a new incremental XML serialisation API, but also fixes a couple of bugs. Upgrading from 3.0.x should be smooth. The complete changelog follows below. Note that this release is called "3.1.0" instead of just "3.1" in order to simplify version sorting. You can get it from PyPI: http://pypi.python.org/pypi/lxml/3.1.0 The documentation is here: http://lxml.de/ Download: http://lxml.de/files/lxml-3.1.0.tgz Signature: http://lxml.de/files/lxml-3.1.0.tgz.asc GitHub: https://github.com/lxml/lxml/commit/f76cca898b7edefe91be584540883d158416e6cb This release was built using Cython 0.18. Note that the build no longer uses Cython, even if it is installed. Recompilation of the sources has to be requested explicitly with the setup.py option "--with-cython". If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.1.0 (2013-02-10) ================== Features added -------------- * GH#89: lxml.html.clean allows overriding the set of attributes that it considers 'safe'. Patch by Francis Devereux. Bugs fixed ---------- * LP#1104370: ``copy.copy(el.attrib)`` raised an exception. It now returns a copy of the attributes as a plain Python dict. * GH#95: When used with namespace prefixes, the ``el.find*()`` methods always used the first namespace mapping that was provided for each path expression instead of using the one that was actually passed in for the current run. * LP#1092521, GH#91: Fix undefined C symbol in Python runtimes compiled without threading support. Patch by Ulrich Seidl. Other changes ------------- 3.1beta1 (2012-12-21) ===================== Features added -------------- * New build-time option ``--with-unicode-strings`` for Python 2 that makes the API always return Unicode strings for names and text instead of byte strings for plain ASCII content. * New incremental XML file writing API ``etree.xmlfile()``. * E factory in lxml.objectify is callable to simplify the creation of tags with non-identifier names without having to resort to getattr(). Bugs fixed ---------- * When starting from a non-namespaced element in lxml.objectify, searching for a child without explicitly specifying a namespace incorrectly found namespaced elements with the requested local name, instead of restricting the search to non-namespaced children. * GH#85: Deprecation warnings were fixed for Python 3.x. * GH#33: lxml.html.fromstring() failed to accept bytes input in Py3. * LP#1080792: Static build of libxml2 2.9.0 failed due to missing file. Other changes ------------- * The externally useless class ``_ObjectifyElementMakerCaller`` was removed from the module API of lxml.objectify. * LP#1075622: lxml.builder is faster for adding text to elements with many children. Patch by Anders Hammarquist.

5 16

benchmarks using an Element freelist
by Stefan Behnel March 2, 2013

March 2, 2013

Hi, I had some fun during the last days implementing low-level object freelist support in Cython and gave it a try in lxml. It's implemented as a class decorator, so all you have to do is to add a "@cython.freelist(8)" declaration to get an 8 item freelist for the class. Even relatively short freelists tend to bring a lot as they hit typical iteration use cases, for example. For benchmarking, I initially used "Element.attrib", which is a property that instantiates a new object on each request (not cached in order to avoid reference cycles). Most use cases of this class consist of a one-shot operation to read some XML attributes, so it's a classical "create and throw away" thing, the ideal candidate for a freelist. The code that I ran through timeit is this: Setup: 'import lxml.etree as et; el = et.Element("tag", a="1", b="2")' Benchmarked code: 'el.attrib' Initial timings: 10000000 loops, best of 3: 0.0856 usec per loop Replacing "_Attrib(element)" by "_Attrib.__new__(_Attrib, element)": 10000000 loops, best of 3: 0.0781 usec per loop Adding only the freelist decorator to the _Attrib class: 10000000 loops, best of 3: 0.0737 usec per loop Combining both "__new__()" and the freelist: 10000000 loops, best of 3: 0.0608 usec per loop That gives an overall speedup of almost 30%, just by changing two lines of code. The main Element proxy class in lxml is another case where it really makes sense to apply this decorator. It gets instantiated whenever Python access to an XML element is requested, e.g. during iteration. Here are the timings for iterating over a large XML document. Benchmarked code: 'list(islice(tree.iter(), 10000000, 10000001))' Initial timings: 100 loops, best of 3: 15.6 msec per loop Enabling the freelist: 100 loops, best of 3: 13.2 msec per loop That's 15% faster, even all the way through a non-trivial tree iteration operation. And with just one additional declaration in the code. Note that this change will require Cython 0.19, which isn't currently close to release (we just recently released 0.18). So this change may take a bit of time to make it into an lxml release. Stefan

2 6