Mailman 3 January 2012 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 11 Apr '23

11 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 10 Apr '23

10 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) 03 May '13

03 May '13

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

[lxml-dev] confusing xpath performance characteristics
by jholg＠gmx.de 24 Aug '12

24 Aug '12

Hi, I ran into some performance characteristics of lxml/libxml2 xpath that I find rather confusing: I try to find the @type attribute of a certain element in an XML Schema (which contains lots of complexType definitions with lots of elements in them; unfortunately I can't post the schema): >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.095885038375854492, 0.096823930740356445, 0.096174955368041992] So I think I'm being smart and give a little more path information - reckoning that this should *improve* performance: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.1770780086517334, 0.1775970458984375, 0.17748594284057617] Hm. Performance degrades slightly. I'm adding even more of the path to where my desired elements live in the schema: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [103.79744100570679, 103.83671712875366, 103.61817717552185] What??? >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('/ae/data/pydev/hjoukl/NDM/SVN_CO/TRUNK/ndm/reference/xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType/*/xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.044407129287719727, 0.044126987457275391, 0.044229030609130859] >>> Ok, this version's better than my naive approach, which seems logical to me. But why would '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' perform drastically slower than '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' ? libxml2 problem? Running the same xpaths in Oxygen I don't notice performance differences (can't profile this). Holger -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

2 5

Cython errors building from git on Mac OS X
by Laurence Rowe 15 Feb '12

15 Feb '12

I'm having trouble building lxml from a git clone on Mac OS X 10.6 with python.org python 2.7 or my own compiled python 2.6. Any idea what might be causing the errors below? Thanks in advance, Laurence $ python2.7 setup.py build Building lxml version 2.4.dev. Building with Cython 0.15.1. Using build configuration of libxslt 1.1.24 Building against libxml2/libxslt in the following directory: /usr/local/lib running build running build_py creating build/lib.macosx-10.6-intel-2.7 creating build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/__init__.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/_elementpath.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/builder.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/cssselect.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/doctestcompare.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/ElementInclude.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/pyclasslookup.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/sax.py -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/usedoctest.py -> build/lib.macosx-10.6-intel-2.7/lxml creating build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/__init__.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/_dictmixin.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/_diffcommand.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/_html5builder.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/_setmixin.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/builder.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/clean.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/defs.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/diff.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/ElementSoup.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/formfill.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/html5parser.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/soupparser.py -> build/lib.macosx-10.6-intel-2.7/lxml/html copying src/lxml/html/usedoctest.py -> build/lib.macosx-10.6-intel-2.7/lxml/html creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron copying src/lxml/isoschematron/__init__.py -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron copying src/lxml/etreepublic.pxd -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/tree.pxd -> build/lib.macosx-10.6-intel-2.7/lxml copying src/lxml/etree_defs.h -> build/lib.macosx-10.6-intel-2.7/lxml creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/rng copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/rng creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl creating build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.macosx-10.6-intel-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1 running build_ext cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c Error compiling Cython file: ------------------------------------------------------------ ... cdef _Element _node cdef _node_to_node_function _next_element cdef _MultiTagMatcher _matcher @cython.final cdef _initTagMatcher(self, tags): ^ ------------------------------------------------------------ src/lxml/lxml.etree.pyx:2501:9: The final compiler directive is not allowed in function scope Error compiling Cython file: ------------------------------------------------------------ ... def __iter__(self): return self @cython.final cdef int _storeNext(self, _Element node) except -1: ^ ------------------------------------------------------------ src/lxml/lxml.etree.pyx:2508:9: The final compiler directive is not allowed in function scope Error compiling Cython file: ------------------------------------------------------------ ... ] cimport cython cimport tree, python, config from tree cimport xmlDoc, xmlNode, xmlAttr, xmlNs, _isElement, _getNs ^ ------------------------------------------------------------ src/lxml/lxml.etree.pyx:43:0: 'xmlDoc.pxd' not found Error compiling Cython file: ------------------------------------------------------------ ... ] # snipped hundreds of similar errors.

3 3

from xml to sql
by Piotr Oh 25 Jan '12

25 Jan '12

Hi I'm not a programmer, so please be patient I just need some scripting done and my choice is python+lxml. The problem to solve is. 1. xml file with some data is exported from our ERP system using third party tools. It has xml schema. 2. It is intended to import to another system (sql/firebird) 3. Between the export/import process I'd like to validate the xml (using xml schema) 4. Than I need to import it to another system: 4a. check the values of the corresponding data in xml and SQL database, compare them, do some action, write to log etc 4b. put them to SQL database, update (update, insert new) Validating is simple (point 3). then I need to traverse the xml, record by record, do something with each and translate into sql query. >From this point of view IMHO what is the right way: use lxml.objectify or etree? I don't care about efficiency. Instead it should be as simple as possible (to modify, read etc) Could you give me some hints here? Some programming idioms, patterns or just basic pseudocode to follow? Regards Piotr

4 14

Re: [lxml] encoding question
by Tim Arnold 24 Jan '12

24 Jan '12

hi, Just in case my question about replacing characters with entities got lost over the weekend, I'm resending. I'm no expert in this area and want to make sure I'm not doing anything stupid of dangerous. Any help, ideas, comments appreciated. thanks, --Tim Arnold I have to follow a specification for producing xhtml files. The original files are in cp1252 encoding and I must reencode them to utf-8. Also, I have to replace certain characters with html entities. I think I've got this right, but I'd like to hear if there's something I'm doing that is dangerous or wrong. Please see the appended code, and thanks for any comments. I have two functions, translate (replaces high characters with entities) and reencode (um, reencodes): --------------------------------- import codecs, StringIO from lxml import etree high_chars = { 0x2014:'—', # 'EM DASH', 0x2013:'–', # 'EN DASH', 0x0160:'&Scaron;',# 'LATIN CAPITAL LETTER S WITH CARON', 0x201d:'”', # 'RIGHT DOUBLE QUOTATION MARK', 0x201c:'“', # 'LEFT DOUBLE QUOTATION MARK', 0x2019:"’", # 'RIGHT SINGLE QUOTATION MARK', 0x2018:"‘", # 'LEFT SINGLE QUOTATION MARK', 0x2122:'™', # 'TRADE MARK SIGN', 0x00A9:'©', # 'COPYRIGHT SYMBOL', } def translate(string): s = '' for c in string: if ord(c) in high_chars: c = high_chars.get(ord(c)) s += c return s def reencode(filename, in_encoding='cp1252',out_encoding='utf-8'): with codecs.open(filename,encoding=in_encoding) as f: s = f.read() sio = StringIO.StringIO(translate(s)) parser = etree.HTMLParser(encoding=in_encoding) tree = etree.parse(sio, parser) result = etree.tostring(tree.getroot(), method='html', pretty_print=True, encoding=out_encoding) with open(filename,'wb') as f: f.write(result) if __name__ == '__main__': fname = 'mytest.htm' reencode(fname)

1 0