Mailman 3 November 2012 - lxml - The Python XML Toolkit

performance problem with lxml
by Martin Mueller Nov. 26, 2012

Nov. 26, 2012

I have used lxml to extract attribute values specifying lemma and pos tag from some 2000 TEI encoded texts. The program has to chew its way through ~160 million instances of the following type of w element <w lem="be" pos="vbz" reg="is" spe="is" xml:id="K000039_000-000220">is</w> The <w> are leaf nodes occurring at varying levels from the root, with a typical Xpath like TEI/text/body/div/div/p/w. The critical piece of code in the program went like this: filein= open(filename, 'r') text = filein.read() root = etree.fromstring(text) for element in root.iter(tei+'w'): lemma = element.get('lem') pos = element.get('pos') spelling = element.get('spe') entry= '\t'.join ((filenumber, lemma, pos, spelling)) eccodictionary[entry] += 1 filein.close() for eccoitem in sorted(eccodictionary): print >>fileout, eccoitem.encode('utf-8'), '\t', eccodictionary[eccoitem] I am using lxm-l2.3.4, which is part of the Enthought Python 2.7 distribution. It took more than twelve hours to execute, and I had to run it in three batches because the program would slow down after a while. I don't know whether this is an intrinsic scale problem or whether there are better ways of writing the code. I ran it on a desktop Macintosh with plenty of memory (20GB). I noticed that as the program went on, Python seemed to hog more and more memory, showing 400 MB after a few texts, then creeping up to 2.5 GB. In theory, each file is closed after it has been processed. In its three passes through different sets of files, its ouput files grew to 165MB, 153MB, and 39MB. The program clearly slowed down as Python's memory use increases. And I don't know why memory use increases so sharply. Nor do I know whether the slow-down is caused by the memory hogging. At the beginning it would process a file of one megabyte in one or two seconds. But when it had gone through 100 or so it seemed like it was slower by a factor of 3-5. I'll be grateful for any advice. I have no idea whether I have run up against my limits as a programmer (very likely), the limits of Python, or the limits of lxml. Martin Mueller Professor of English and Classics Northwestern University

3 2

Incorrect serialization with a default namespace
by Simon Sapin Nov. 23, 2012

Nov. 23, 2012

Hi, While investigating namespaces in the new incremental serializer, I also tested a bit the normal serializer, for comparison. It appears that the serialization is incorrect for null-namespace elements with a default namespace declaration: >>> print(etree.tostring(etree.Element('test', nsmap={None: 'ns'}))) <test xmlns="ns"/> Expected output: <test/> Since there is no way to express the null namespace when a default namespace is in scope, the default namespace in nsmap should be ignored. A more subtle case: >>> root = etree.Element('{ns}test', nsmap={None: 'ns'}) >>> etree.SubElement(root, 'a') >>> print(etree.tostring(root)) <test xmlns="ns"><a/></test> Expected output: <test xmlns="ns"><a xmlns=""/></test> Here xmlns="" is used to "undeclare" the parent’s default namespace. Is this a bug in lxml or in libxml2? Cheers, -- Simon Sapin

2 4

lxml not working against python 3.2.3 ?
by sang-suan gam Nov. 23, 2012

Nov. 23, 2012

hi All, i've installed lxml using pip for python3, but when i try to: "from lxml import etree", it failed with an exslt symbol not found: : [lab146]# p3 : Python 3.2.3 (default, Nov 21 2012, 18:28:16) : [GCC 4.5.2] on sunos5 : Type "help", "copyright", "credits" or "license" for more information. : >>> import lxml : >>> from lxml import etree : Traceback (most recent call last): : File "<stdin>", line 1, in <module> : ImportError: ld.so.1: python3: fatal: relocation error: file /usr/local/lib/python3.2/site-packages/lxml/etree.so: symbol exsltStrXpathCtxtRegister: referenced symbol not found : [lab146]# lsf /usr/lib/*xslt* : /usr/lib/libexslt.so@ /usr/lib/libxslt.so@ /usr/lib/llib-lexslt /usr/lib/llib-lxslt : /usr/lib/libexslt.so.0* /usr/lib/libxslt.so.1* /usr/lib/llib-lexslt.ln /usr/lib/llib-lxslt.ln : [lab146]# ll /usr/lib/*xslt* : lrwxrwxrwx 1 root root 15 Oct 15 16:22 /usr/lib/libexslt.so -> ./libexslt.so.0 : -r-xr-xr-x 1 root bin 134792 Oct 15 16:22 /usr/lib/libexslt.so.0 : lrwxrwxrwx 1 root root 14 Oct 15 16:22 /usr/lib/libxslt.so -> ./libxslt.so.1 : -r-xr-xr-x 1 root bin 346644 Oct 15 16:22 /usr/lib/libxslt.so.1 : -r--r--r-- 1 root bin 487 Oct 15 16:22 /usr/lib/llib-lexslt : -r--r--r-- 1 root bin 135186 Oct 15 16:22 /usr/lib/llib-lexslt.ln : -r--r--r-- 1 root bin 487 Oct 15 16:22 /usr/lib/llib-lxslt : -r--r--r-- 1 root bin 185413 Oct 15 16:22 /usr/lib/llib-lxslt.ln : [lab146]# : [lab146]# echo $LD_LIBRARY_PATH : /lib:/usr/lib:/usr/lib/openssl:/usr/X11/lib:/usr/sfw/lib:/usr/local/ssl/lib:/usr/local/lib:/usr/local/instantclient_11_2/32/lib : [lab146]# the "pip" install output showed that the etree.so was indeed linked against the libs ... : gcc -shared build/temp.solaris-2.11-i86pc.32bit-3.2/src/lxml/lxml.etree.o -L/usr/lib -lxslt -lexslt -lxml2 -lz : -lm -o build/lib.solaris-2.11-i86pc.32bit-3.2/lxml/etree.so are there some steps that i'm missing ... the python2 that was bundled with the system, works: : [lab146]# p2 : Python 2.6.4 (r264:75706, Oct 17 2011, 16:55:05) [C] on sunos5 : Type "help", "copyright", "credits" or "license" for more information. : >>> from lxml import etree : >>> : : [lab146]# pldd 21862 : 21862: python : /usr/lib/libc/libc_hwcap1.so.1 : /usr/lib/libpython2.6.so.1.0 : /lib/libdl.so.1 : /usr/lib/python2.6/lib-dynload/readline.so : /usr/lib/libreadline.so.5 : /lib/libm.so.2 : /usr/local/lib/libncurses.so.5.7 : /usr/local/lib/libintl.so.8.1.1 : /usr/gcc/4.5/lib/libgcc_s.so.1 : /usr/local/lib/libiconv.so.2.5.1 : /usr/lib/python2.6/vendor-packages/lxml/etree.so : /usr/lib/libxslt.so.1 : /usr/lib/libexslt.so.0 : /lib/libxml2.so.2 : /lib/libz.so.1 : /lib/libpthread.so.1 : /lib/libsocket.so.1 : /lib/libnsl.so.1 : /lib/libmd.so.1 : /lib/libmp.so.2 : /usr/lib/python2.6/lib-dynload/_fileio.so : /usr/lib/python2.6/lib-dynload/_functools.so : /usr/lib/python2.6/lib-dynload/time.so : /usr/lib/python2.6/lib-dynload/_collections.so : /usr/lib/python2.6/lib-dynload/operator.so : /usr/lib/python2.6/lib-dynload/_bytesio.so : /usr/lib/python2.6/lib-dynload/strop.so : [lab146]# thanks in advance for all help, cheers, sam

2 1

new API for incremental XML generation
by Stefan Behnel Nov. 21, 2012

Nov. 21, 2012

Hi, I've implemented the new incremental XML serialisation API. It's not documented yet and also not entirely tested. It basically works like this: with etree.xmlfile("somefile.xml", encoding='utf-8') as xf: xf.write_declaration(standalone=True) xf.write_doctype('<!DOCTYPE root SYSTEM "some.dtd">') # generate an element (the root element) with xf.Element('root'): # write a complete Element xf.write(etree.Element('test')) # generate more Elements, e.g. through iterparse for element in generate_some_elements(): # serialise generated elements into the XML file xf.write(element) You can find some more examples here: https://github.com/lxml/lxml/blob/master/src/lxml/tests/test_incremental_xm… It also works with BytesIO and other file-like objects. The implementation is here: https://github.com/lxml/lxml/blob/ccd5d94d2c49cb67b1224c6c682f4ee86bfba46b/… It's really straight forward to use. Since I'd like to get it into the next lxml release, I'd be happy about any contributions in terms of real world testing, a couple of examples for the tutorial section and general feedback about the API and its usability for different purposes. Thanks for any comments. Stefan

2 1

create xml file incrementally
by James Housden Nov. 18, 2012

Nov. 18, 2012

Hello, I have a large xml file that I need to modify and then store as a new xml file. The file has a structure similar to <root> <header> <txt>header txt</txt> </header> <record> <field1>1.0</field1> <subrecord> <field2>A1</field2> <field3>C1</field3> <subrecord> </record> <record> <field1>1.0</field1> <subrecord> <field2>A2</field2> <field3>C3</field3> <subrecord> </record> <record> <field1>1.0</field1> <subrecord> <field2>A4</field2> <field3>B</field3> <subrecord> </record> </root> I would like to modify the contents of the field3 tags. Now, due to the file size, I cannot load the complete document into memory and so I intend to use 'iterparse'. Traversing the document and updating the fields is no problem. What I am not sure about is how to write the modified data to a new xml file. The root tag is only complete when I have processed the complete file. What I need to do is write the start of the root tag (<root>) then write the header and the records and finally the end of root tag (</root>). Is there functionality in lxml to do this or should I use standard python writes for the initial <root> and final >/root>?

5 14

"from lxml import etree" leads to ImportError on Lxml-3.0.1
by L. Dubyzu Nov. 8, 2012

Nov. 8, 2012

When i try "from lxml import etree" is give be the following ImportError: >>> from lxml import etree Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /share/HDA_DATA/.qpkg/Python/lib/python2.7/site-packages/lxml-3.0.1- py2.7-linux-armv5tel.egg/lxml/etree.so: undefined symbol: __xmlStructuredErrorContext I've tried downgrading to lxml 3.0, 2.3.6, and 2.3.5 - its the same error. Any suggestions on what needs to be adjusted?

3 10

XMLSchema validator problem with [0-9]{0,n}|TEXT
by Linux - Junior Polegato Nov. 8, 2012

Nov. 8, 2012

Hi! I have had problems with XML Schema validation with [0-9]{0,n} condition. Can you help me? I have many xsd files with conditions like [0-9]{0,n}|TEXT. I need investigate each file and adjust by hand? I will first ask, then will post the output and after will post the example code: --------- Questions --------- Why doesn't it work in 0.3 1.0 2.3? Why does it work in 3 and 4? --------- Output --------- 0. pattern: [0-9]{0,4}|TEXT 0.0. string: "" => valid: True 0.1. string: "123" => valid: True 0.2. string: "12345" => valid: False 0.3. string: "TEXT" => valid: False 0.4. string: "TEST" => valid: False 1. pattern: TEXT|[0-9]{0,4} 1.0. string: "" => valid: False 1.1. string: "123" => valid: True 1.2. string: "12345" => valid: False 1.3. string: "TEXT" => valid: True 1.4. string: "TEST" => valid: False 2. pattern: [0-9]{0,4}|TEXT| 2.0. string: "" => valid: True 2.1. string: "123" => valid: True 2.2. string: "12345" => valid: False 2.3. string: "TEXT" => valid: False 2.4. string: "TEST" => valid: False 3. pattern: [0-9]{0,4}|TEXT|1234 3.0. string: "" => valid: True 3.1. string: "123" => valid: True 3.2. string: "12345" => valid: False 3.3. string: "TEXT" => valid: True 3.4. string: "TEST" => valid: False 4. pattern: TEXT|[0-9]{0,4}| 4.0. string: "" => valid: True 4.1. string: "123" => valid: True 4.2. string: "12345" => valid: False 4.3. string: "TEXT" => valid: True 4.4. string: "TEST" => valid: False --------- Code --------- #!/usr/bin/env python # -*- coding: utf-8 -*- import lxml.etree DEBUG = False for i, p in enumerate(['[0-9]{0,4}|TEXT', 'TEXT|[0-9]{0,4}', '[0-9]{0,4}|TEXT|', '[0-9]{0,4}|TEXT|1234', 'TEXT|[0-9]{0,4}|']): print '%i. pattern: %s' % (i, p) xsd = lxml.etree.fromstring('<?xml version="1.0" encoding="UTF-8"?>' '<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" attributeFormDefault="unqualified">' ' <xs:simpleType name="TTest">' ' <xs:restriction base="xs:string">' ' <xs:whiteSpace value="preserve"/>' ' <xs:pattern value="' + p + '"/>' ' </xs:restriction>' ' </xs:simpleType>' ' <xs:element name="Test" type="TTest"/>' '</xs:schema>') for j, s in enumerate(['', '123', '12345', 'TEXT', 'TEST']): xml = lxml.etree.fromstring('<?xml version="1.0" encoding="utf-8"?><Test>' + s + '</Test>') validator = lxml.etree.XMLSchema(xsd) valid = validator(xml) print '%i.%i. string: "%s" => valid: %s' % (i, j, s, valid) if not valid and DEBUG: for error in validator.error_log: print "message :", error.message print "domain :", error.domain print "type :", error.type print "level :", error.level print "line :", error.line print "column :", error.column print "filename :", error.filename print "domain_name:", error.domain_name print "type_name :", error.type_name print "level_name :", error.level_name print ''

3 4

Please see http://stackoverflow.com/questions/12591/using-an-xml-catalog-with-pythons-lxml
by Jean-Claude Magras Nov. 8, 2012

Nov. 8, 2012

Hi, I don't want to be rude but take a look at this: http://stackoverflow.com/questions/12591/using-an-xml-catalog-with-pythons-… I have tried for an internet disconnected machine to lxml parse with xsd validation even using no_network=True I still get an insistence that "connection refused". I do not have lots of experience with python and most of the code is from lxml.de . Sincerely, JC Magras

2 1

Re: [lxml] no xpath evaluation in attributes of xslt extension elements?
by Eric Levy Nov. 5, 2012

Nov. 5, 2012

Well, maybe. Unfortunately, I can't test it in my application because of a second problem. Ultimately, I would like to create an extension element that performs an XPath query on another document and inserts the result of the query into the output of the transformation. However, XPath query results come back as different types from what output_parent.append() expects, so simply passing the query result to the transformation result breaks the script. The documentation suggests that read-only types can be made into regular types by calling deepcopy(), but in my case this has no effect. Is there a solution to this new problem? ----- Original Message ----- Date: Thu, 01 Nov 2012 14:24:43 +0100 From: Stefan Behnel <stefan_ml(a)behnel.de> Subject: [lxml] no xpath evaluation in attributes of xslt extension elements? To: lxml mailing list <lxml(a)lxml.de> Eric Levy, 29.10.2012 08:31: > I'm back with another question about lxml. > > I am tinkering with the extension elements feature of the XSLT engine. > > I notice that if I try to give an XPath expression enclosed in {} as an attribute value in my own element type, the contents of the {} are not evaluated as an XPath expression, even though they would be if the element were not an extension. > > For example, in normal XSLT processing: > > <xslt:apply-templates match="foo"> > <bar attrib-of-bar="{@attrib-of-foo}" /> > </xslt:apply-templates> > > This would create a <bar/> element with the attrib-of-bar attribute set to the value of the attrib-of-foo attribute of foo. > > However, if <bar/> is lxml extension element, then the {} contents are not evaluated, but rather the entire attribute value is returned verbatim. Does this help? http://lxml.de/extensions.html#applying-xsl-templates Stefan

1 0

ElementTree as secondary input to transformation
by Eric Levy Nov. 1, 2012

Nov. 1, 2012

Hi all, I wish for two separate ElementTree objects to serve as input for the same XSLT transformation. As I understand, XSLT in general supports secondary input documents by way of the XPath doc() function. Normally the function represents an XML tree from a named file. It appears that lxml supports custom resolvers in case the document is not simply a file on disk as named. Further, it appears that a custom resolver may resolve a URI as string, another filename, or a file object. What about the case of the secondary document being an ElementTree in memory? Is this case supported? Thanks. Sincerely, Eric Levy

2 1