Mailman 3 November 2012 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 10 Apr '23

10 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 09 Apr '23

09 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 30 Aug '22

30 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) 03 May '13

03 May '13

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

Apache2 crashes
by Brieuc SCHAFF 12 Feb '13

12 Feb '13

Hi, We are using lxml in several of our applications with Python 2.6 and from time to time, the application stops responding after a segmentation fault error, and this kind of backtrace: *Jul 1 15:35:01 server4 httpd: *** glibc detected *** /usr/sbin/apache2: malloc(): smallbin double linked list corrupted: 0x00007f583a03f500 **** *Jul 1 15:35:01 server4 httpd: ======= Backtrace: =========* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f58376c2806]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(+0x7bb39)[0x7f58376c6b39]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(__libc_malloc+0x6e)[0x7f58376c77de]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlInitParserCtxt+0x317)[0x7f58343f2887]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libxml2.so.2(xmlNewParserCtxt+0x2c)[0x7f58343f2acc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x54eda)[0x7f5818db6eda]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xddcf3)[0x7f5818e3fcf3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xab5ee)[0x7f5818e0d5ee]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb7d8e)[0x7f5818e19d8e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xb827e)[0x7f5818e1a27e]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f582face030]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5a98)[0x7f582face928]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f582facc96f]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f582facfd60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f582fa55d60]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f582fa39cef]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f582fa8e6dc]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f582fa282e3]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f582fac8193] * *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f582fe9bfdb]* *Jul 1 15:35:01 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f582fea0eed]* *Jul 1 15:35:01 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f5837c14eb3]* *Jul 1 15:35:01 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f58379d49ca]* *Jul 1 15:35:01 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f5837731cdd]* ----------------- *Jul 11 10:44:48 server4 httpd: *** glibc detected *** /usr/sbin/apache2: munmap_chunk(): invalid pointer: 0x00007f3968b8cb68 **** *Jul 11 10:44:48 server4 httpd: ======= Backtrace: =========* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(+0x77806)[0x7f39789dd806]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlResetError+0x22)[0x7f39755038e2]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlRaiseError+0x237)[0x7f3975504e07]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(__xmlSimpleError+0x6b)[0x7f397550543b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputBufferGrow+0x16b)[0x7f397553271b]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParserInputGrow+0x4c)[0x7f397550739c]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x3bd52)[0x7f397550bd52]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(xmlParseDocument+0x4b8)[0x7f39755204c8]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libxml2.so.2(+0x505c5)[0x7f39755205c5]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x8fab0)[0x7f3958ad8ab0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2daa0)[0x7f3958a76aa0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0x2ddc9)[0x7f3958a76dc9]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc26f0)[0x7f3958b0b6f0]* *Jul 11 10:44:48 server4 httpd: /usr/lib/python2.6/dist-packages/lxml/etree.so(+0xc431e)[0x7f3958b0d31e]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x51a0)[0x7f397063b030]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x3adf)[0x7f397063996f]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x920)[0x7f397063cd60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x7dd60)[0x7f39705c2d60]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0x61cef)[0x7f39705a6cef]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(+0xb66dc)[0x7f39705fb6dc]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyObject_Call+0x53)[0x7f39705952e3]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43)[0x7f3970635193] * *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x12fdb)[0x7f3970a08fdb]* *Jul 11 10:44:48 server4 httpd: /usr/lib/apache2/modules/mod_wsgi.so(+0x17eed)[0x7f3970a0deed]* *Jul 11 10:44:48 server4 httpd: /usr/lib/libapr-1.so.0(+0x29eb3)[0x7f3978f2feb3]* *Jul 11 10:44:48 server4 httpd: /lib/libpthread.so.0(+0x69ca)[0x7f3978cef9ca]* *Jul 11 10:44:48 server4 httpd: /lib/libc.so.6(clone+0x6d)[0x7f3978a4ccdd]* Stopping apache and restarting it fix the issue, but we have no idea why this is happening. We use Unbuntu 10.10's package of lxml, which is version 2.2.4. *>>> print("%-20s: %s" % ('Python', sys.version_info))* *Python : (2, 6, 5, 'final', 0)* *>>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))* *lxml.etree : (2, 2, 4, 0)* *>>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))* *libxml used : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))* *libxml compiled : (2, 7, 6)* *>>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))* *libxslt used : (1, 1, 26)* *>>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))* *libxslt compiled : (1, 1, 26)* * * Do you have any idea of what could cause these issues ? I wonder if upgrading to a newer version of the library could change anything. Thanks in advance for any answer you could provide. Regards, B. _____________________ Brieuc SCHAFF +33609898260 _____________________

2 3

lxml without libxslt
by Andriy Kornatskyy 02 Dec '12

02 Dec '12

Is it possible to build lxml without libxslt installed? Thanks. Andriy Kornatskyy

2 7

Baffled by documentation of namespace in XPath
by Martin Mueller 29 Nov '12

29 Nov '12

I have always found it difficult to wrap my head around the details of namespaces in XML processing, but I am completely baffled by the discussion and examples of namespaces in the XPath section. Consider the following minimal TEI document: <?xml version="1.0" encoding="UTF-8"?> <?xml-model href="http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_bare.r ng" schematypens="http://relaxng.org/ns/structure/1.0"?> <TEI xmlns="http://www.tei-c.org/ns/1.0"> <teiHeader> <fileDesc> <titleStmt> <title>A minimal TEI document</title> </titleStmt> <publicationStmt> <p>unpublished</p> </publicationStmt> <sourceDesc> <p>born digital</p> </sourceDesc> </fileDesc> </teiHeader> <text> <front> <div><head>Preface</head> <p>This is the preface</p></div> </front> <body> <div><head>Chapter 1</head> <div><head>Subsection 1.1</head><p>The text of 1.1</p></div> <div><head>Subsection 1.2</head><p>The text of 1.2</p></div> </div> </body> </text> </TEI> If I process this with xquery, I need to have a namespace declaration of the type declare namespace tei = "http://www.tei-c.org/ns/1.0"; The miniscript xquery version "1.0"; declare namespace tei = "http://www.tei-c.org/ns/1.0"; let $text := doc('/users/martin/dropbox/learnpython.txt/bareguineapig.xml') return $text/tei:TEI//tei:front//tei:p will return <?xml version="1.0" encoding="UTF-8"?> <p xmlns="http://www.tei-c.org/ns/1.0">This is the preface</p> What is the equivalent for this normal case of an XML document? The documentation gives the following XML fragment <a:foo xmlns:a="http://codespeak.net/ns/test1"... xmlns:b="http://codespeak.net/ns/test2"> <b:bar>Text</b:bar> </a:foo> I've never seen XML documents that use namespace prefixes in the closing tags, and I can't figure out how to apply the information from the documentation to my case. I know how to process the document without a namespace. For instance f = '/users/martin/dropbox/learnpython.txt/bareguineapignonamespace.xml' tree = etree.parse(f) r = tree.xpath('/TEI/text/front') for element in r[0].iter('p'): print element.text will print out "This is the preface." But what do I need to do to get the same result when the TEI root element contains the tei namespace? I also wonder whether there is an error in this code: r = doc.xpath('/t:foo/b:bar', namespaces={'t': 'http://codespeak.net/ns/test1', 'b': 'http://codespeak.net/ns/test2'} Shouldn't the 't' be 'a'? It doesn't seem to affect the way the code works, though . Martin Mueller Professor of English and Classics Northwestern University

4 5

Possible bug in getpath
by Ahmed 26 Nov '12

26 Nov '12

Hi All, I'm trying to use getpath() to get the absolute xpath for an element in a tree. This works fine when all namespaces have consistent prefixes in the document. However when I deal with a document in which the same namespace can have multiple prefixes I get wrong path. I wrote this little script to re-produce the error. #!/usr/bin/env python from lxml import etree from StringIO import StringIO def generate_sample_tree_with_multiple_prefixes(): root = etree.Element('root', nsmap={'a': 'http://a.b/c'}) for x in range(10): ns = "http://ns.org/ns/" prefix = 'l_%s' % x node = etree.SubElement(root, "{%s}l1" % ns, nsmap={prefix: ns}) node.set('id', 'id_%s' % x) return etree.parse(StringIO(etree.tostring(root))) def generate_sample_tree_with_single_prefixes(): root = etree.Element('root', nsmap={'a': 'http://a.b/c'}) for x in range(10): ns = "http://ns.org/ns/" prefix = 'ns' node = etree.SubElement(root, "{%s}l1" % ns, nsmap={prefix: ns}) node.set('id', 'id_%s' % x) return etree.parse(StringIO(etree.tostring(root))) def test_multiple(): tree = generate_sample_tree_with_multiple_prefixes() r = tree.getroot() for l1 in r.iterchildren(): path = tree.getpath(l1) ret = tree.xpath(path, namespaces=l1.nsmap) assert len(ret) == 1, "Multiple elements returned: %s" % ret assert l1 == ret[0], "It's not the same" def test_single(): tree = generate_sample_tree_with_single_prefixes() r = tree.getroot() for l1 in r.iterchildren(): path = tree.getpath(l1) ret = tree.xpath(path, namespaces=l1.nsmap) assert len(ret) == 1, "Multiple elements returned" assert l1 == ret[0], "It's not the same" if __name__ == '__main__': test_single() test_multiple() thanks -- -Ahmed

2 2

performance problem with lxml
by Martin Mueller 25 Nov '12

25 Nov '12

I have used lxml to extract attribute values specifying lemma and pos tag from some 2000 TEI encoded texts. The program has to chew its way through ~160 million instances of the following type of w element <w lem="be" pos="vbz" reg="is" spe="is" xml:id="K000039_000-000220">is</w> The <w> are leaf nodes occurring at varying levels from the root, with a typical Xpath like TEI/text/body/div/div/p/w. The critical piece of code in the program went like this: filein= open(filename, 'r') text = filein.read() root = etree.fromstring(text) for element in root.iter(tei+'w'): lemma = element.get('lem') pos = element.get('pos') spelling = element.get('spe') entry= '\t'.join ((filenumber, lemma, pos, spelling)) eccodictionary[entry] += 1 filein.close() for eccoitem in sorted(eccodictionary): print >>fileout, eccoitem.encode('utf-8'), '\t', eccodictionary[eccoitem] I am using lxm-l2.3.4, which is part of the Enthought Python 2.7 distribution. It took more than twelve hours to execute, and I had to run it in three batches because the program would slow down after a while. I don't know whether this is an intrinsic scale problem or whether there are better ways of writing the code. I ran it on a desktop Macintosh with plenty of memory (20GB). I noticed that as the program went on, Python seemed to hog more and more memory, showing 400 MB after a few texts, then creeping up to 2.5 GB. In theory, each file is closed after it has been processed. In its three passes through different sets of files, its ouput files grew to 165MB, 153MB, and 39MB. The program clearly slowed down as Python's memory use increases. And I don't know why memory use increases so sharply. Nor do I know whether the slow-down is caused by the memory hogging. At the beginning it would process a file of one megabyte in one or two seconds. But when it had gone through 100 or so it seemed like it was slower by a factor of 3-5. I'll be grateful for any advice. I have no idea whether I have run up against my limits as a programmer (very likely), the limits of Python, or the limits of lxml. Martin Mueller Professor of English and Classics Northwestern University

3 2

Incorrect serialization with a default namespace
by Simon Sapin 23 Nov '12

23 Nov '12

Hi, While investigating namespaces in the new incremental serializer, I also tested a bit the normal serializer, for comparison. It appears that the serialization is incorrect for null-namespace elements with a default namespace declaration: >>> print(etree.tostring(etree.Element('test', nsmap={None: 'ns'}))) <test xmlns="ns"/> Expected output: <test/> Since there is no way to express the null namespace when a default namespace is in scope, the default namespace in nsmap should be ignored. A more subtle case: >>> root = etree.Element('{ns}test', nsmap={None: 'ns'}) >>> etree.SubElement(root, 'a') >>> print(etree.tostring(root)) <test xmlns="ns"><a/></test> Expected output: <test xmlns="ns"><a xmlns=""/></test> Here xmlns="" is used to "undeclare" the parent’s default namespace. Is this a bug in lxml or in libxml2? Cheers, -- Simon Sapin

2 4

lxml not working against python 3.2.3 ?
by sang-suan gam 23 Nov '12

23 Nov '12

hi All, i've installed lxml using pip for python3, but when i try to: "from lxml import etree", it failed with an exslt symbol not found: : [lab146]# p3 : Python 3.2.3 (default, Nov 21 2012, 18:28:16) : [GCC 4.5.2] on sunos5 : Type "help", "copyright", "credits" or "license" for more information. : >>> import lxml : >>> from lxml import etree : Traceback (most recent call last): : File "<stdin>", line 1, in <module> : ImportError: ld.so.1: python3: fatal: relocation error: file /usr/local/lib/python3.2/site-packages/lxml/etree.so: symbol exsltStrXpathCtxtRegister: referenced symbol not found : [lab146]# lsf /usr/lib/*xslt* : /usr/lib/libexslt.so@ /usr/lib/libxslt.so@ /usr/lib/llib-lexslt /usr/lib/llib-lxslt : /usr/lib/libexslt.so.0* /usr/lib/libxslt.so.1* /usr/lib/llib-lexslt.ln /usr/lib/llib-lxslt.ln : [lab146]# ll /usr/lib/*xslt* : lrwxrwxrwx 1 root root 15 Oct 15 16:22 /usr/lib/libexslt.so -> ./libexslt.so.0 : -r-xr-xr-x 1 root bin 134792 Oct 15 16:22 /usr/lib/libexslt.so.0 : lrwxrwxrwx 1 root root 14 Oct 15 16:22 /usr/lib/libxslt.so -> ./libxslt.so.1 : -r-xr-xr-x 1 root bin 346644 Oct 15 16:22 /usr/lib/libxslt.so.1 : -r--r--r-- 1 root bin 487 Oct 15 16:22 /usr/lib/llib-lexslt : -r--r--r-- 1 root bin 135186 Oct 15 16:22 /usr/lib/llib-lexslt.ln : -r--r--r-- 1 root bin 487 Oct 15 16:22 /usr/lib/llib-lxslt : -r--r--r-- 1 root bin 185413 Oct 15 16:22 /usr/lib/llib-lxslt.ln : [lab146]# : [lab146]# echo $LD_LIBRARY_PATH : /lib:/usr/lib:/usr/lib/openssl:/usr/X11/lib:/usr/sfw/lib:/usr/local/ssl/lib:/usr/local/lib:/usr/local/instantclient_11_2/32/lib : [lab146]# the "pip" install output showed that the etree.so was indeed linked against the libs ... : gcc -shared build/temp.solaris-2.11-i86pc.32bit-3.2/src/lxml/lxml.etree.o -L/usr/lib -lxslt -lexslt -lxml2 -lz : -lm -o build/lib.solaris-2.11-i86pc.32bit-3.2/lxml/etree.so are there some steps that i'm missing ... the python2 that was bundled with the system, works: : [lab146]# p2 : Python 2.6.4 (r264:75706, Oct 17 2011, 16:55:05) [C] on sunos5 : Type "help", "copyright", "credits" or "license" for more information. : >>> from lxml import etree : >>> : : [lab146]# pldd 21862 : 21862: python : /usr/lib/libc/libc_hwcap1.so.1 : /usr/lib/libpython2.6.so.1.0 : /lib/libdl.so.1 : /usr/lib/python2.6/lib-dynload/readline.so : /usr/lib/libreadline.so.5 : /lib/libm.so.2 : /usr/local/lib/libncurses.so.5.7 : /usr/local/lib/libintl.so.8.1.1 : /usr/gcc/4.5/lib/libgcc_s.so.1 : /usr/local/lib/libiconv.so.2.5.1 : /usr/lib/python2.6/vendor-packages/lxml/etree.so : /usr/lib/libxslt.so.1 : /usr/lib/libexslt.so.0 : /lib/libxml2.so.2 : /lib/libz.so.1 : /lib/libpthread.so.1 : /lib/libsocket.so.1 : /lib/libnsl.so.1 : /lib/libmd.so.1 : /lib/libmp.so.2 : /usr/lib/python2.6/lib-dynload/_fileio.so : /usr/lib/python2.6/lib-dynload/_functools.so : /usr/lib/python2.6/lib-dynload/time.so : /usr/lib/python2.6/lib-dynload/_collections.so : /usr/lib/python2.6/lib-dynload/operator.so : /usr/lib/python2.6/lib-dynload/_bytesio.so : /usr/lib/python2.6/lib-dynload/strop.so : [lab146]# thanks in advance for all help, cheers, sam

2 1

new API for incremental XML generation
by Stefan Behnel 21 Nov '12

21 Nov '12

Hi, I've implemented the new incremental XML serialisation API. It's not documented yet and also not entirely tested. It basically works like this: with etree.xmlfile("somefile.xml", encoding='utf-8') as xf: xf.write_declaration(standalone=True) xf.write_doctype('<!DOCTYPE root SYSTEM "some.dtd">') # generate an element (the root element) with xf.Element('root'): # write a complete Element xf.write(etree.Element('test')) # generate more Elements, e.g. through iterparse for element in generate_some_elements(): # serialise generated elements into the XML file xf.write(element) You can find some more examples here: https://github.com/lxml/lxml/blob/master/src/lxml/tests/test_incremental_xm… It also works with BytesIO and other file-like objects. The implementation is here: https://github.com/lxml/lxml/blob/ccd5d94d2c49cb67b1224c6c682f4ee86bfba46b/… It's really straight forward to use. Since I'd like to get it into the next lxml release, I'd be happy about any contributions in terms of real world testing, a couple of examples for the tutorial section and general feedback about the API and its usability for different purposes. Thanks for any comments. Stefan

2 1

create xml file incrementally
by James Housden 18 Nov '12

18 Nov '12

Hello, I have a large xml file that I need to modify and then store as a new xml file. The file has a structure similar to <root> <header> <txt>header txt</txt> </header> <record> <field1>1.0</field1> <subrecord> <field2>A1</field2> <field3>C1</field3> <subrecord> </record> <record> <field1>1.0</field1> <subrecord> <field2>A2</field2> <field3>C3</field3> <subrecord> </record> <record> <field1>1.0</field1> <subrecord> <field2>A4</field2> <field3>B</field3> <subrecord> </record> </root> I would like to modify the contents of the field3 tags. Now, due to the file size, I cannot load the complete document into memory and so I intend to use 'iterparse'. Traversing the document and updating the fields is no problem. What I am not sure about is how to write the modified data to a new xml file. The root tag is only complete when I have processed the complete file. What I need to do is write the start of the root tag (<root>) then write the header and the records and finally the end of root tag (</root>). Is there functionality in lxml to do this or should I use standard python writes for the initial <root> and final >/root>?

5 14

"from lxml import etree" leads to ImportError on Lxml-3.0.1
by L. Dubyzu 08 Nov '12

08 Nov '12

When i try "from lxml import etree" is give be the following ImportError: >>> from lxml import etree Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /share/HDA_DATA/.qpkg/Python/lib/python2.7/site-packages/lxml-3.0.1- py2.7-linux-armv5tel.egg/lxml/etree.so: undefined symbol: __xmlStructuredErrorContext I've tried downgrading to lxml 3.0, 2.3.6, and 2.3.5 - its the same error. Any suggestions on what needs to be adjusted?

3 10

XMLSchema validator problem with [0-9]{0,n}|TEXT
by Linux - Junior Polegato 08 Nov '12

08 Nov '12

Hi! I have had problems with XML Schema validation with [0-9]{0,n} condition. Can you help me? I have many xsd files with conditions like [0-9]{0,n}|TEXT. I need investigate each file and adjust by hand? I will first ask, then will post the output and after will post the example code: --------- Questions --------- Why doesn't it work in 0.3 1.0 2.3? Why does it work in 3 and 4? --------- Output --------- 0. pattern: [0-9]{0,4}|TEXT 0.0. string: "" => valid: True 0.1. string: "123" => valid: True 0.2. string: "12345" => valid: False 0.3. string: "TEXT" => valid: False 0.4. string: "TEST" => valid: False 1. pattern: TEXT|[0-9]{0,4} 1.0. string: "" => valid: False 1.1. string: "123" => valid: True 1.2. string: "12345" => valid: False 1.3. string: "TEXT" => valid: True 1.4. string: "TEST" => valid: False 2. pattern: [0-9]{0,4}|TEXT| 2.0. string: "" => valid: True 2.1. string: "123" => valid: True 2.2. string: "12345" => valid: False 2.3. string: "TEXT" => valid: False 2.4. string: "TEST" => valid: False 3. pattern: [0-9]{0,4}|TEXT|1234 3.0. string: "" => valid: True 3.1. string: "123" => valid: True 3.2. string: "12345" => valid: False 3.3. string: "TEXT" => valid: True 3.4. string: "TEST" => valid: False 4. pattern: TEXT|[0-9]{0,4}| 4.0. string: "" => valid: True 4.1. string: "123" => valid: True 4.2. string: "12345" => valid: False 4.3. string: "TEXT" => valid: True 4.4. string: "TEST" => valid: False --------- Code --------- #!/usr/bin/env python # -*- coding: utf-8 -*- import lxml.etree DEBUG = False for i, p in enumerate(['[0-9]{0,4}|TEXT', 'TEXT|[0-9]{0,4}', '[0-9]{0,4}|TEXT|', '[0-9]{0,4}|TEXT|1234', 'TEXT|[0-9]{0,4}|']): print '%i. pattern: %s' % (i, p) xsd = lxml.etree.fromstring('<?xml version="1.0" encoding="UTF-8"?>' '<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" attributeFormDefault="unqualified">' ' <xs:simpleType name="TTest">' ' <xs:restriction base="xs:string">' ' <xs:whiteSpace value="preserve"/>' ' <xs:pattern value="' + p + '"/>' ' </xs:restriction>' ' </xs:simpleType>' ' <xs:element name="Test" type="TTest"/>' '</xs:schema>') for j, s in enumerate(['', '123', '12345', 'TEXT', 'TEST']): xml = lxml.etree.fromstring('<?xml version="1.0" encoding="utf-8"?><Test>' + s + '</Test>') validator = lxml.etree.XMLSchema(xsd) valid = validator(xml) print '%i.%i. string: "%s" => valid: %s' % (i, j, s, valid) if not valid and DEBUG: for error in validator.error_log: print "message :", error.message print "domain :", error.domain print "type :", error.type print "level :", error.level print "line :", error.line print "column :", error.column print "filename :", error.filename print "domain_name:", error.domain_name print "type_name :", error.type_name print "level_name :", error.level_name print ''

3 4

Please see http://stackoverflow.com/questions/12591/using-an-xml-catalog-with-pythons-lxml
by Jean-Claude Magras 07 Nov '12

07 Nov '12

Hi, I don't want to be rude but take a look at this: http://stackoverflow.com/questions/12591/using-an-xml-catalog-with-pythons-… I have tried for an internet disconnected machine to lxml parse with xsd validation even using no_network=True I still get an insistence that "connection refused". I do not have lots of experience with python and most of the code is from lxml.de . Sincerely, JC Magras

2 1

Re: [lxml] no xpath evaluation in attributes of xslt extension elements?
by Eric Levy 05 Nov '12

05 Nov '12

Well, maybe. Unfortunately, I can't test it in my application because of a second problem. Ultimately, I would like to create an extension element that performs an XPath query on another document and inserts the result of the query into the output of the transformation. However, XPath query results come back as different types from what output_parent.append() expects, so simply passing the query result to the transformation result breaks the script. The documentation suggests that read-only types can be made into regular types by calling deepcopy(), but in my case this has no effect. Is there a solution to this new problem? ----- Original Message ----- Date: Thu, 01 Nov 2012 14:24:43 +0100 From: Stefan Behnel <stefan_ml(a)behnel.de> Subject: [lxml] no xpath evaluation in attributes of xslt extension elements? To: lxml mailing list <lxml(a)lxml.de> Eric Levy, 29.10.2012 08:31: > I'm back with another question about lxml. > > I am tinkering with the extension elements feature of the XSLT engine. > > I notice that if I try to give an XPath expression enclosed in {} as an attribute value in my own element type, the contents of the {} are not evaluated as an XPath expression, even though they would be if the element were not an extension. > > For example, in normal XSLT processing: > > <xslt:apply-templates match="foo"> > <bar attrib-of-bar="{@attrib-of-foo}" /> > </xslt:apply-templates> > > This would create a <bar/> element with the attrib-of-bar attribute set to the value of the attrib-of-foo attribute of foo. > > However, if <bar/> is lxml extension element, then the {} contents are not evaluated, but rather the entire attribute value is returned verbatim. Does this help? http://lxml.de/extensions.html#applying-xsl-templates Stefan

1 0

ElementTree as secondary input to transformation
by Eric Levy 01 Nov '12

01 Nov '12

Hi all, I wish for two separate ElementTree objects to serve as input for the same XSLT transformation. As I understand, XSLT in general supports secondary input documents by way of the XPath doc() function. Normally the function represents an XML tree from a named file. It appears that lxml supports custom resolvers in case the document is not simply a file on disk as named. Further, it appears that a custom resolver may resolve a URI as string, another filename, or a file object. What about the case of the secondary document being an ElementTree in memory? Is this case supported? Thanks. Sincerely, Eric Levy

2 1

no xpath evaluation in attributes of xslt extension elements?
by Eric Levy 01 Nov '12

01 Nov '12

Hi, I'm back with another question about lxml. I am tinkering with the extension elements feature of the XSLT engine. I notice that if I try to give an XPath expression enclosed in {} as an attribute value in my own element type, the contents of the {} are not evaluated as an XPath expression, even though they would be if the element were not an extension. For example, in normal XSLT processing: <xslt:apply-templates match="foo"> <bar attrib-of-bar="{@attrib-of-foo}" /> </xslt:apply-templates> This would create a <bar/> element with the attrib-of-bar attribute set to the value of the attrib-of-foo attribute of foo. However, if <bar/> is lxml extension element, then the {} contents are not evaluated, but rather the entire attribute value is returned verbatim. Is this behavior intentional? Is there anyway around it? Sincerely, Eric Levy

2 1

Fwd: POST method form in raises TypeError with submit_form in python3
by Tim Brooks 31 Oct '12

31 Oct '12

Hi all, I'm getting an error while trying to submit a form that uses the POST method. I've posted on StackExchange ( http://stackoverflow.com/questions/13151333/post-method-form-in-lxml-raises… ) but have yet to recieve any feedback. The minimal reproduction of the problem (this site responds to the post with 404, but the error looks to be in submission): > >>> import lxml.html > >>> page = lxml.html.parse("http://www.webcom.com/html/tutor/forms/start.shtml") > >>> form = page.getroot().forms[0] >>> form.fields['your_name'] = 'Tim' > >>> result = lxml.html.parse(lxml.html.submit_form(form)) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "/usr/lib/python3.3/site-packages/lxml/html/__init__.py", line 887, in submit_form > return open_http(form.method, url, values) > File "/usr/lib/python3.3/site-packages/lxml/html/__init__.py", line 907, in open_http_urllib > return urlopen(url, data) > File "/usr/lib/python3.3/urllib/request.py", line 160, in urlopen > return opener.open(url, data, timeout) > File "/usr/lib/python3.3/urllib/request.py", line 471, in open > req = meth(req) > File "/usr/lib/python3.3/urllib/request.py", line 1183, in do_request_ > raise TypeError(msg) > TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str. Line 3 (setting the 'your_name' field) can be changed to use a bytes object or omitted yet give the same result. I'm using python 3 on Arch Linux. Following your bug procedure I dumped the library versions: > >>> print("%-20s: %s" % ('Python', sys.version_info)) > Python : sys.version_info(major=3, minor=3, micro=0, releaselevel='final', serial=0) > >>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION)) > lxml.etree : (3, 0, 1, 0) > >>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION)) > libxml used : (2, 8, 0) > >>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION)) > libxml compiled : (2, 8, 0) > >>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION)) > libxslt used : (1, 1, 26) > >>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION)) > libxslt compiled : (1, 1, 26) Being Arch, I expect these are quite up to date though I've not looked into them further. I'd appreciate if someone can point out if I'm doing something wrong, but this looks like an internal issue with lxml. Cheers, Tim.

1 0