Mailman 3 December 2011 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 11 Apr '23

11 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 10 Apr '23

10 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Memory leak when parsing XML files in sequence?
by Maarten van Gompel (proycon) 03 May '13

03 May '13

Hi, I stumbled across what I think it a memory leak within the lxml module. I am parsing literally millions of mostly small XML files, in sequence. In the following, simplified, fashion: index = glob.glob('/path/to/dir/with/huge/number/of/xml/files/*xml') for f in index: d = lxml.etree.parse(f) The problem is that (almost) every iteration, memory usage is increased. But note that d gets overwritten everytime and the reference to the previous document should be lost (I don't reference it anywhere else). Even an explicit 'del d' and gc.collect() within the loop doesn't help to clear up the extra memory. I used objgraph to debug a bit and the Python reference counts remain unchanged as I would expect, leaving me to conclude that this is a memory leak problem in the lxml module. This becomes problematic quickly when dealing with millions of XML files. I attach a short log excerpt in which I extracted resident memory usage from ps after each iteration and measure the increase. Note that I only parse the documents, to be overwritten each time, I don't do anything else with them in this test case. Is this a known problem? Is there anything else I explicitly need to do to free the memory used? The problem does not reproduce if I reload the same document over and over again. Memory usage remains constant then. It only happens when new documents are loaded, and even then in some rare cases the problem dos not occur for some or several iterations, most notably at the start of the log. I also attach an example of an XML file. Python 2.7.2 (ubuntu 11.10, x86_64) lxml.etree : (2, 3, 0, 0) libxml used : (2, 7, 8) libxml compiled : (2, 7, 8) libxslt used : (1, 1, 26) libxslt compiled : (1, 1, 26) Regards, -- Maarten van Gompel (Proycon) E-mail: proycon(a)anaproy.nl Homepage: http://proycon.anaproy.nl Google+: https://plus.google.com/105334152965507305708 Facebook: http://facebook.com/proycon Twitter: http://twitter.com/proycon

6 16

[lxml-dev] confusing xpath performance characteristics
by jholg＠gmx.de 24 Aug '12

24 Aug '12

Hi, I ran into some performance characteristics of lxml/libxml2 xpath that I find rather confusing: I try to find the @type attribute of a certain element in an XML Schema (which contains lots of complexType definitions with lots of elements in them; unfortunately I can't post the schema): >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.095885038375854492, 0.096823930740356445, 0.096174955368041992] So I think I'm being smart and give a little more path information - reckoning that this should *improve* performance: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.1770780086517334, 0.1775970458984375, 0.17748594284057617] Hm. Performance degrades slightly. I'm adding even more of the path to where my desired elements live in the schema: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [103.79744100570679, 103.83671712875366, 103.61817717552185] What??? >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('/ae/data/pydev/hjoukl/NDM/SVN_CO/TRUNK/ndm/reference/xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType/*/xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.044407129287719727, 0.044126987457275391, 0.044229030609130859] >>> Ok, this version's better than my naive approach, which seems logical to me. But why would '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' perform drastically slower than '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' ? libxml2 problem? Running the same xpaths in Oxygen I don't notice performance differences (can't profile this). Holger -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

2 5

Minimal example of lxml writing an XML file and then reading and validating it against an XSD
by Duane Kaufman 03 Jan '12

03 Jan '12

Hi, My name is Duane Kaufman, and I am new to this list (please be gentle :) I am new to XML (not to Python), and I am wrestling with trying to get lxml to perform a task for me (I am on Windows XP, Python 2.7 lxml 2.3). I want to: 1) Use lxml to create an XML file 2) (Manually) create an XML Schema (XSD) file for the created XML file 3) Use lxml to read the XML file, validating it against the XSD file from 2) I have tried: MyXMLWriter.py: #------------------------- ScriptRootDir = r'H:\My Documents\Manufacturing\my_python\XML_Test\XMLSchema_test' def main(): global ScriptRootDir # Script to test the use of package lxml to pass XML messages containing data # This script will write out an XML file which will be a message with data from lxml import objectify from lxml import etree import os root = objectify.Element("root") root.StationName = objectify.DataElement("BaseDispensing", _xsi="string") root.MessageType = objectify.DataElement("BasePartUnloadedData", _xsi="string") objectify.deannotate(root, xsi=False) print(etree.tostring(root, pretty_print=True, xml_declaration=True)) et = etree.ElementTree(root) fp = open(os.path.join(ScriptRootDir, "Test.xml"), "w") et.write(fp, pretty_print=True, xml_declaration=True) fp.close() if __name__ == '__main__': main() #------------------------ Output from MyXMLWriter.py (Test.xml): #------------------------ <?xml version='1.0' encoding='ASCII'?> <root xmlns:py="http://codespeak.net/lxml/objectify/pytype":xsd="http://www.w3 .org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <StationName xsi:type="xsd:string">BaseDispensing</StationName> <MessageType xsi:type="xsd:string">BasePartUnloadedData</MessageType> </root> #----------------------- Test.xsd: #------------------ <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> </xsd:schema> #------------------ MyXMLReader.py: #------------------ ScriptRootDir = r'H:\My Documents\Manufacturing\my_python\XML_Test\XMLSchema_test' XMLSchemaFilename = "Test2.xsd" def main(): global ScriptRootDir # Script to test the use of package lxml to pass XML messages containing data # This script will read in an XML file which will be a message with data # into a Python data structure from lxml import etree import os # read and parse XML Schema file xsd_doc = etree.parse(XMLSchemaFilename) xsd = etree.XMLSchema(xsd_doc) MessageObject = etree.parse(os.path.join(ScriptRootDir, "Test.xml")) xsd.validate(MessageObject) print xsd.error_log if __name__ == '__main__': main() #------------------------ When running MyXMLReader.py, I get the output: file:///H:/My%20Documents/Manufacturing/my_python/XML_Test/XMLSchema_tes t/Test.xml:1:0:ERROR:SCHEMASV:SCHEMAV_CVC_ELT_1: Element 'root': No matching global declaration available for the validation root. How do I get this to work from end-to-end? Thanks in advance, Duane

2 1

Passing xpath results to ElementMaker
by Jeffrey Yasskin 03 Jan '12

03 Jan '12

I'm writing an XML-transforming script with lxml, and I've run into two unexpected behaviors: First, ElementMaker doesn't accept the string results of xpath expressions: >>> elem = etree.parse(io.StringIO('<root><node>text</node></root>')) >>> E.b(elem.xpath('string(node)')) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/lxml/builder.py", line 220, in __call__ raise TypeError("bad argument type: %r" % item) TypeError: bad argument type: 'text' >>> type(elem.xpath('string(node)')) <class 'lxml.etree._ElementUnicodeResult'> This happens because https://github.com/lxml/lxml/blob/master/src/lxml/builder.py#L215 looks up the exact type of the argument in a dict, rather than doing something that can respect _ElementUnicodeResult's inheritance from str. Is this the right behavior for some reason I haven't thought of, or is it just an oversight? The workaround is straightforward—just pass the xpath result through str()—but it seems more verbose than should be necessary. Second, ElementMaker doesn't accept lists at all: >>> etree.tostring(E.b(elem.xpath('node'))) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/lxml/builder.py", line 220, in __call__ raise TypeError("bad argument type: %r" % item) TypeError: bad argument type: [<Element node at 0x106eb8d70>] Accepting lists would also be useful for building HTML documents as E.section("Header text", function_returning_element_sequence(), "Footer text"). The workarounds for this are straightforward too: >>> etree.tostring(E.b(elem.xpath('node')[0])) b'<b><node>text</node></b>' >>> etree.tostring(E.b(*elem.xpath('node'))) b'<b/>' ... oops. Looks like ElementMaker re-parents nodes passed to it rather than copying them when they already have a parent. That's a third surprise, although it makes sense if ElementMaker was only intended for building entirely new documents, rather than copying bits from existing documents. This is with python-3.2.2, lxml-2.3.0, and libxml-2.7.8, Are these worth filing bugs/feature requests about? Thanks, Jeffrey

2 1

Fwd: lxml on mod_wsgi
by Damian Powązka 27 Dec '11

27 Dec '11

> Damian Powązka, 26.12.2011 18:36: >> W dniu 21 grudnia 2011 18:51 użytkownik Stefan Behnel napisał: >>> Damian Powązka, 21.12.2011 18:11: >>>> I don't share objects across threads. Problem occurs when trying to use >>>> etree.XMLSchema or etree.XMLParser with schema provided the error that I >>>> sometime see is: >>>> >>>> File "xmlschema.pxi", line 105, in lxml.etree.XMLSchema.__init__ >>>> (src/lxml/lxml.etree.c:132605) >>>> lxml.etree.XMLSchemaParseError: Operation in progress >>> >>> That's completely unrelated to the FAQ entry you referenced then. >>> >>> I pasted your error message into a search engine and it turned up a couple >>> of references that hint at HTTP access, or at least an I/O error (that's >>> where the error message comes from: errno == EINPROGRESS). Just a stab in >>> the dark: maybe your XML catalogues are not properly set up? Apart from >>> that, I have no idea what kind of I/O you do, so I can't comment on it. >> >> I have found a temporary work around for the problem: >> https://techknowhow.library.emory.edu/blogs/branker/2010/07/30/django-lxml-… > > For which of the problems you described is that? > > >> but in long term limitation to only one WSGI is bit problematic. Any >> chance that this issue could be fixed? > > Potentially, yes. It may or may not get fixed by a change I've been working > on for a while, which would basically introduce a per-document lock around > tree modifications, thus making it safe to read and modify trees > concurrently in different threads. However, I don't know when (or if) > that'll get finished. > > It would certainly help if you could come up with a short code snippet that > shows the problem (and that doesn't depend on mod_wsgi). > > Stefan Hi, Same problematic code run without mod_wsgi executes without issue. >From what I learned the problem occurs when making an instance of etree.XMLSchema on the xsd that contains xsd:import. I have tried using resolvers but it wasn't called when running on mod_wsgi, so the problem must be somewhere before Resolvers are used. I may try to simulate way how python instance is created by mod_wsgi. For now I could provide you a VirtualBox machine that has everything setup, so I could see the issue for yourself. I am not sure that is has to do something with threading or locking as the problem occurs always - I am not able to make a single successful http call. WSGIApplicationGroup %{GLOBAL} fixes a problem. My guess would be that XMLSchema on xsd:import does something odd with pointer to python interpreter. Best Regards, Damian Powazka

1 0