Mailman 3 March 2011 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 11 Apr '23

11 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 10 Apr '23

10 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

[lxml-dev] confusing xpath performance characteristics
by jholg＠gmx.de 24 Aug '12

24 Aug '12

Hi, I ran into some performance characteristics of lxml/libxml2 xpath that I find rather confusing: I try to find the @type attribute of a certain element in an XML Schema (which contains lots of complexType definitions with lots of elements in them; unfortunately I can't post the schema): >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.095885038375854492, 0.096823930740356445, 0.096174955368041992] So I think I'm being smart and give a little more path information - reckoning that this should *improve* performance: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.1770780086517334, 0.1775970458984375, 0.17748594284057617] Hm. Performance degrades slightly. I'm adding even more of the path to where my desired elements live in the schema: >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType//xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [103.79744100570679, 103.83671712875366, 103.61817717552185] What??? >>> timeit.Timer(stmt="""xpath(schema)""", setup="""from lxml import etree, objectify; schema=etree.parse('/ae/data/pydev/hjoukl/NDM/SVN_CO/TRUNK/ndm/reference/xsd/NDM.xsd').getroot(); xpath = etree.XPath('/xs:schema/xs:complexType/*/xs:element[@name="equity"]/@type', namespaces={'xs': 'http://www.w3.org/2001/XMLSchema'})""").repeat(number=10) [0.044407129287719727, 0.044126987457275391, 0.044229030609130859] >>> Ok, this version's better than my naive approach, which seems logical to me. But why would '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' perform drastically slower than '/xs:schema/xs:complexType//xs:element[@name="equity"]/@type' ? libxml2 problem? Running the same xpaths in Oxygen I don't notice performance differences (can't profile this). Holger -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

2 5

[lxml-dev] Codespeak shutting down: migration plans?
by Luciano Ramalho 23 Jul '11

23 Jul '11

Dear colleagues, As a long-time user of lxml I was shocked to read a message today from Holger Krekel about the end of Codespeak (see full text at the end of this). First we need to thank Holger for all these years of free service. Then we need to quickly plan and execute a migration to some other repository. I am willing to help, but I am no core developer of lxml, and in fact this is my first message ever to this mailing list, so I think we need someone better known to this community to lead the migration effort. lxml as a key piece of the Python ecosystem and for the benefit of its present and future users and the wider Python community we need to make sure it continues to be available, and to make sure as many references (pypi!) as possible point to the new canonical repository. Finally, a big thank you to Martijn, Philikon, Stefan and all the others who have built this great piece of software. Cheers, -- Luciano Ramalho programador repentista || stand-up programmer Twitter: @luciano ######################################### hi codespeak.net users, (sorry if you get mail twice, i wanted to make sure ...) after 8 years of operation codespeak.net services are bound to terminate, starting END OF FEBRUARY 2011 Background: one of the original codespeak purposes was to offer subversion (then in version 0.17) for the PyPy and other projects but today this is not too interesting given the pletora of VCS hosting solutions. Also, there aren't too many admins besides me, the hosting is costing money, PyPy's repository has moved to Bitbucket and i am re-shuffling my priorities preparing for my soon to emerge father-hood. After February 2011 i probably won't be able to help much with any transition issues or questions. The host will keep on running for a while but i give no guaruantees. Some remarks regarding termination wrt to the FEB 2011 deadline: * the subversion repo will turn read-only (and will eventually be switched off). * Shell accounts will be restricted to those people who need it *and* mail me about it. Some time later they will be gone as well. * Mailing lists will be terminated as well unless i get a mail asking me to postpone termination for a specific time. You can go to your respective mailman admin page and extract a list of members. If you mail me i can also provide a list of members. * Any remaining web docs/pages will probably continue to exist for a while but i also prefer them to be moved away by end Feb 2011. Note that the codespeak svn repository contains a lot of projects. For migration you have two options: do a flat import just of your project checkout directory into a new version system. This is super-simple, obviously. If you want to preserve history for your project please mail me and i either provide you a full dump or a filtered dump only containing your project. So long and I hope you all had a good time and enjoyed the services and also have a good transition now. see you in other places, holger krekel _______________________________________________ codespeak-ann(a)codespeak.net http://codespeak.net/mailman/listinfo/codespeak-ann

4 7

[lxml-dev] DLL issue with win32 static builds of lxml
by Scott Smith 04 Apr '11

04 Apr '11

Hi I'm looking at upgrading our python installations from 2.6(.2) to 2.7 and upgrading the packages we use to the latest versions. We use python and the mod_wsgi module in Apache 2.2 for our application server and easy_install to install egg packages. I'm currently testing on a Windows 7 machine. The current egg version of lxml that we're using (before upgrading) is: lxml-2.2.4-py2.6-win32.egg In trying to upgrade things, I've found that all the later builds of lxml cause a DLL problem through apache. For example, all of these builds (from pipy) cause the problem: lxml-2.2.8-py2.6-win32.egg lxml-2.2.8-py2.7-win32.egg lxml-2.2beta1-py2.6-win32.egg lxml-2.3beta1-py2.6-win32.egg lxml-2.3beta1-py2.7-win32.egg (Note that there are no static win32 builds available for 2.2.5-2.2.7, or I can't find them) The error thrown in the Apache logs is: mod_wsgi (pid=4812): Target WSGI script 'C:/wsgi/handler.py' cannot be loaded as Python module. mod_wsgi (pid=4812): Exception occurred processing WSGI script 'C:/wsgi/handler.py'. Traceback (most recent call last): File "C:/wsgi/handler.py", line 2, in <module> from lxml import etree ImportError: DLL load failed: The specified module could not be found. The same file works in Python 2.6 if lxml 2.2.4 is installed, but fails for any other higher version of lxml (either in python 2.6 or 2.7). I've been able to trace the problem down to the builds not being able to find the msvcr90.dll file properly. I've tried modifying path variables and other changes, but can't figure out what the problem is. A couple of other things to note: - Using just "import lxml" doesn't cause the error, just "from lxml import etree" - The problem only occurs through Apache. If I write a script that references etree and call it through the command line, the problem doesn't occur. Does anyone know what I need to do? What changed after the 2.2.4 lxml builds? Thanks for any help you can give. Scott Smith

5 5

Re: [lxml] Error with validating xml file against schema
by Stefan Behnel 30 Mar '11

30 Mar '11

[I assume your response should have gone to the list] Wu, Hua, 28.03.2011 16:07: > Thank you for your response. > The bug 619640 that reported on May 25,2010 looks like to describe a > similar issue, doesn't it? > https://bugzilla.gnome.org/show_bug.cgi?id=619640 Similar, yes. However, given the complexity of XML Schema, it's hard to say if it's the exact same issue or not. Stefan

1 0

relaxng validation error
by Tim Arnold 29 Mar '11

29 Mar '11

hi, python 2.7, lxml2.3, freebsd amd64 I'm using lxml to validate against the DocBook v5 relaxng schema. I get an error that says "Expecting element set, got chapter, line 2", but the actual schema seems to allow chapters to be starting elements; I've appended the relevant snippets below. First, my test code: --------------------------- from lxml import etree tree = etree.parse('dbtest.xml') relaxng = etree.RelaxNG(file='/path_to/DocBook/V5.0/rng/docbookxi.rng') relaxng.assertValid(tree) --------------------------- results in: File "lxml.etree.pyx", line 3006, in lxml.etree._Validator.assertValid (src/lxml/lxml.etree.c:125415) lxml.etree.DocumentInvalid: Expecting element set, got chapter, line 2. the dbtest.xml starts out with this line: --------------------------- <chapter xmlns="http://docbook.org/ns/docbook" xmlns:xl="http://www.w3.org/1999/xlink" version="5.0"> --------------------------- As far as I can tell, the rng says chapters are okay. The first bit of the rng: <start> <choice> <choice> <ref name="db.set"/> <ref name="db.book"/> <ref name="db.divisions"/> <ref name="db.components"/> etc. and later on db.components is defined: <define name="db.components"> <choice> <choice> <ref name="db.dedication"/> <ref name="db.acknowledgements"/> <ref name="db.preface"/> <ref name="db.chapter"/> etc. thanks for reading. Is this a bug? --Tim Arnold

2 2

Re: [lxml] Error with validating xml file against schema
by Stefan Behnel 28 Mar '11

28 Mar '11

Wu, Hua, 25.03.2011 13:22: > I am trying using lxml to validate xml files against opengis gml 3.1.1 > schema but I got an error.I googled for solutions, I only found > similar errors out there but no solutions.Could anyone help me with > this error? I use lxml 2.3 and python 2.7. > > > code snippet: > > from lxml import etree > > schema = etree.parse("http://www.isotc211.org/2005/gml/gml.xsd") > xmlschema = etree.XMLSchema(schema) > > I got: > Error : local list type: A type, derived by list or union, must have > the simple ur-type definition as base type, not > '{http://www.opengis.net/gml}doubleList'., line 1 > > > Others run xmllint got similar error > http://gis.stackexchange.com/questions/6721/why-doesnt-the-gml-3-1-1-s > chema-validate-with-xmllint > > Someone thought this could be a bug, does this make sense? > http://www.mail-archive.com/xml@gnome.org/msg07477.html Yes, likely a bug in libxml2. XML Schema support is known to have its rough edges. Could you check if there's a ticket for this on the libxml2 bug tracker? https://bugzilla.gnome.org/buglist.cgi?product=libxml2 Stefan

1 0

Error with validating xml file against schema
by Wu, Hua 25 Mar '11

25 Mar '11

> Hi, > > I am trying using lxml to validate xml files against opengis gml 3.1.1 > schema but I got an error.I googled for solutions, I only found > similar errors out there but no solutions.Could anyone help me with > this error? I use lxml 2.3 and python 2.7. > > > code snippet: > > from lxml import etree > > schema = etree.parse("http://www.isotc211.org/2005/gml/gml.xsd") > xmlschema = etree.XMLSchema(schema) > > I got: > Error : local list type: A type, derived by list or union, must have > the simple ur-type definition as base type, not > '{http://www.opengis.net/gml}doubleList'., line 1 > > > Others run xmllint got similar error > http://gis.stackexchange.com/questions/6721/why-doesnt-the-gml-3-1-1-s > chema-validate-with-xmllint > > Someone thought this could be a bug, does this make sense? > http://www.mail-archive.com/xml@gnome.org/msg07477.html > Regards Hua

1 0

Error loging of lxml xonsume all disk space
by Андрей Никишаев 24 Mar '11

24 Mar '11

Hello, I have problem with lxml. Fore some reason sometimes it consumes all server memory when parsing data. I using it in web-spider(daemonized) so it's not so cool. So i set ulimit -m 600000 and ulimit -v 500000 for spider process. After this lxml start write data to error log, till there is free space on disk. So i get this error: "Exception MemoryError: MemoryError() in 'lxml.etree._BaseErrorLog._receive' ignored" One more trouble that i can't handle this exception in my code. To make this stop i have monkeypatched lxml.etree._BaseErrorLog but it's not helped. Can someone tell me how i can handle this exception, and how i can make lxml to not write at error log bypassing the python logging module? Thanks for help. Best regards, Andrey Nikishaev Blog http://creotiv.in.ua/ Skype creotiv.in.ua

2 1

Iterparse error
by Roger Hoover 23 Mar '11

23 Mar '11

Hi, I compiled lxml version 2.3 on Centos 5 and get this error trying to parse anything with iterparse. Any ideas what the problem could be? File "lxml_test.py", line 6, in <module> context = etree.iterparse("foo.xml", events=('end,'), tag='event') File "iterparse.pxi", line 421, in lxml.etree.iterparse.__init__ (src/lxml/lxml.etree.c:94766) File "parser.pxi", line 766, in lxml.etree._BaseParser._getPushParserContext (src/lxml/lxml.etree.c:73704) File "iterparse.pxi", line 450, in lxml.etree.iterparse._createContext (src/lxml/lxml.etree.c:94972) File "iterparse.pxi", line 123, in lxml.etree._IterparseContext._setEventFilter (src/lxml/lxml.etree.c:92127) File "iterparse.pxi", line 30, in lxml.etree._buildIterparseEventFilter (src/lxml/lxml.etree.c:91456) ValueError: invalid event name 'e' Thanks, Roger

2 3

whitespace and the HTMLParser
by Tim Arnold 22 Mar '11

22 Mar '11

Hi, I don't understand how whitespace is handled and I want to make sure the following is the desired behavior. The first tree= statement uses the HTMLParser which drops the space between the <img/> and the <a/> tags. If I do not use the HTMLParser, the whitespace is kept. Am I doing something wrong? Obviously I want to keep the whitespace and the files I'm working with are XHTML. Should I be using the normal XML parser in that case? thanks, --Tim Arnold from lxml import etree s = ''' <p>Suppose you have a <img src="images/g0088.png" alt="" /> <a href="g278.htm">matrix</a> as seen in Figure 3. </p> ''' parser = etree.HTMLParser() tree = etree.fromstring(s, parser=parser)#space is dropped #tree = etree.fromstring(s)# space is kept print etree.tostring(tree)

2 3

specifying catalog location
by Greg Baker 21 Mar '11

21 Mar '11

I'm trying to do a validating parse of HTML and XHTML content. I have gotten this far easily enough: parser = etree.XMLParser(dtd_validation=True, no_network=True) etree.fromstring(content, parser=parser) But this may be running on a machine where I don't have root access, so populating /etc/xml and /etc/sgml isn't an option. Is there some way to specify the location of the catalog so I can store the DTDs in a local directory? (Or is there some other way entirely of doing this that I haven't thought of?) -- Greg Baker, Senior Lecturer School of Computing Science Simon Fraser University Burnaby, BC, V5A 1S6 E-mail: ggbaker(a)cs.sfu.ca

2 2

Re: [lxml] no xml_declaration for unicode?
by info＠damiro.net 18 Mar '11

18 Mar '11

Hi Holger, please don't get me wrong. I don't want to mix 'utf-8' or 'unicode' or something else. Now I've understood what happend and everything is working fine. I think, the missing distinction between byte-buffers and string-types in py <2.6 leads to misapprehension in my case (see also pep-0358). The method called 'tostring', so I expect to get a object which I can handle as string and not a byte-buffer. At this point of view, it makes no sense to get a py 2.x 'str' type which really acts as bytebuffer of unicode characters. Imho it's more clear, if there two methods. One for string- and one for byte-buffer style returns. Take this as a hint. Apart from that, lxml is really great and makes fun to work with. Daniel

1 0

no xml_declaration for unicode?
by info＠damiro.net 17 Mar '11

17 Mar '11

Hi, there is a strange behavior, if im trying to use the 'xml_declaration' parameter on unicode encoding: >>> from lxml import etree >>> root = etree.Element("root") >>> res = etree.tostring(root, encoding="unicode", xml_declaration=True) raises: Traceback (most recent call last): File "<pyshell#1>", line 4, in <module> res = etree.tostring(root, encoding="unicode", xml_declaration=True) File "lxml.etree.pyx", line 2838, in lxml.etree.tostring (src/lxml/lxml.etree.c:53452) ValueError: Serialisation to unicode must not request an XML declaration I don't understand why it is not allowed to control the xml declaration if i'm using unicode. My environment is lxml 2.3.0 on CPython 2.7.1 (WinXP 32bit) Thanks in advance, Daniel

3 5

Google Summer of Code 2011
by Arc Riley 09 Mar '11

09 Mar '11

Hey guys Have you considered participating in Google's Summer of Code? We (PSF) are taking applications for Python-based smaller Python-based projects to participate under our umbrella as we've done the last five years. The gist is you'd potentially get a student or two being paid by Google to work on lxml through the Summer. They propose concrete projects to work on which they're expected to complete (by judgement of their mentors) in order to get paid the full amount. The application is here: https://spreadsheets.google.com/viewform?formkey=dHh3WFNGYzkyWWE0ZjM1eFFoUU…

2 1

Re: [lxml] lxml pxd files (+ .h files) in binary distributions
by Stefan Behnel 09 Mar '11

09 Mar '11

Giovanni Bajo, 08.03.2011 14:12: > Hello Stefan, Hi! > we are trying to use lxml through its cython API. This is made more > difficult than necessary because binary packages for Windows do not contain > lxml pxd files necessary to interface with it That's a bug. The public API currently only includes etreepublic.pxd and tree.pxd (ok, and python.pxd, but that's just a left-over), so that's all that should be needed, plus "etree_defs.h", which defines certain C level things, some of which are exported in etreepublic.pxd as well. You will need those three to use lxml.etree from Cython code. > nor libxml .h files they depend upon. That shouldn't matter all that much. Is this supposed to run only on Windows, locally for you, or do you want to support different (user) platforms? I.e. are you going to ship your binaries to users and expect this to work on their side? > Finding the correct libxml .h files requires knowledge of which libxml > version was used to compile the binary package, to avoid random crashes in > case the ABI is different between the version the binary package was > compiled against and the one that the developer downloaded from the website. The ABI shouldn't normally differ, libxml2 is pretty good in that. There may be a dependency on the C compiler, but that should normally be the VC version that fits the CPython version. > To simplify this, my proposal would be to modify lxml's setup.py to include > both the pxd files and the required libxml .h files in the bdist target; > the files would then be automatically bundled within a subdirectory called > "include" within the lxml package (eg: site-packages/lxml/include), so that > the correct versions will be available to developers willing to play with > them. > > Does this sound a good plan? You should be able to build against other libxml2 versions, but it depends on what you want to do with it. Working with the in-memory tree shouldn't be a problem at all as Daniel takes great care to keep the ABI stable. Using things like libxml2's parser directly may cause problems as you will be using a separate library instance with its own separate setup if lxml is built statically. Could you provide some details of what you are going to do on the other side of the interface? Stefan

2 6

lxml removes line breaks from XML attributes
by Giovanni Torres 07 Mar '11

07 Mar '11

Hello All! Some of you might not be happy with my question. But, I'm dealing with an XML file that has line breaks in XML attributes. I use lxml to parse the file, run some XPath queries, make changes to it and write it back. Unfortunately, lxml removes the line breaks from the attributes. Here is what I mean more clearly: $ cat example.xml <example attr="This is an attribute with several break lines"/> $ cat test.py import sys import lxml.etree xml = lxml.etree.parse(sys.stdin) xml.write(sys.stdout) print() $ python test.py < example.xml <example attr="This is an attribute with several break lines"/>() $ python -c 'import lxml.etree ; print(lxml.etree.__version__)' 2.3.0 $ python -V Python 2.6.5 $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 10.04.2 LTS Release: 10.04 Codename: lucid $ Is there any way I can get lxml to write those line breaks back? I'm actually not sure if they are even legal. But, they seem to be according to this: http://stackoverflow.com/questions/449627/are-line-breaks-in-xml-attribute-… Thanks in advance for your help! -- Giovanni

3 3

availability of lxml 2.3 windows binaries
by Piotr Dobrogost 01 Mar '11

01 Mar '11

Hi! When could we expect to be able to download lxml 2.3 windows binaries from PyPi (http://pypi.python.org/pypi/lxml/2.3#downloads)?

3 3