Mailman 3 October 2007 - lxml - The Python XML Toolkit

[lxml-dev] Problem with lxml library running on Windows
by Roberto Carrasco 09 Oct '07

09 Oct '07

Hi: We have an issue with libxml library running on Windows. We are trying to read an xml document from a string over and over but the program crashes in the while loop. We suspect the problem is that we cannot run the function etree.parse too much times when we are reading a xml document from a string. The code crashes when the program read a xml document repeatedly. The issue is on Windows becuase on an Linux environment there is no problem excecuting it. We are trying to execute the piece of code shown below in this environment: - Windows XP Service Pack 2 - Python 2.5 - lxml 1.3.4 and 2.0 alpha 3 The question is: what we are doing wrong? or is this a problem with the library running on Windows? # -*- coding: UTF-8 -*- from lxml import etree from StringIO import StringIO if __name__ == "__main__": document=""" <doc type="Ficha Visado" id="2"> <attribute name="Rut Afiliado" type="string">1-3</attribute> <attribute name="Fecha Escaneo" type="date">2006-03-13 08:44:52</attribute> <attribute name="Suc Promotor" type="string">SANTIAGO</attribute> <attribute name="Apellido Paterno" type="string">PUENTE</attribute> <attribute name="Fecha Visacion" type="string">13/03/2006</attribute> <attribute name="Nombres Afiliado" type="string">Robertin</attribute> <attribute name="Apellido Materno" type="string">MANZANO</attribute> <doc type="DPS" id="1"> <attribute name="Fecha Escaneo" type="date">2006-03-10 15:52:29</attribute> <link href="1-0.img" rel="media"/> <link href="1-1.img" rel="media"/> </doc> </doc>""" j=0 while 1: print j j+=1 #tree = etree.parse(StringIO(docRauco0)) tree = etree.fromstring(document) images_url = tree.xpath('//link[@rel="media"][@href]') image_url_name=images_url[0].attrib['href'] -- Regards, Roberto

4 5

[lxml-dev] is there a binary windows installer for lxml 2.0alpha4 release?
by Kelie 09 Oct '07

09 Oct '07

as subject. thanks.

2 1

[lxml-dev] Issues with objectify.ObjectifiedElement: assignment, attribute handling, and documentation
by Michael Pechal 08 Oct '07

08 Oct '07

Hello, I am new to XML and I have found lxml.objectify to be very useful. I am using XML to store register settings. I use the register mnemonic as the tag. I use custom attributes to store additional information, such as address, description, apply, etc. I am also using the _pytype and _xsi attributes. I am using the binary install of lxml 1.3.4 on WinXP running Python 2.5.1. My main problem is that assigning a new value to an objectify.DataElement destroys the existing attribute list. My current workaround is to retrieve the attributes with the items() call, assign the new value, and then reapply attributes with set() method for each pair in the items dict. I dug through the API documentation and I did not see a way around this issue. Am I missing something here? I thought about subclassing DataElement and then I scanned the SVN development change list. I saw some discussion about preserving _pytype or _xsi attributes, but does this include ALL attributes? If so, I will proceed with a build from the latest SVN copy. How stable are dev versions? Are there automated acceptance tests (unittest) that gate the check in? I may just use my workaround until 1.3.5 arrives. Another issue I noticed is that if I specify _xsi='int', the _pytype attribute will be 'long' instead of 'integer', so I am forced to use _pytype='integer' for all integer data elements. Also, if you run objectify.annonate(), the integer becomes a long type again. Annotate should look to the _xsi or even pyval type. Has this been fixed? This is not really an issue for me, since I always keep the list annotated. The objectify API documentation was helpful. As a new user, I had a few problems with save and retrieve from file. I would suggest updating the objectify API document to provide a full example of saving to and loading from a file. I have provided a test case from my unittest code below that may be useful for the documentation: # ------------------------------------------------------------------------ - def testFileSaveAndLoad(self): """ Save to XML file, then reload and compare data. """ # note the self.objRoot is created in the setUp() method tofile = etree.tostring(self.objRoot, pretty_print=True) xmlFH = open('test.xml', 'w') xmlFH.write(tofile) xmlFH.close() parser = etree.XMLParser(remove_blank_text=True) lookup = objectify.ObjectifyElementClassLookup() parser.setElementClassLookup(lookup) tree = etree.parse('test.xml', parser) root = tree.getroot() # crucial step, as parse() doesn't return the root fromfile = etree.tostring(root, pretty_print=True) self.assertEqual(tofile, fromfile) Also on the documentation front, there is a failure with help() on the objectify module. : Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import objectify >>> from lxml import etree >>> help(objectify) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python25\lib\site.py", line 346, in __call__ return pydoc.help(*args, **kwds) File "C:\Python25\lib\pydoc.py", line 1645, in __call__ self.help(request) File "C:\Python25\lib\pydoc.py", line 1689, in help else: doc(request, 'Help on %s:') File "C:\Python25\lib\pydoc.py", line 1481, in doc pager(title % desc + '\n\n' + text.document(object, name)) File "C:\Python25\lib\pydoc.py", line 324, in document if inspect.ismodule(object): return self.docmodule(*args) File "C:\Python25\lib\pydoc.py", line 1070, in docmodule inspect.getclasstree(classlist, 1), name)] File "C:\Python25\lib\inspect.py", line 656, in getclasstree for parent in c.__bases__: TypeError: 'functools.partial' object is not iterable >>> objectify <module 'lxml.objectify' from 'C:\Python25\lib\site-packages\lxml\objectify.pyd'> Note that help(etree) works fine. Thanks, Michael This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

3 6

[lxml-dev] Dealing with segfaults in lxml?
by Mike Meyer 08 Oct '07

08 Oct '07

I'm getting crashes - by which I mean the python process is segfaulting and, with some tweaking of GNU/Linux, leaving me a core file - while using lxml to parse data. Versions: OS: RHEL 5 Python: 2.5.1 (custom built). lxml: 1.3.3 libxml: 2.6.26 (both compiled and built) libxslt: 1.1.17 [Yes, I know those are a bit out of date, but we had to give our client host requirements months ago, and those were current at the time, and changing them is a non-trivial process, and I've already started on it, but I'd rather not do that if I can avoid it....] Rebuilding python with OPTS=-g (I set that for the lxml build as well), I can get a "where" output that points at lxml: #0 0x00002aaaaf906c3a in rename () from /usr/local/lib/python2.5/site-packages/lxml/etree.so #1 0x00002aaaaf906be7 in rename () from /usr/local/lib/python2.5/site-packages/lxml/etree.so #2 0x00002aaaaf8ebdfe in rename () from /usr/local/lib/python2.5/site-packages/lxml/etree.so #3 0x00002aaaaf966a5c in findOrBuildNodeNs () from /usr/local/lib/python2.5/site-packages/lxml/etree.so The first problem is that this isn't repeatable. I've got test data that will make it happen, but I have to feed that data through the system a few thousand times in. This is part of a database ETL system, parsing data from the XML to load into the database. If I feed it the exact same data over and over again, it'll work 9999 times out of ten thousand - but then fail that ten thousands time with a segfault. While this might not seem like a big deal, we're planning on processing hundreds of thousands of documents a day, so we're talking about having an instance of the process die tens of times a day. So I sorta need to fix it. The document is straightforward: it starts with a meta element with a set of attributes, and then has a lot of data elements, all the same type, all with the same attributes (give or take an optional one), and I just use document.xpath to find the elements, and then read off their attribute values to save to a database load file. Hints on how to proceed - setting things up so I can use gdb on the lxml sources, for instance - would be greatly appreciated. If this looks like a bug that's been fixed if I update one or more libraries, that would be great information (i.e. - I can use it to get all the libraries updated). Anything else that you think I oughta know would be nice as well. The sample document is almost half a megabyte (and might be proprietary). If you'd like to look at it, drop me a line. thanks, <mike -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information.

4 5

[lxml-dev] did windows binary versions ever get removed from the cheeseshop?
by Martijn Faassen 08 Oct '07

08 Oct '07

Hi there, To start off, I'm not 100% sure on this, so I'm just checking. I thought at some stage I had a working windows installation of my software that was using lxml 1.3 binaries (not 1.3.1 or something, just 1.3). When I tried again today it didn't work anymore, and instead had to start using lxml 1.3.4 (for instance). Is it possible that someone for some reason removed the versions for 1.3 from the cheeseshop? If so, my general recommendation would be never to do this, even if the packages are broken somehow. A release is a release, and people might be depending on it. For this reason, never remove release files, and also never overwrite release files. Anyway, I'm not at all sure this actually happened with lxml, but I'm just writing this to make sure it won't happen. :) Regards, Martijn

2 1

[lxml-dev] lxml 2.0alpha4 released
by Stefan Behnel 07 Oct '07

07 Oct '07

Hi all, I just released a 4th alpha version of lxml 2.0 to PyPI. It hopefully sets an end to the tag name validation problems by distinguishing between HTML tags and XML tags based on the associated parser, i.e. either the one that parsed it or the one that created the element through its "makeelement" method. Note that the Element factory of lxml.etree uses the XMLParser by default, while the factory in lxml.html uses the HTMLParser, and thus allows HTML tag names. Everyone who bumped into and/or reported problems with this, please verify that this provides a viable solution to you. Have fun, Stefan 2.0alpha4 (2007-10-07) Features added Bugs fixed * AttributeError in feed parser on parse errors Other changes * Tag name validation in lxml.etree (and lxml.html) now distinguishes between HTML tags and XML tags based on the parser that was used to parse or create them. HTML tags no longer reject any non-ASCII characters in tag names but only spaces and the special characters <>&/'"

1 0

[lxml-dev] Namespace serialization patch
by Lawrence Oluyede 06 Oct '07

06 Oct '07

I had the same problem Anders Bruun Olsen had in this thread: http://comments.gmane.org/gmane.comp.python.lxml.devel/2924 What I'd like to know if I have to wait for 2.0 completion (using the alpha is not an option AFAIK) to use it or you plan to release an interim 1.3.x version with that patch applied. Thanks -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 3

[lxml-dev] prefix mappings
by FnH 06 Oct '07

06 Oct '07

Hi, I would like to generate the following serialization: <a xmls="foo"> <b xmls="bar"/> </a> Currently, the closest I can come is: <a xmls="foo" xmlns:x="bar"> <x:b/> </a> using the following code: a = Element("{foo}a", nsmap={None:"foo", "x":"bar"}) a.append(Element("{bar}b")) Note that even though these serializations are equivalent, sometimes remapping namespaces is neccessary to work around faulty implementations in other products. The feed validator for example warns against using namespace prefixes on elements for good reason. If you don't, many readers aren't able to parse your feed (http://www.intertwingly.net/wiki/pie/XmlNamespaceConformanceTests). If you're using xhtml as content this means remapping the default namespace. In order to solve this I think it would be a good idea to allow (or take into account) prefix mappings on non root nodes as well. The output I'd like could then be achieved by the following code snippet: a = Element("{foo}a", nsmap={None:"foo"}) a.append(Element("{bar}b", nsmap={None:"bar"})) What do you think? Regards, Nick

2 1

[lxml-dev] Tag name validation and HTML
by James Graham 06 Oct '07

06 Oct '07

The development branch of lxml 2 appears to restrict the characters that may appear in a tag name. Whilst this may be appropriate for XML, it does not match the behavior of all common HTML UAs and, as such, does not match the current draft of the HTML 5 spec [1]. This is an issue for html5lib [2] as we are keen to keep support for building lxml trees from HTML input, something which is currently possible with lxml 1.3. In an only tangentially related question, is there a recommended way of creating a custom tag type, preferably using the same code for ElementTree and lxml.etree? In particular html5lib needs to create a notional document root element whilst parsing. So far, we have been using an ordinary Element with a .tag that cannot be produced by parsing any input e.g. root.tag="<DOCUMENT_ROOT>" but this doesn't feel very elegant. [1] http://www.whatwg.org/specs/web-apps/current-work/#tag-name0 [2] http://code.google.com/p/html5lib/ -- "Eternity's a terrible thought. I mean, where's it all going to end?" -- Tom Stoppard, Rosencrantz and Guildenstern are Dead

2 5

[lxml-dev] lxml 2.0alpha1 released
by Stefan Behnel 06 Oct '07

06 Oct '07

Hi all, I'm proudly announcing the first alpha release of lxml 2.0. It features a major cleanup both behind the scenes and at the surface, that improves the XML tool integration and makes the API clearer and more consistent in many places. The major new addition, however, is the lxml.html package, a new toolkit for HTML handling. The web site for the pre-2.0 series is online at http://codespeak.net/lxml/dev/ The "what's new" page has a description of the major changes: http://codespeak.net/lxml/dev/lxml2.html and the ChangeLog has a more detailed list, see below. This being an alpha release means that not everything is stable, both in terms of crashes and the API. There will be a small number of alpha releases to make the advancements publicly available, before the beta releases focus on improving the stability. I warmly invite everyone to contribute to the final release by discussing the API changes and the new features on the mailing list. There is always space for improvements! There is currently a known problem with Microsoft's compilers, so Windows builds may not become available for 2.0alpha1. The next alpha will hopefully come with prebuilt binaries for that platform. Building with the more standards compliant MinGW compilers should work. Note that working on the code now requires Cython (version 0.9.6.5), an enhanced fork of Pyrex. lxml therefore no longer ships with a copy of Pyrex or Cython, but as usual, building from the distribution sources does not require Cython. It can be installed with "easy_install Cython" or from here: http://www.cython.org/ I hope that lxml 2.0 will become a straight continuation of the success story that lxml 1.x was already. Have fun, Stefan 2.0alpha1 (2007-09-02) Features added * Reimplemented objectify.E for better performance and improved integration with objectify. Provides extended type support based on registered PyTypes. * XSLT objects now support deep copying * New makeSubElement() C-API function that allows creating a new subelement straight with text, tail and attributes. * XPath extension functions can now access the current context node (context.context_node) and use a context dictionary (context.eval_context) from the context provided in their first parameter * HTML tag soup parser based on BeautifulSoup in lxml.html.ElementSoup * New module lxml.doctestcompare by Ian Bicking for writing simplified doctests based on XML/HTML output. Use by importing lxml.usedoctest or lxml.html.usedoctest from within a doctest. * New module lxml.cssselect by Ian Bicking for selecting Elements with CSS selectors. * New package lxml.html written by Ian Bicking for advanced HTML treatment. * Namespace class setup is now local to the ElementNamespaceClassLookup instance and no longer global. * Schematron validation (incomplete in libxml2) * Additional stringify argument to objectify.PyType() takes a conversion function to strings to support setting text values from arbitrary types. * Entity support through an Entity factory and element classes. XML parsers now have a resolve_entities keyword argument that can be set to False to keep entities in the document. * column field on error log entries to accompany the line field * Error specific messages in XPath parsing and evaluation NOTE: for evaluation errors, you will now get an XPathEvalError instead of an XPathSyntaxError. To catch both, you can except on XPathError. * The regular expression functions in XPath now support passing a node-set instead of a string * Extended type annotation in objectify: new xsiannotate() function * EXSLT RegExp support in standard XPath (not only XSLT) Bugs fixed * lxml.etree did not check tag/attribute names * The XML parser did not report undefined entities as error * The text in exceptions raised by XML parsers, validators and XPath evaluators now reports the first error that occurred instead of the last * Passing '' as XPath namespace prefix did not raise an error * Thread safety in XPath evaluators Other changes * objectify.PyType for None is now called "NoneType" * el.getiterator() renamed to el.iter(), following ElementTree 1.3 - original name is still available as alias * In the public C-API, findOrBuildNodeNs() was replaced by the more generic findOrBuildNodeNsPrefix * Major refactoring in XPath/XSLT extension function code * Network access in parsers disabled by default

3 20