Mailman 3 September 2006 - lxml - The Python XML Toolkit

[lxml-dev] lxml 1.1.1 released
by Stefan Behnel 23 Sep '06

23 Sep '06

Hi all, lxml 1.1.1 is on cheeseshop. It's mainly a bugfix release for the 1.1 series. Changelog follows: Features added * XSLT profiling support (profile_run keyword) * countchildren() method on objectify.ObjectifiedElement * Support custom elements for tree nodes in lxml.objectify Bugs fixed * Error messages from XSLT did not reach XSLT.error_log * Factories objectify.Element() and objectify.DataElement() were missing attrib and nsmap keyword arguments * Changing the default parser in lxml.objectify did not update the factories Element() and DataElement() * Let lxml.objectify.Element() always generate tree elements (not data elements) * Build under Windows failed ('0' bug in patched Pyrex version) I uploaded X86-64 eggs for both Python 2.4 and 2.5, and I'd be very happy to see more contributions by others. Now is the right time to bring 1.1 to the masses. :) Have fun, Stefan

3 4

[lxml-dev] lxml 1.0.3 and lxml 1.1beta builds for various platforms?
by Martijn Faassen 20 Sep '06

20 Sep '06

Hey, Compare: http://cheeseshop.python.org/pypi/lxml/1.0.2 with http://cheeseshop.python.org/pypi/lxml/1.0.3 and we see that 1.0.2 has support for lots of different platforms, including the nice static windows build, but 1.0.3 has not. In part this is my fault, as it appears I need to do various linux eggs, but a couple of more egg donations from others would be appreciated! The same story applies to 1.1 beta. Regards, Martijn

3 17

[lxml-dev] Handling of <xsl:message terminate="yes"> ...
by Matthew Cruickshank 19 Sep '06

19 Sep '06

Hi, I've been writing this XML chain/pipeline - sort of a multi-pipeline version of Apache Cocoon and lxml has been great -- very fast! I'm having some problems with <xsl:message> and I don't know how to read errors caused by that. They terminate the processing but I can't find the error message text (from within the <xsl:message> tag) anywhere... it doesn't seem to be in error_log and no exception is thrown. Any ideas -- am I looking in the wrong place? I'm using <xsl:message> to assert things about document structure and runtime parameters (things that couldn't be asserted in RelaxNG, I think). .Matthew Cruickshank http://docvert.org << MSWord to HTML or any XML

2 2

[lxml-dev] egg package upload commands
by Martijn Faassen 18 Sep '06

18 Sep '06

Hi there, I wonder whether I'm doing something wrong, as previously this worked. To verify I'm not doing something stupid, could someone please say this is the proper sequence for uploading the lxml egg? download the tgz from codespeak.net. This contains the .c file that has been created by our special version of Pyrex. Then unpack it, and: $ python2.4 setup.py build [go to lib dir, strip the .so files] $ python2.4 setup bdist_egg $ python2.4 setup upload The last step fails with the following error: error: No dist file created in earlier command Given that I get this with another package too, this means one of two things: I'm doing something wrong, or my version of setuptools has a bug... Regards, Martijn

2 2

[lxml-dev] XSLT profiling
by Stefan Behnel 18 Sep '06

18 Sep '06

Hi, while I was looking through the libxslt API to check for stylesheet PI support, I stumbled over the profiling support that I had completely forgotten about. The result is that lxml.etree can now profile stylesheet runs. If you pass 'profile_run=True' to the transform call, the new 'xslt_profile' property of the result tree will contain an ElementTree with profiling data for each template, similar to the following (a 1-template XSLT result): <profile> <template rank="1" match="/" name="" mode="" calls="1" time="1" average="1"/> </profile> Hope it's useful, Stefan

1 0

Re: [lxml-dev] [objectify] inconsistent element types for different element access orders
by Holger Joukl 15 Sep '06

15 Sep '06

Hi Stefan, "Stefan Behnel" <behnel(a)dvs1.informatik.tu-darmstadt.de> schrieb am 15.09.2006 11:11:24: > > I really think anybody considering to use amara.bindery or gnosis.objectify > > should have a good look at lxml.objectify. > > Hear, hear! :) > > Can we quote you on our web page? > > Like: "Google uses Python, but for critical stuff, my bank depends on lxml, > because ..." Yes you can, and I'd also be willing to write a short success story or s.th. like that. But let's just delay any quoting or success stories for now, until we have actually realized the success ;-) > > > I chose "tree_class" and "empty_data_class" now. I think that's > > > sufficiently telling. > > > > I'm still not quite sure what you mean by that. In my words: That will > > allow for > > customizing the behaviour when encountering > > - an empty leaf node (empty_data_class gets chosen) > > - an empty tree node = a node that contains no text but has children > > (tree_class gets chosen) > > Right. There are a few more rules: > has no parent -> tree > has xsi:nil attribute -> NullElement > has parsable type -> type class > > > > This is to not force an objectify user to follow our arbitrary (though with > > good reason ;-) decision to use StringElement for empty leaves. > > Right? > > For that, yes, and for easily replacing the inner tree classes. I think that > can be a pretty helpful thing if you want to extend the API. You can now > replace the type classes through PyType, and the inner tree class and the > default leaf class through the lookup mechanism. I think that's all you might > need to extend objectify. Understood, thanks. > > > What about adding the attribute in objectify.Element()? You can't > > > normally > > > change the data value of an Element itself, so the only real reason why > > > you > > > would call objectify.Element() is to create a structural element (usually > > > a root node). > > > > > > I called the corresponding pytype value "TREE" for now, I think it's > > > unlikely that someone would use that as custom type name. > > > > Great, I'll try it out. But I'm still voting for a TreeElement() factory > > as I'll have to write s.th. like this anyway: > > > > def TreeElement(): > > return objectify.Element('tree', {objectify.PYTYPE_ATTRIBUTE: 'TREE'}) > > No, that's redundant. Just use > > objectify.Element('tree') Fine. Just wanted to try it out and got this error: >>> objectify.Element('foo', {'a': '1'}) Traceback (most recent call last): File "<stdin>", line 1, in ? File "objectify.pyx", line 1481, in objectify.Element NameError: python I don't really understand that as python.<whatever> is used all over the place. Has this s.th. to with python being cimported and so far having only been used in the cdef'ed stuff but now it shows up in a def'ed module function? I must admit I haven't actually looked very much at pyrex itself. Will now try to add an import python somewhere... > I'll add a countchildren() method, then. > > Stefan Thanks, Holger Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene Empfänger sind oder falls diese E-Mail irrtümlich an Sie adressiert wurde, verständigen Sie bitte den Absender sofort und löschen Sie die E-Mail sodann. Das unerlaubte Kopieren sowie die unbefugte Übermittlung sind nicht gestattet. Die Sicherheit von Übermittlungen per E-Mail kann nicht garantiert werden. Falls Sie eine Bestätigung wünschen, fordern Sie bitte den Inhalt der E-Mail als Hardcopy an. The contents of this e-mail are confidential. If you are not the named addressee or if this transmission has been addressed to you in error, please notify the sender immediately and then delete this e-mail. Any unauthorized copying and transmission is forbidden. E-Mail transmission cannot be guaranteed to be secure. If verification is required, please request a hard copy version.

2 1

[lxml-dev] Static win32 builds for lxml 1.0.4
by Ashish Kulkarni 15 Sep '06

15 Sep '06

Hello, I finally got the VC++ Toolkit 2003 compiler, and have made static builds for lxml 1.0.4. They are available at: http://puggy.symonds.net/~ashish/downloads/ Regards, ashish

2 4

[lxml-dev] lxml 1.1 released
by Stefan Behnel 13 Sep '06

13 Sep '06

Hi all, I'm proud to announce the release of lxml 1.1. It is a major new release that builds upon the changes in 1.0.4 and introduces many new features compared to the 1.0 series. The main improvements are: * threading support * XPath axis iteration * iterparse() and iterwalk() * configurable Element class lookup methods * lxml.objectify - a data binding API on top of lxml.etree The complete changelog is below. lxml 1.1 has been tested in an alpha and beta version and received various bug fixes before the final release. It is currently considered stable and ready for production use, whereas the 1.0 series are considered rock-stable and may still be the right thing to choose for more conservative environments. Bug-fix releases and minor improvements to the 1.1 and 1.0 series will continue to become available at need. That way, 1.1 will soon be considered as stable as 1.0 to become its the main line successor. Given the many helpful egg contributions we received for past releases, I hope that 1.1 (and 1.0.4) will be as well supported in that regard to help in further increasing our community. Thanks to everyone who helped in getting 1.1 done, in getting bugs fixed and helping others in using it. Have fun, Stefan Changes since 1.0.4: 1.1 (2006-09-13) ================ Features added -------------- * Comments and processing instructions return '' and '<?pi-target content?>' for repr() * Parsers are now the preferred (and default) place where element class lookup schemes should be registered. Namespace lookup is no longer supported by default. Bugs fixed ---------- * filenames with local 8-bit encoding were not supported * 1.1beta did not compile under Python 2.3 * ignore unknown 'pyval' attribute values in objectify * objectify.ObjectifiedElement.addattr() failed to accept Elements and Lists * objectify.ObjectPath.setattr() failed to accept Elements and Lists 1.1beta (2006-08-08) ==================== Features added -------------- * Support for Python 2.5 beta * Unlock the GIL for deep copying documents and for XPath() * New ``compact`` keyword argument for parsing read-only documents * Support for parser options in iterparse() * The ``namespace`` axis is supported in XPath and returns (prefix, URI) tuples * The XPath expression "/" now returns an empty list instead of raising an exception * XML-Object API on top of lxml (lxml.objectify) * Customizable Element class lookup: * different pre-implemented lookup mechanisms * support for externally provided lookup functions * Support for processing instructions (ET-like, not compatible) * Public C-level API for independent extension modules Bugs fixed ---------- * XPathSyntaxError now inherits from XPathError * Threading race conditions in RelaxNG and XMLSchema * Crash when mixing elements from XSLT results into other trees, concurrent XSLT is only allowed when the stylesheet was parsed in the main thread * The EXSLT ``regexp:match`` function now works as defined (except for some differences in the regular expression syntax) * Setting element.text to '' returned None on request, not the empty string * ``iterparse()`` could crash on long XML files * Creating documents no longer copies the parser for later URL resolving. For performance reasons, only a reference is kept. Resolver updates on the parser will now be reflected by documents that were parsed before the change. Although this should rarely become visible, it is a behavioral change from 1.0. 1.1alpha (2006-06-27) ===================== Features added -------------- * Module level ``iterwalk()`` function as 'iterparse' for trees * Module level ``iterparse()`` function similar to ElementTree (see documentation for differences) * Element.nsmap property returns a mapping of all namespace prefixes known at the Element to their namespace URI * Reentrant threading support in RelaxNG, XMLSchema and XSLT * Threading support in parsers and serializers: * All in-memory operations (tostring, parse(StringIO), etc.) free the GIL * File operations (on file names) free the GIL * Reading from file-like objects frees the GIL and reacquires it for reading * Serialisation to file-like objects is single-threaded (high lock overhead) * Element iteration over XPath axes: * Element.iterdescendants() iterates over the descendants of an element * Element.iterancestors() iterates over the ancestors of an element (from parent to parent) * Element.itersiblings() iterates over either the following or preceding siblings of an element * Element.iterchildren() iterates over the children of an element in either direction * All iterators support the ``tag`` keyword argument to restrict the generated elements * Element.getnext() and Element.getprevious() return the direct siblings of an element

1 0

[lxml-dev] Unicode munging in element tag and text
by Nicola Larosa 13 Sep '06

13 Sep '06

Hi all, thanks for a great library. :-) I found a rather peculiar behavior in Unicode object handling for element tag and text. It looks like they get converted to a plain string if they only contains ASCII chars, but not always. ElementTree instead always keeps them as Unicode objects. >>> from lxml.etree import Element as lxElem >>> from elementtree.ElementTree import Element as etElem 1) Let's first build an element from a Unicode object with ASCII chars; only ElementTree keeps it as Unicode: >>> lx = lxElem(u'ascii') >>> et = etElem(u'ascii') >>> lx.tag 'ascii' >>> et.tag u'ascii' while when the Unicode object contains non-ASCII chars, both libraries correctly keep it as Unicode: >>> lx = lxElem(u'mòrèthànàscìì') >>> et = etElem(u'mòrèthànàscìì') >>> lx.tag u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec' >>> et.tag u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec' 2) The same happens for the element text; ASCII: >>> lx.text = u'ascii' >>> et.text = u'ascii' >>> lx.text 'ascii' >>> et.text u'ascii' non-ASCII: >>> lx.text = u'mòrèthànàscìì' >>> et.text = u'mòrèthànàscìì' >>> lx.text u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec' >>> et.text u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec' 3) OTOH, when directly setting the element tag, lxml keeps the Unicode object too: >>> lx.tag = u'ascii' >>> et.tag = u'ascii' >>> lx.tag u'ascii' >>> et.tag u'ascii' while both libraries keep working correctly when using non-ASCII chars: >>> lx.tag = u'mòrèthànàscìì' >>> et.tag = u'mòrèthànàscìì' >>> lx.tag u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec' >>> et.tag u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec' This inconsistent behavior does not seem intentional. In my opinion, in the cases 1) and 2) lxml should work as it already does in the case 3), and as ElementTree always does. Thanks again. -- Nicola Larosa - http://www.tekNico.net/ There is more money being spent on breast implants and Viagra today than on Alzheimer's research. This means that by 2040, there should be a large elderly population with perky boobs and huge erections and absolutely no recollection of what to do with them. -- David Icke, April 2006

3 6

Re: [lxml-dev] File name encoding problems
by Stefan Behnel 10 Sep '06

10 Sep '06

Hi, Paweł Pałucha wrote: > Stefan Behnel wrote: > >> first thing to note: lxml uses UTF-8 internally, also for filenames, as >> libxml2 requires a char sequence for their representation. If your system >> can't handle that, we'll have to figure out a way to make it work. > > Looking at the code of libxml2, filenames are passed as 'const char *' and not > 'const xmlChar *'. with "char*" == "xmlChar*", but that's not what I meant. >> Right, I guess that treating the filename with the _utf8() function is not the >> right thing to do for 8-bit strings. We should have a separate way of treating >> filenames. I'll look into it. > > I'm not sure, but perhaps the only way is to leave this as user responsibilty, > and pass filenames without any modifications. There's no way to check what > encoding is used for file name. At least, there is sys.getfilesystemencoding(), which tells us what it *should* be. > As for libc open() it's just a NULL terminated > data. But I don't know if Python can handle it... Python just passes it on. >> I assume you get the same file-not-found error if you pass the filename as >> unicode string? >> >> d = lxml.etree.parse(u'/tmp/ó.xml') > > Yes, btw - the file-not-found error is just "IOError%" - not very helpful message. Might have been because of the same problem as the exception you got from funicode(). >>> This did not happen with python 2.3 - the problem is only with >>> python2.4. There's a workaround - reading file to StringIO object and >>> then parsing XML from that object. It works fine but it's silly. >> You can also pass an opened file object. > > No ;-) That's strange but code like that: > > f = open('/tmp/ó.xml') > d = lxml.etree.parse(f) > > behaves the same way (assertion) - it looks like you check the file name > remembered in file object. Possibly just another side effect of funicode. :) I fixed some more places where this was used, so here is a new patch. Stefan PS: please remember replying to the list also, so that this discussion gets a) archived and b) looked at by others.

3 7