Mailman 3 February 2009 - lxml - The Python XML Toolkit

[lxml-dev] how to inherit from ElementBase?
by TP 23 Feb '09

23 Feb '09

Hi everybody, For my application, I try to inherit the ElementBase class of lxml. I have read the following page: http://codespeak.net/lxml/dev/api/lxml.etree.ElementBase-class.html "Class ElementBase: The public Element class. All custom Element classes must inherit from this one. To create an Element, use the Element() factory." So I have tried to define a new class deriving from ET.ElementBase, and set the factory (__metaclass__) to ET.Element. You will find my code below. I obtain: $ p test_element_factory.py Traceback (most recent call last): File "test_element_factory.py", line 3, in <module> class NewElement( ET.ElementBase ): File "etree.pyx", line 1844, in etree.Element File "apihelpers.pxi", line 129, in etree._makeElement File "apihelpers.pxi", line 116, in etree._makeElement File "etree.pyx", line 362, in etree._Document._setNodeNamespaces File "apihelpers.pxi", line 651, in etree._utf8 TypeError: Argument must be string or unicode. What is the problem? Thanks in advance, Julien ############################## import lxml.etree as ET class NewElement( ET.ElementBase ): __metaclass__ = ET.Element def _init( self , *args , **kwargs ): super( NewElement, self ).__init__( self , *args , **kwargs ) a = NewElement( "root" ) print a ############################## -- python -c "print ''.join([chr(154 - ord(c)) for c in '*9(9&(18%.\ 9&1+,\'Z4(55l4('])" "When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong." (first law of AC Clarke)

2 3

Re: [lxml-dev] Behaviour change in findtext
by Stefan Behnel 23 Feb '09

23 Feb '09

Hi Fredrik, thanks for the clarification. Fredrik Lundh wrote: > Not sure - that you can get None back from findtext when the element > is there looks like an accidental change when the ElementPath engine > was rewritten. I think I'll consider that a bug in findtext. I thought so, too. > As for distinguishing between <element/> and <element></element> That's not what I meant, although that actually is the result when you serialise with or without an empty string value. A parsed empty element will always have its .text set to None in lxml.etree, regardless of the way the parser saw it. I rather meant the difference between users setting el.text = None and el.text = '' in the code. In the second case, lxml.etree creates a text node with an empty string in the underlying libxml2 tree. That way, it can return the expected result on later requests. This is actually compatible with ET, which (obviously) also remembers what the user set as value. You can think of the above as an emulation of the ET behaviour, but also as a way to prevent surprised faces on user side when you see el.text = '' for i in range(10: el.text += 'xyz' fail mysteriously. > the ET specification allows an implementation to use either > None or an empty string for the text and tail attributes in either > case to simplify the tree building. However, an application shouldn't > abuse this - an XML producer should be free to use either form to > indicate an empty element, and application code should use "truth > testing" when necessary, when inspecting the text/tail attributes of a > given element. I fully agree. > And I think findtext should be reverted to the 1.2 > behaviour - just add an <or ""> to the suitable place in ElementPath, > and leave the rest as is. That's what I did for lxml 2.2. It just makes findtext() simpler to use. Stefan

2 1

[lxml-dev] LXML utf-8 problem...
by Douglas Mayle 21 Feb '09

21 Feb '09

Hi all, Unfortunately, I'm running into an error that I thought I had licked before. I've running lxml 2.1.2 on OS X and python 2.5. I have a 'str' object that contains html with utf-8 bytes and a utf-8 encoding specified by the directive, which should be properly handled, to my understanding, but is not: douglas$ python Python 2.5.1 (r251:54863, Jul 23 2008, 11:00:16) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import lxml.html >>> lxml.html.parse(u'<?xml version="1.0" encoding="utf-8"? ><html><body><p>\xa9</p></body></html>'.encode('utf-8')) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/private/var/folders/Qk/QkUDmW61GouWAJY+hKgwL++++TI/-Tmp-/ tmp0Dd1ub/lxml-2.1.2-py2.5-macosx-10.5-i386.egg/lxml/html/ __init__.py", line 651, in parse File "lxml.etree.pyx", line 2578, in lxml.etree.parse (src/lxml/ lxml.etree.c:25269) File "parser.pxi", line 1466, in lxml.etree._parseDocument (src/ lxml/lxml.etree.c:63768) File "parser.pxi", line 1495, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:64012) File "parser.pxi", line 1395, in lxml.etree._parseDocFromFile (src/ lxml/lxml.etree.c:63169) File "parser.pxi", line 969, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:60461) File "parser.pxi", line 538, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c: 56751) File "parser.pxi", line 624, in lxml.etree._handleParseResult (src/ lxml/lxml.etree.c:57595) File "parser.pxi", line 558, in lxml.etree._raiseParseError (src/ lxml/lxml.etree.c:56936) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 53: ordinal not in range(128) >>> Why is ascii being used as a codec? It's properly identified in the string. It's a valid character (in this case a copyright symbol). What can I do?

3 3

[lxml-dev] Behaviour change in findtext
by Aloys Baillet 20 Feb '09

20 Feb '09

Hi, I was planning on upgrading to a recent version of lxml but found that our code was failing in numerous places with None objects found in unexpected places. In lxml 2+ the findtext method will ignore the default and return None if the element is found but the text is empty. In elementtree and lxml before 2 the findtext method would never return None, if the element is found but empty it would return the default. Unfortunately I am unable to tell which exact version of lxml introduced that change. And I am not sure if this change is by design, but I would say it is not. A proposed fix (if it is a bug...) is included in the attached unified diff. Cheers, Aloys -- Aloys Baillet Research & Development - Animal Logic --

2 1

Re: [lxml-dev] [Question #61584]: Is it possible to make lxml use hex instead of decimal for unicode entities?
by Stefan Behnel 19 Feb '09

19 Feb '09

usernamenumber wrote: > Thanks very much for the assistance, Stefan. You are a great help! As it > turns out, write_c14n() actually uses yet another method of rendering > entities, (\xc2\xa9 as opposed to ©) Ah, right. Sure, it serialises to UTF-8, which doesn't require Unicode character escaping. What you see is just what the Python prompt makes of the byte series on output. > so it looks like I may have > to just suck it up and deal with the output of my port being slightly > different from that of the original tool (I don't think it's worth the > extra processing to translate all the entities after the fact). But > being able to write out to C14N (which I hadn't known about before now), > might at least be able to avoid this problem in the future. Wise choice. > I do have one other question coming from this: I can find functions for > writing out c14n content for ElementTree objects, but nothing for > rendering an Element (the result of etree.fromstring(), for example) in > this way. Am I missing something, or if I am working with a string do I > just need to load it into a StringIO and run etree.parse() on it? You can get an ElementTree either by calling parse() or by wrapping an Element in it, i.e. tree = etree.ElementTree(root_node) http://codespeak.net/lxml/tutorial.html#the-elementtree-class http://effbot.org/zone/element.htm#reading-and-writing-xml-files http://effbot.org/zone/pythondoc-elementtree-ElementTree.htm#elementtree.El… Stefan

1 0

Re: [lxml-dev] [Question #61584]: Is it possible to make lxml use hex instead of decimal for unicode entities?
by Stefan Behnel 19 Feb '09

19 Feb '09

usernamenumber wrote: > I am porting a perl/SAX tool to python/lxml. Ideally, given the same > input, the new tool should produce the same output as the old tool. In > fact, it introduces a number of problems for me if this is not the case. It's always bad style to make applications depend on a specific XML serialisation done by a specific tool. That's exactly what canonical XML (C14N) was designed for. > One annoying problem I am encountering is that SAX seems to store unicode > entity IDs in hex, whereas lxml uses decimal, regardless of what value is > used in the input: > > >>> import lxml.etree as etree > >>> example_sax_output = "<foo>Copyright © 2009 Foocorp, Inc</foo>" # Note: xA9 > >>> e = etree.fromstring(example_sax_output) > >>> etree.tostring(e) > <foo>Copyright © 2009 Foocorp, Inc</foo> # Note: 169 > > Is it possible to avoid this without doing something horribly kludgey > like going through the output with a regex search and manually > converting the values to hex? There isn't a straight way to do that. Decimal character references were chosen for compatibility with ElementTree, which uses "xmlcharrefreplace". However, if you have a bit of memory and do not care too much about raw performance, you can do this: # Python 2.6 unicode_xml = etree.tostring(tree, encoding=unicode) bytes_xml = b''.join(chr(c) if c < 0x80 else b'&#x%X;' % c for c in imap(ord, unicode_xml)) There's also a separate serialiser API in libxml2 that happens to output hex entities. However, that's not used for backward compatibility reasons. Stefan

1 1

[lxml-dev] lxml 2.2beta3 released
by Stefan Behnel 17 Feb '09

17 Feb '09

Hi all, I'm happy to release the third (and hopefully last) beta of lxml 2.2. This release features some major bug fixes, so upgrading is highly recommended. http://pypi.python.org/pypi/lxml/2.2beta3 http://codespeak.net/lxml/dev/ It was generated with revision 1716 of the Cython 0.11 development branch, which was hardened against various memory leaks using Dag Seljebotn's ref-nanny. This release also marks the end of the lxml 2.1 maintenance cycle. As I stated a while ago, I think that the stability of the current trunk makes backporting fixes to older release series no longer worth the effort. The only reason why this beta version was not released as lxml 2.2 final is that I want to wait for an official Cython release series to go with lxml 2.2. Therefore a note to distributors: please start shipping lxml 2.2 as soon as possible. Note that while this release does not officially support Python 3.0, it should work well in that environment in almost all situations. It was tested with Python 3.0.1. Hope you like it, Stefan 2.2beta3 (2009-02-17) Features added * XSLT.strparam() class method to wrap quoted string parameters that require escaping. Bugs fixed * Memory leak in XPath evaluators. * Crash when parsing indented XML in one thread and merging it with other documents parsed in another thread. * Setting the base attribute in lxml.objectify from a unicode string failed. * Fixes following changes in Python 3.0.1. * Minor fixes for Python 3. Other changes * The global error log (which is copied into the exception log) is now local to a thread, which fixes some race conditions. * More robust error handling on serialisation.

1 0

[lxml-dev] altering the indent of the pretty output
by Ronny Pfannschmidt 08 Feb '09

08 Feb '09

Hi, i'm currently porting gazpacho (a wysiwyg gtk ui file editor) to lxml, unfortunately the pretty printer prints with an indent of 2 and in order to match the convention i need an indent of 4 is there any simple way to archive pretty dumping to a file with an indent of 4? Regards Ronny

3 3

Re: [lxml-dev] etree.clear_error_log() causes segfaults
by Stefan Behnel 07 Feb '09

07 Feb '09

Christian Heimes wrote: > I guess it was a relict from the time lxml.etree didn't clear the local > error log when an XSLT document was called. I think you are referring to the error log on XSLT objects, which is actually local to the object and cleared before each transformation. (A copy of) the global error log is attached to exceptions, so you will notice the difference when you do multiple transformations and one of the later ones raises an exception. I guess there might still be space left for improvements... Stefan

1 0

[lxml-dev] etree.clear_error_log() causes segfaults
by Christian Heimes 07 Feb '09

07 Feb '09

This is a friendly word of warning! Don't call etree.clear_error_log() from multiple threads. We found this issue during stress tests of our CherryPy based application. Every worker thread was calling etree.clear_error_log() after the page was rendered. Apparently we hit some sort of race condition. Backtrace: C [etree.so+0x21977] C [etree.so+0x22d48] C [libxslt.so.1+0xdc17] xsltTransformError+0xf7 C [libxslt.so.1+0x839d] C [libxslt.so.1+0xa5da] xsltParseStylesheetProcess+0x80a C [libxslt.so.1+0x1efbc] xsltParseStylesheetInclude+0x1ac C [libxslt.so.1+0xa0b4] xsltParseStylesheetProcess+0x2e4 C [libxslt.so.1+0xb4b9] xsltParseStylesheetImportedDoc+0x1e9 C [libxslt.so.1+0xb5b8] xsltParseStylesheetDoc+0x28 C [etree.so+0xe640c] C [python2.5+0x4ff83] C [python2.5+0x11e67] PyObject_Call+0x27 Christian

2 2