Mailman 3 July 2007 - lxml - The Python XML Toolkit

Re: [lxml-dev] objectify E-factory does not handle unicode text
by Stefan Behnel July 5, 2007

July 5, 2007

Hi Holger, jholg(a)gmx.de wrote: >>> What about simply adding an optional argument python_type where one can >> supply the actual python type/class the custom element class does mimic? >> >> Well, all you'd really need is a conversion to a string, so, given such a >> type >> would Do The Right Thing for __str__, that would work. But then, if str() >> did >> the right thing, we could just as well use the existing behaviour of the E >> factory and just extend typemap to also check for the type /name/, not >> only >> the type itself. Maybe that's the way to go? > > You mean by using a customized typemap that uses additional(typename, > <function>) entries and a get() method that also tries to lookup by > typename? The convention then being that the typenames one uses in the > PyType registry must correspond to the actual python type name he models, > if he wants to make use of objectify.E for custom DataElements. Sounds > reasonable. Here is a patch that (I think) might be a way to solve this problem. The idea is to use a custom class instead of the typemap dictionary and have it fall back to the PyType registry. If your type does not support a simple str() conversion, you can pass a conversion function as an additional argument to PyType() when registering your type. It's currently untested, so please play with it if you think it's the right approach. Stefan

2 3

[lxml-dev] xslt exceptions
by Roger Patterson July 4, 2007

July 4, 2007

Hello, I am getting errors from within an XPath call within a custom extension being called while doing an XSLT transform. I am able to access the global error log as well as the error_log on the transform object, but the error information is sketchy at best. Unfortunately no exception is being thrown. Is this normal? Is there a way of turning on exception throwing? -Roger

2 1

[lxml-dev] objectify E-factory does not handle unicode text
by jholg＠gmx.de July 4, 2007

July 4, 2007

Hi, playing around with the new E-factory I found that it does not handle unicode the way the rest of the API does: >>> STR = objectify.E.str >>> STR(unicode("äöü", 'latin-1')) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/data/pydev/hjoukl/LXML/lxml-1.3/build/lib.solaris-2.8-sun4u-2.4/lxml/builder.py", line 43, in <lambda> return lambda *args, **kwargs: func(tag, *args, **kwargs) File "/data/pydev/hjoukl/LXML/lxml-1.3/build/lib.solaris-2.8-sun4u-2.4/lxml/builder.py", line 177, in __call__ v = t(elem, item) File "objectify.pyx", line 1661, in objectify.__add_text UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) >>> This is easily fixed by changing __add_text to def __add_text(_Element elem not None, text): cdef tree.xmlNode* c_child if not python._isString(text): if isinstance(text, bool): text = str(text).lower() else: text = str(text) c_child = cetree.findChildBackwards(elem._c_node, 0) [...] >>> STR = objectify.E.str >>> STR(unicode("äöü", 'latin-1')) Patches for trunk / 1.3 branch appended. Another issue with E-factory is that it currently does not have support for the custom objectify classes that you can add with the PyType mechanisms: E.g. I'm using datetime and decimal additions, which leads to >>> import decimal >>> DEC = objectify.E.decimal >>> DEC(decimal.Decimal(0)) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/data/pydev/hjoukl/LXML/lxml-1.3/build/lib.solaris-2.8-sun4u-2.4/lxml/builder.py", line 43, in <lambda> return lambda *args, **kwargs: func(tag, *args, **kwargs) File "/data/pydev/hjoukl/LXML/lxml-1.3/build/lib.solaris-2.8-sun4u-2.4/lxml/builder.py", line 175, in __call__ raise TypeError("bad argument type: %r" % item) TypeError: bad argument type: Decimal("0") >>> So I'd have to add decimal.decimal into objectify.E._typemap. The nicest way to handle this would be PyType.register() doing it for me, but PyType uses type names rather than type objects for its purposes. Maybe the easiest thing is to instrument ElementMaker with its own register/unregister(<type>) methods and well-document it? Holger -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer

2 4

[lxml-dev] lxml 1.3.2 released to cheeseshop
by Stefan Behnel July 3, 2007

July 3, 2007

Hi all, due to a severe crash bug in 1.3.1, I released 1.3.2 today. The only change is the bug fix. So please don't use or package 1.3.1, use 1.3.2 instead. Have fun, Stefan ChangeLog: 1.3.2 (2007-07-03) ================== Bugs fixed ---------- * "deallocating None" crash bug 1.3.1 (2007-07-02) ================== Features added -------------- * objectify.DataElement now supports setting values from existing data elements (not just plain Python types) and reuses defined namespaces etc. * E-factory support for lxml.objectify (``objectify.E``) Bugs fixed ---------- * Better way to prevent crashes in Element proxy cleanup code * objectify.DataElement didn't set up None value correctly * objectify.DataElement didn't check the value against the provided type hints * Reference-counting bug in ``Element.attrib.pop()``

1 0

[lxml-dev] Ref-counting bug returns in 1.3.1
by David M. Grimes July 3, 2007

July 3, 2007

I posted a patch for what I believed to be a reference-counting bug in Attrib.pop() based on the 1.3 release. The patch was accepted, and is present in 1.3.1. The patch is included at the end of this message. Looking through the generated C code, I'm no longer sure my patch was correct - perhaps just masking the underlying problem in 1.3. I'm not fluent in Pyrex, so not sure if the python.Py_INCREF is really necessary for something which would be a "borrwed reference" in the C-API (PyTuple_GET_ITEM result). It looks like the Pyrex "return" is generating it's own INCREF ... Now, what is intriguing is that the 1.3.1 stock build is crashing again with the same symptom, and is easily reproducable with the following test program (this crashed after iteration 956 in i686 with python 2.4.4: import lxml.etree as etree xml = '''\ <?xml version="1.0"?> <xml/>''' for i in range(10000): print i et = etree.fromstring(xml) et.attrib.pop('x', None) This dies at this point: ... ... 951 952 953 954 955 956 Fatal Python error: deallocating None Aborted Original 1.3 patch: diff -urN lxml-1.3~/src/lxml/etree.pyx lxml-1.3/src/lxml/etree.pyx --- lxml-1.3~/src/lxml/etree.pyx 2007-06-25 02:25:37.000000000 -0400 +++ lxml-1.3/src/lxml/etree.pyx 2007-06-27 15:36:15.000000000 -0400 @@ -1480,10 +1480,12 @@ if python.PyTuple_GET_SIZE(default) == 0: raise KeyError, key else: - return python.PyTuple_GET_ITEM(default, 0) + result = python.PyTuple_GET_ITEM(default, 0) + python.Py_INCREF(result) else: _delAttribute(self._element, key) - return result + + return result def clear(self): cdef xmlNode* c_node

3 3

[lxml-dev] lxml 1.3.1 setup.py bug
by jholg＠gmx.de July 3, 2007

July 3, 2007

Hi, the setup.py script in 1.3.1 seems to try to remove the dependency on setuptools (which is a very good thing imho!) but fails: Traceback (most recent call last): File "setup.py", line 7, in ? except pkg_resources.VersionConflict, e: NameError: name 'pkg_resources' is not defined 1 lb54320@adevp02 .../lxml-1.3 $ I must admit I don't fully undestand the intention of the relevant code portion, as it raises ImportError even if pkg_resources import and version check runs smoothly; maybe this is the intended behaviour? try: import pkg_resources try: pkg_resources.require("setuptools>=0.6c5") except pkg_resources.VersionConflict, e: from ez_setup import use_setuptools use_setuptools(version="0.6c5") from setuptools import setup except ImportError: # not setuptools installed from distutils.core import setup (Note: This is untested code, I have not tested with setuptools installed) Oh, btw I couldn't find a 1.3.1 tag in the repository when trying to check out 1.3.1. Holger -- Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kanns mit allen: http://www.gmx.net/de/go/multimessenger

2 2

Re: [lxml-dev] Some XPath questions...
by Ian Bicking July 3, 2007

July 3, 2007

Mike Meyer wrote: > In <4689898E.9080509(a)colorstudy.com>, Ian Bicking <ianb(a)colorstudy.com> typed: >> Stefan Behnel wrote: >>>> So when I use // it works. Huh. I prefer descendant-or-self, because I >>>> find it peculiar to do a search from the root when you've called the >>>> method on some particular element (that may not be at the root). >>> There's also ".//*". >> That seems to be equivalent to //*, i.e., // goes directly to the root >> regardless of context. > > Not quite. '//*' always goes to the root. './/*' starts at the current > node and matches from there down. If you always test at the root of > the document, they'll look the same. It seems to be changing the results when I replace 'descendant-or-self::' with './/'. I want to include the current node if it matches; at least to me, that seems most logical. Also necessary when I was doing microformat parsing, as a single element can have multiple roles. It seems like .// excludes the current node, only looking at descendants. >>>>>>>> div:empty (no children, including text, maybe not including whitespace). >>>>>>> Ouch. let me think about that one. >>>>>> Yeah, I couldn't figure that one out. I thought this might work: >>>>>> >>> xpath('E:empty') >>>>>> e[count(./children::*) = 0 and string(.) = ''] >>>>>> But maybe I don't understand how count() works; this isn't a valid XPath >>>>>> expression. >>>>> You want "child" not "children". Using normalize-space(.) instead of >>>>> string(.) will exclude whitespace. This does assume you are ignoring >>>>> comments and PIs; I believe that's the behavior you want. >>>> Cool, that seems to work right. >>> What about "e[not(*) and not(normalize-space())]" ? >> Yes, that works too. > > That's the 'implicit conversion' I was talking about. You're relying > on 0 and the empty string being false. It's a standard idiom, and > pythonic, but I'm not sure you want to use it in automatically > generated code, since it means you can't generalize the code from "has > 0 children" to "has n children". In this case it's a fixed expression used for e:empty, and nothing else, so it seems fine. And possibly makes the resulting expression a bit easier to recognize from its CSS roots. -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

2 1

Re: [lxml-dev] Some XPath questions...
by Ian Bicking July 2, 2007

July 2, 2007

Mike Meyer wrote: > In <468579E3.7010802(a)colorstudy.com>, Ian Bicking <ianb(a)colorstudy.com> typed: >> Thanks, very helpful. I'm guessing it was an oversight that you didn't >> copy the list... > > I wasn't sure which way to go. Without CC'ing people won't know you've already answered my questions. >>>> There's some other tricky ones I'm not sure about either, though they >>>> seem to be kind of working. Things like div:only-child (when it's a div >>> //*[name() = 'div' and last() = 1] >> This doesn't seem to be working for me: >> >>> xpath('span:only-child') >> *[name() = 'span' and (last() = 1)] >> But testing with <div><span></span></div> in the document, I don't get >> anything returned. > > Seems to work ok for me: > >>>> d = fromstring('<html><div><span></span></div><span /></html>') >>>> d.xpath('//span') > [<Element span at 918500>, <Element span at 9182d0>] >>>> d.xpath('//*[name() = "span" and last() = 1]') > [<Element span at 918500>] > > But I may be missing something. Got a full test case? Here's a small example: >>> from lxml.etree import HTML >>> h = '<html><body><div></div><div><span></span></div></body></html>' >>> h = HTML(h) >>> h.xpath("descendant-or-self::*[name() = 'span' and (last() = 1)]") [] >>> h.xpath("//*[name() = 'span' and (last() = 1)]") [<Element span at -4866f944>] So when I use // it works. Huh. I prefer descendant-or-self, because I find it peculiar to do a search from the root when you've called the method on some particular element (that may not be at the root). >>>> div:empty (no children, including text, maybe not including whitespace). >>> Ouch. let me think about that one. >> Yeah, I couldn't figure that one out. I thought this might work: >> >>> xpath('E:empty') >> e[count(./children::*) = 0 and string(.) = ''] >> But maybe I don't understand how count() works; this isn't a valid XPath >> expression. > > You want "child" not "children". Using normalize-space(.) instead of > string(.) will exclude whitespace. This does assume you are ignoring > comments and PIs; I believe that's the behavior you want. Cool, that seems to work right. One query I'm realizing might be really hard (maybe too hard in XPath) is *:first-of-type, *:last-of-type, and *:only-of-type, since they match in a funny sort of way. You can't really do: *[count(../*[name() = name()) = 1] But it's kind of what *:only-of-type means. Or: *[count(following-sibling::name()) = 0 and count(previous-sibling::name()) = 0] You just can't use name() that way. Hmm... well, it's not that important of a query to me, I guess, so maybe I'll just catch it and give an error. -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

2 2

[lxml-dev] Some XPath questions...
by Ian Bicking July 2, 2007

July 2, 2007

I'm trying to implement CSS selectors, by translating them into XPath. There's some CSS expressions that I'm having a hard time with, so maybe someone can tell me how they might work. Expression: div:first-child -- means a div element when it is the first child of its parent. I.e.: <li> <div id="a">...</div> <div id="b">...</div> </li> It makes the first div and not the second. I thought this could be: descendant-or-self::*/div[0] or... descendant-or-self::*/div[position() = 0] Those two should be equivalent; the second is a bit easier to handle programmatically. But it doesn't work (doesn't match anything). Another expreesion: div.foo + div -- means a div element that is the immediately next sibling of a div element with the class .foo. I would translate this to: descendant-or-self::div[@class='foo']/following-sibling::div[0] (The class matching is actually a bit more complex, but it doesn't actually matter to this.) I'm (a) not sure if this is right, because maybe it means the next div after the matching div, even if there's another element in-between, and (b) it doesn't return any results regardless. Another expression: div:contains('celia') -- means a div where the textual content has the word 'celia' in it, case insensitive. At least, I think it's case insensitive -- the CSS spec is annoyingly vague, but implementations seem to work like this. I translate this to: descendant-or-self::div[contains(css:lower-case(string(.)), 'celia'] I added the lower-case function like: def _make_lower_case(context, s): return s.lower() etree.FunctionNamespace("css")['lower-case'] = _make_lower_case But XPath gives so few errors that it's hard to tell if it's really working. The XPath expression returns some elements, but not the correct number from what I can tell. Especially since when I had a bug and wasn't lowercasing the second argument (using 'CELIA') it still returned elements. There's some other tricky ones I'm not sure about either, though they seem to be kind of working. Things like div:only-child (when it's a div with no siblings), div:last-child (no next sibling), div:first-child (no previous sibling), div:first-of-type (no preceding siblings that are divs), div:last-of-type (no following siblings that are divs), div:only-of-type (you are probably getting the pattern), div:empty (no children, including text, maybe not including whitespace). There's also div:nth-child(matcher) and div:nth-of-type(matcher), which selects among siblings with patterns like "2" (second sibling), "3n" (every third element), "odd" (odd elements) and some other selections. I kind of see how to deal with this using position(), but I'm not sure how to do either nth-of-type or nth-child (and the ones I do understand I am also vague about). I've committed the incomplete code in lxml.html.css -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

2 2

[lxml-dev] lxml 1.3.1 on cheeseshop
by Stefan Behnel July 2, 2007

July 2, 2007

Hi all, I just released lxml 1.3.1. This is a bugfix release for the stable 1.3 series. Changelog follows. Have fun, Stefan 1.3.1 (2007-07-02) ================== Features added -------------- * objectify.DataElement now supports setting values from existing data elements (not just plain Python types) and reuses defined namespaces etc. * E-factory support for lxml.objectify (``objectify.E``) Bugs fixed ---------- * Better way to prevent crashes in Element proxy cleanup code * objectify.DataElement didn't set up None value correctly * objectify.DataElement didn't check the value against the provided type hints * Reference-counting bug in ``Element.attrib.pop()``

1 0