Mailman 3 June 2007 - lxml - The Python XML Toolkit

[lxml-dev] Building LXML Trunk
by Sidnei da Silva Aug. 31, 2022

Aug. 31, 2022

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

[lxml-dev] Pickling objectified trees
by Christian Zagrodnick June 9, 2008

June 9, 2008

Hi, the other day I had to pickle objectified trees. I just thought to share my findings. Pickling is about serialization. IMHO the natural serialization of an objectified tree is its XML representation. So the following basically does that: -------------------------- import copy_reg import lxml.etree import lxml.objectify def treeFactory(state): """Un-Pickle factory.""" return lxml.objectify.fromstring(state) copy_reg.constructor(treeFactory) def reduceObjectifiedElement(object): """Reduce function for lxml.objectify trees. See http://docs.python.org/lib/pickle-protocol.html for details. """ return (treeFactory, (lxml.etree.tostring(object), )) copy_reg.pickle(lxml.objectify.ObjectifiedElement, reduceObjectifiedElement, treeFactory) ----------------------------------------- You might consider just registering the reduce function in lxml itself. Shouldn't hurt, should it. -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891

2 4

[lxml-dev] lxml namespaces
by Maxim Sloyko Feb. 20, 2008

Feb. 20, 2008

Hi All! I have a little problem with XML namespaces. In my application I have two XML processors, that process the same document, one after the other. The first one looks for nodes in 'ns1' namespace, and substitutes them, according to some algorithm. After this processor is finished, it is guaranteed that there are no more 'ns1' nodes left in the tree. 'ns1' namespace dclaration is still there, in the root node (well, I put it there manually). Now, when this namespace is no longer needed, I want to get rid of it, because it confuses some other processors (namely, my browser) So, the question is, how do I do that? del tree.getroot().nsmap['ns1'] does not seem to do the trick :( Thanks in advance and Happy Holidays! -- Maxim Sloyko

3 5

Re: [lxml-dev] Some XPath questions...
by Ian Bicking July 2, 2007

July 2, 2007

Mike Meyer wrote: > In <468579E3.7010802(a)colorstudy.com>, Ian Bicking <ianb(a)colorstudy.com> typed: >> Thanks, very helpful. I'm guessing it was an oversight that you didn't >> copy the list... > > I wasn't sure which way to go. Without CC'ing people won't know you've already answered my questions. >>>> There's some other tricky ones I'm not sure about either, though they >>>> seem to be kind of working. Things like div:only-child (when it's a div >>> //*[name() = 'div' and last() = 1] >> This doesn't seem to be working for me: >> >>> xpath('span:only-child') >> *[name() = 'span' and (last() = 1)] >> But testing with <div><span></span></div> in the document, I don't get >> anything returned. > > Seems to work ok for me: > >>>> d = fromstring('<html><div><span></span></div><span /></html>') >>>> d.xpath('//span') > [<Element span at 918500>, <Element span at 9182d0>] >>>> d.xpath('//*[name() = "span" and last() = 1]') > [<Element span at 918500>] > > But I may be missing something. Got a full test case? Here's a small example: >>> from lxml.etree import HTML >>> h = '<html><body><div></div><div><span></span></div></body></html>' >>> h = HTML(h) >>> h.xpath("descendant-or-self::*[name() = 'span' and (last() = 1)]") [] >>> h.xpath("//*[name() = 'span' and (last() = 1)]") [<Element span at -4866f944>] So when I use // it works. Huh. I prefer descendant-or-self, because I find it peculiar to do a search from the root when you've called the method on some particular element (that may not be at the root). >>>> div:empty (no children, including text, maybe not including whitespace). >>> Ouch. let me think about that one. >> Yeah, I couldn't figure that one out. I thought this might work: >> >>> xpath('E:empty') >> e[count(./children::*) = 0 and string(.) = ''] >> But maybe I don't understand how count() works; this isn't a valid XPath >> expression. > > You want "child" not "children". Using normalize-space(.) instead of > string(.) will exclude whitespace. This does assume you are ignoring > comments and PIs; I believe that's the behavior you want. Cool, that seems to work right. One query I'm realizing might be really hard (maybe too hard in XPath) is *:first-of-type, *:last-of-type, and *:only-of-type, since they match in a funny sort of way. You can't really do: *[count(../*[name() = name()) = 1] But it's kind of what *:only-of-type means. Or: *[count(following-sibling::name()) = 0 and count(previous-sibling::name()) = 0] You just can't use name() that way. Hmm... well, it's not that important of a query to me, I guess, so maybe I'll just catch it and give an error. -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

2 2

[lxml-dev] Some XPath questions...
by Ian Bicking July 2, 2007

July 2, 2007

I'm trying to implement CSS selectors, by translating them into XPath. There's some CSS expressions that I'm having a hard time with, so maybe someone can tell me how they might work. Expression: div:first-child -- means a div element when it is the first child of its parent. I.e.: <li> <div id="a">...</div> <div id="b">...</div> </li> It makes the first div and not the second. I thought this could be: descendant-or-self::*/div[0] or... descendant-or-self::*/div[position() = 0] Those two should be equivalent; the second is a bit easier to handle programmatically. But it doesn't work (doesn't match anything). Another expreesion: div.foo + div -- means a div element that is the immediately next sibling of a div element with the class .foo. I would translate this to: descendant-or-self::div[@class='foo']/following-sibling::div[0] (The class matching is actually a bit more complex, but it doesn't actually matter to this.) I'm (a) not sure if this is right, because maybe it means the next div after the matching div, even if there's another element in-between, and (b) it doesn't return any results regardless. Another expression: div:contains('celia') -- means a div where the textual content has the word 'celia' in it, case insensitive. At least, I think it's case insensitive -- the CSS spec is annoyingly vague, but implementations seem to work like this. I translate this to: descendant-or-self::div[contains(css:lower-case(string(.)), 'celia'] I added the lower-case function like: def _make_lower_case(context, s): return s.lower() etree.FunctionNamespace("css")['lower-case'] = _make_lower_case But XPath gives so few errors that it's hard to tell if it's really working. The XPath expression returns some elements, but not the correct number from what I can tell. Especially since when I had a bug and wasn't lowercasing the second argument (using 'CELIA') it still returned elements. There's some other tricky ones I'm not sure about either, though they seem to be kind of working. Things like div:only-child (when it's a div with no siblings), div:last-child (no next sibling), div:first-child (no previous sibling), div:first-of-type (no preceding siblings that are divs), div:last-of-type (no following siblings that are divs), div:only-of-type (you are probably getting the pattern), div:empty (no children, including text, maybe not including whitespace). There's also div:nth-child(matcher) and div:nth-of-type(matcher), which selects among siblings with patterns like "2" (second sibling), "3n" (every third element), "odd" (odd elements) and some other selections. I kind of see how to deal with this using position(), but I'm not sure how to do either nth-of-type or nth-child (and the ones I do understand I am also vague about). I've committed the incomplete code in lxml.html.css -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

2 2

[lxml-dev] lxml 1.3 coming up
by Stefan Behnel July 1, 2007

July 1, 2007

Hi all, I finally found some time to work on lxml 1.3 again. Since the current trunk will become 2.0 (though not that soon :), I created a new branch "lxml-1.3" that will become the 1.3 release. I worked through the trunk commits to see which ones to copy over, found quite a few and rejected some bigger ones. I'd be glad if others could give this branch a final try before the release, see if it compiles (e.g. on MS Windows), if it contains the long awaited bug fixes, etc. I'm planning the release for the end of the week. http://codespeak.net/svn/lxml/branch/lxml-1.3/ Note that I tried to avoid behavioural changes, so some 'fixes' might have been left out for that reason. If unsure, it might be worth asking. Have fun, Stefan

5 29

[lxml-dev] Another (last?) take on proxy deallocation
by Stefan Behnel June 30, 2007

June 30, 2007

Hi all, while Ian Bicking was working on the "lxml.html" trunk, I noticed that the way some of the modules were implemented could crash lxml.etree during garbage collection. I know, Martijn and I have already reimplemented the proxy code a couple of times, each time solving more and more of the encountered problems, but I really hope this is the last time we have to reimplement this. I already disliked the last way I had to rewrite it, as it required an additional document traversal step each time a document is deallocated. While we accept this behaviour for disconnected tree fragments (a trade-off between different overheads), it should not be necessary for the whole document (at least not more often than required by xmlFreeDoc()). But the problem is that Python's cyclic garbage collector gives no guarantees about the order in which the collected objects are cleaned up - and libxml2 requires access to the document node when cleaning up a tree fragment. So it is actually required that we clean up all _Element proxies first and *then* free the xmlDoc. So, _Document must really always be deallocated *after* all its _Element proxies have been garbage collected. I was thinking about a way to do this for a while and experimented with it on a "proxy-deallocation" branch - until I realised that the best way to control the garbage collector was the garbage collection mechanism itself - i.e. reference counting. So, I checked in a small patch (SVN revision 44623 on the trunk) that simply doubles the ref-counts that an _Element holds to its _Document so that we can control when the ref-count to the document is decreased. lxml.etree now does this explicitly in the tp_dealloc function of the _Element class, *after* cleaning up the proxy, so that the ref-count of the _Document never goes down to 0 before the last of its _Element proxies was deallocated. It is then safe to run xmlFreeDoc() on the libxml2 document from _Document.__dealloc__. https://codespeak.net/viewvc/?view=rev&revision=44623 I really like this approach and I also like that it removes the need for document traversal on _Document deallocation. And: it keeps the code on the lxml.html branch from crashing, which is a *really* good sign. :] So, I will also merge this into the 1.3 branch and release a 1.3.1 soon. We already had a couple of small fixes on the branch, so a bug-fix release next week should nicely improve the code quality of the official release. Have fun, Stefan

1 0

Re: [lxml-dev] Some XPath questions...
by Ian Bicking June 29, 2007

June 29, 2007

Thanks, very helpful. I'm guessing it was an oversight that you didn't copy the list... Mike Meyer wrote: > In <468538ED.9060004(a)colorstudy.com>, Ian Bicking <ianb(a)colorstudy.com> typed: >> I'm trying to implement CSS selectors, by translating them into XPath. >> There's some CSS expressions that I'm having a hard time with, so maybe >> someone can tell me how they might work. >> >> Expression: >> >> div:first-child -- means a div element when it is the first child of its >> parent. I.e.: >> >> <li> >> <div id="a">...</div> >> <div id="b">...</div> >> </li> >> >> It makes the first div and not the second. >> >> I thought this could be: >> >> descendant-or-self::*/div[0] >> or... descendant-or-self::*/div[position() = 0] > >> Those two should be equivalent; the second is a bit easier to handle >> programmatically. But it doesn't work (doesn't match anything). > > XPath arrays are 1-indexed, not 0-indexed, so position() will never be > 0. I understand some version of IE get this wrong as well. > > To pick out all the div elements that are the first child of their > parent, use: > > //*[position() = 1 and name() = 'div'] > > or equivalently: > > descendant-or-self::*[position() = 1 and name() = 'div'] I don't know how I missed the fact they are 1-indexed... I guess it's become such an unusual choice these days. But handy anyway, since CSS is also 1-indexed. >> Another expreesion: >> >> div.foo + div -- means a div element that is the immediately next >> sibling of a div element with the class .foo. I would translate this to: >> >> descendant-or-self::div[@class='foo']/following-sibling::div[0] >> >> (The class matching is actually a bit more complex, but it doesn't >> actually matter to this.) I'm (a) not sure if this is right, because >> maybe it means the next div after the matching div, even if there's >> another element in-between, and (b) it doesn't return any results >> regardless. > > > Your maybe is right - it means the first div after, whether or not > there are following siblings. You then select the first element from > that list (or would, if you used a 1 instead of a 0). Same solution: > the last bit is following-sibling::*[position() = 1 and name() = 'div'] > >> Another expression: >> >> div:contains('celia') -- means a div where the textual content has the >> word 'celia' in it, case insensitive. At least, I think it's case >> insensitive -- the CSS spec is annoyingly vague, but implementations >> seem to work like this. I translate this to: >> >> descendant-or-self::div[contains(css:lower-case(string(.)), 'celia'] >> >> I added the lower-case function like: >> >> def _make_lower_case(context, s): >> return s.lower() >> etree.FunctionNamespace("css")['lower-case'] = _make_lower_case >> >> But XPath gives so few errors that it's hard to tell if it's really >> working. The XPath expression returns some elements, but not the >> correct number from what I can tell. Especially since when I had a bug >> and wasn't lowercasing the second argument (using 'CELIA') it still >> returned elements. > > I think you've got the parens in the wrong place - the last close goes > after 'celia', not the comma. That was just a typo in the email; copying and pasting directly: >>> xpath('E:contains("foo")') e[contains(css:lower-case(string(.)), 'foo')] However, now that I'm writing my own tests it seems fine (I was using someone else's tests, and I think they were wrong; though I'm not sure -- you'll always get all the parents of an element if you use that, since if a child contains text then necessarily all their parents contain the same text). >> There's some other tricky ones I'm not sure about either, though they >> seem to be kind of working. Things like div:only-child (when it's a div > > //*[name() = 'div' and last() = 1] This doesn't seem to be working for me: >>> xpath('span:only-child') *[name() = 'span' and (last() = 1)] But testing with <div><span></span></div> in the document, I don't get anything returned. These all work now... >> with no siblings), > >> div:last-child (no next sibling) > > //*[name() = 'div' and position() = last()] > >> div:first-child (no previous sibling) > > Didn't we just coer that one? >> div:first-of-type (no preceding siblings that are divs > > //div[position() = 1] > >> div:last-of-type (no following siblings that are divs), > > //div[position() = last()] > >> div:only-of-type (you are probably getting the pattern) > > //div[last() = 1] > >> div:empty (no children, including text, maybe not including whitespace). > > Ouch. let me think about that one. Yeah, I couldn't figure that one out. I thought this might work: >>> xpath('E:empty') e[count(./children::*) = 0 and string(.) = ''] But maybe I don't understand how count() works; this isn't a valid XPath expression. >> There's also >> div:nth-child(matcher) and div:nth-of-type(matcher), which selects among > > Those should be easy with the above examples. > >> siblings with patterns like "2" (second sibling), "3n" (every third >> element), "odd" (odd elements) and some other selections. I kind of see >> how to deal with this using position(), but I'm not sure how to do >> either nth-of-type or nth-child (and the ones I do understand I am also >> vague about). -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

1 0

[lxml-dev] lxml.html.parse() should behave like lxml.etree.parse()
by Stefan Behnel June 29, 2007

June 29, 2007

Hi, I noticed that lxml.html.parse() currently takes a string as argument that is interpreted as HTML code. However, lxml.etree.parse() also takes a string, which is interpreted as a filename. I think we should not divert from the other lxml APIs here. At least, it surprised me when I called etree.tostring(html.parse("doc/html/api.html")) and got "<p>doc/html/api.html</p>" as a result. I really want lxml to stay an integrated set of tools, things that work together smoothly. And a commen base API is very important here. I don't mind Elements having different APIs in different packages (that's the main idea after all), but I would like to keep functions and methods with similar names semantically close wherever possible. It's hard to come up with a good name for the functions, though, as the function that comes closest is called HTML(). Not really a perfect name for a function (but ok for a factory). What about "parse_chunk()" or "parse_string()"? Then the other functions would become something like "parse_string_element()" and "parse_string_elements()". I know, that's long, but I wouldn't mind that, since the meaning is clear and most of the time, you'd use "parse_string()" anyway. We could then add a "parse()" function that basically does what the current "parse()" function does, but for files as input. Would that be ok for you? Stefan

3 2

[lxml-dev] Building with distutils only
by Itamar Shtull-Trauring June 29, 2007

June 29, 2007

Though it is possible to build lxml 1.2 with distutils only, no need to install anything extra, the try/except that used to be in 1.1 for this no longer exists. This is very frustrating since I don't want to use ezsetup or whatever it's called when building lxml; especially when at one point it tried to download egg from non-existent URL. At work we have the following patch; it could be easily replaced with a try/except that imports from distutils when setuptools is not installed: --- lxml-1.2.1/setup.py 2007-02-20 08:43:16.000000000 -0500 +++ lxml-1.2.1.new/setup.py 2007-04-06 15:02:13.000000000 -0400 @@ -1,7 +1,4 @@ -from ez_setup import use_setuptools -use_setuptools(version="0.5") - -from setuptools import setup +from distutils.core import setup import sys, os # need to insert this to python path so we're sure we can import --- lxml-1.2.1/setupinfo.py 2007-02-27 03:23:38.000000000 -0500 +++ lxml-1.2.1.new/setupinfo.py 2007-04-06 15:01:04.000000000 -0400 @@ -1,5 +1,5 @@ import sys, os -from setuptools.extension import Extension +from distutils.extension import Extension try: from Pyrex.Distutils import build_ext as build_pyx

2 3

[lxml-dev] Encoding problems with lxml
by Bruno Barberi Gnecco June 29, 2007

June 29, 2007

Hello, I'm having some encoding problems with lxml that I can't solve. My application is a small web mining spider. Pages downloaded can be in any encoding, but I'm expecting mostly utf8 and iso-8859-1. I need to get the parsed data in iso-8859-1. I'm having two problems: a) when reading pages in iso-8859-1, accented characters are converted to HTML sequences, such as à for ` + a. I don't want this to happen, how to avoid it? b) I can't convert pages originally in UTF to ISO, even using etree.tostring(entry, 'iso-8859-1') or string.encode("iso-8859-1"). Have I missed something in the docs? I want to have a homogeneous behavior for all encodings--even if it means to convert first to UTF and later to ISO. Thanks a lot for any help, Bruno

3 5

[lxml-dev] [PATCH] 1.3 Fatal Python error: deallocating None
by David M. Grimes June 27, 2007

June 27, 2007

All - I have identified a bug with the new attrib.pop() method in lxml-1.3 There is a piece of code in etree.pyx (line 1483) related to using the provided default if the key lookup fails. The issue is that the returned value (from PyTuple_GET_ITEM) is a borrowed reference. I was using it as such: v = elt.attrib.get(k, None) And eventually (after several iterations of my parsing loop while processing various documents) I would get this error: Fatal Python error: deallocating None I was able to create a trivial test case to reproduce: import lxml.etree as etree xml = '''\ <?xml version="1.0"?> <xml/>''' for i in range(10000): et = etree.fromstring(xml) et.attrib.pop('x', None) And the patched code replaces the pop() method in etree.pyx with: def pop(self, key, *default): if python.PyTuple_GET_SIZE(default) > 1: raise TypeError, "pop expected at most 2 arguments, got %d" % \ (python.PyTuple_GET_SIZE(default)+1) result = _getAttributeValue(self._element, key, None) if result is None: if python.PyTuple_GET_SIZE(default) == 0: raise KeyError, key else: result = python.PyTuple_GET_ITEM(default, 0) python.Py_INCREF(result) else: _delAttribute(self._element, key) return result This solves the issue for me, and should not introduce any memory leaks, as the returned object in the defaulted path should be INCREF'd. Thanks, --Dave

2 1

[lxml-dev] merge annotate() and xsiannotate() into single function
by Stefan Behnel June 27, 2007

June 27, 2007

Hi Holger, in the current trunk, objectify has those two big functions "annotate()" and "xsiannotate()" that do very similar things in similar ways. When I was changing the annotation behaviour for empty elements, I noticed that it would be hard to keep the two in sync, so I decided to merge them. Here is a patch that matches the current test cases on the trunk (note that I did not yet merge your tests for 1.3 back). Holger, could you look through it to check if this is what you would expect? I know that it does not guarantee exactly the same behaviour as before, so there will likely be cases where some annotations come out different than now. Actually, xsiannotate() was not yet in a release, but I even think that objectify shouldn't guarantee any particular annotations anyway. I rather prefer something like: "it always yields types that make sense for the current data". Another point here: the current XSI type order prefers "integer" over "long" and "float" over "double". Shouldn't those be swapped to always annotate with the widest possible type? Stefan

2 14

[lxml-dev] lxml 1.3 is on cheeseshop
by Stefan Behnel June 25, 2007

June 25, 2007

Hi all, I'm proud to announce the release of lxml 1.3. This is a major new release with various new features and lots of fixes compared to the 1.2 series. The complete changelog follows below. Major objectives of this release were: - API consolidation: make everything work with everything - improved namespace handling: avoid redundant namespaces wherever possible - simplicity and accessibility: improved, restructured documentation and simpler XML/HTML generation Future versions of lxml will continue this trend to make lxml the leading tool for XML and HTML in the Python world. Have fun, Stefan Behnel 1.3 (2007-06-24) ================ Features added -------------- * Module ``lxml.pyclasslookup`` implemens an Element class lookup scheme that can access the entire tree to determine a suitable Element class * Parsers take a ``remove_comments`` keyword argument that skips over comments * ``parse()`` function in ``objectify``, corresponding to ``XML()`` etc. * ``Element.addnext(el)`` and ``Element.addprevious(el)`` methods to support adding processing instructions and comments around the root node * Extended type annotation in objectify: cleaner annotation namespace setup plus new ``deannotate()`` function * Support for custom Element class instantiation in lxml.sax: passing a ``makeelement()`` function to the ElementTreeContentHandler will reuse the lookup context of that function * '.' represents empty ObjectPath (identity) Bugs fixed ---------- * Removing Elements from a tree could make them loose their namespace declarations * ``ElementInclude`` didn't honour base URL of original document * Replacing the children slice of an Element would cut off the tails of the original children * ``Element.getiterator(tag)`` did not accept ``Comment`` and ``ProcessingInstruction`` as tags * API functions now check incoming strings for XML conformity. Zero bytes or low ASCII characters are no longer accepted. * XSLT parsing failed to pass resolver context on to imported documents * More ET compatible behaviour when writing out XML declarations or not * ``Element.attrib`` was missing ``clear()`` and ``pop()`` methods * More robust error handling in ``iterparse()`` * Documents lost their top-level PIs and comments on serialisation * lxml.sax failed on comments and PIs. Comments are now properly ignored and PIs are copied. * Raise AssertionError when passing strings containing '\0' bytes 1.3beta (2007-02-27) ==================== Features added -------------- * ``DTD`` validator class (like ``RelaxNG`` and ``XMLSchema``) * HTML generator helpers by Fredrik Lundh in ``lxml.htmlbuilder`` * ``ElementMaker`` XML generator by Fredrik Lundh in ``lxml.builder.E`` * Support for pickeling ``objectify.ObjectifiedElement`` objects to XML * ``update()`` method on Element.attrib * Optimised replacement for libxml2's _xmlReconsiliateNs(). This allows lxml a better handling of namespaces when moving elements between documents. Bugs fixed ---------- * Possible memory leaks in namespace handling when moving elements between documents Other changes ------------- * major restructuring in the documentation

3 8

[lxml-dev] Final check for lxml 1.3 - anyone?
by Stefan Behnel June 22, 2007

June 22, 2007

Hi all, since there were a few more additions since the last time I asked, could a few people please test the current 1.3 branch on their machines? Preferably Windows, Mac-OS (with current libraries), etc. Please run the tests, maybe also run your own code against it. That would make it less likely that I will have to release a 1.3.1 right next week. :) If all works fine, I'll release 1.3 either this week-end or early next week. Have fun, Stefan @Holger: sorry, but xsiannotate() won't make it. I wouldn't mind adding the functionality to "annotate()" later on in the 1.3 series, though.

1 0

Re: [lxml-dev] documentation wiki
by Stefan Behnel June 22, 2007

June 22, 2007

Hi Roger, Roger Patterson wrote: > I don't know what the ML's etiquette is for tangential > discussions, so I didn't include the mailing list on this email. this kind of discussion definitely belongs on the ML. We only have one, so we use it to discuss basically everything from infrastructure to development to helping users. :) It's rather low traffic, so it's usually not a big problem if there is one mail more coming through. > Thanks for that link to the mailing-list search tool. Very handy. > Also - this led me to actually find http://codespeak.net/lxml/dev for > documentation, which seems far more detailed than what I was looking at > before: http://codespeak.net/lxml/api.html It's the new web page that will replace the current one with the release of lxml 1.3 (hopefully soon). > I'm not an uber-expert with wikis. I've just done very basic > administration for my own company's documentation, using dokuwiki > (http://wiki.splitbrain.org/wiki:dokuwiki) - which uses PHP to serve > file-system-based wiki files. Not speaking for anyone else here, but I guess any PHP based solution is rather out - unless we find nothing else :) > I did see some discussion, in searching the ML, on moving lxml to Trac a > while ago - is this still under consideration? codespeak has its own tracker by now. Not wiki based, just a simple tracker. So I guess the decision on the codespeak side was rather against Trac. > I don't have oodles of time, but I'd be willing to help administer a > wiki to help out. > I might be able to put a wiki on my company's website if codespeak isn't > up for it. Thanks for the offer. But let's wait for a word by the codespeak people first. Stefan

1 0

[lxml-dev] documentation wiki
by Roger Patterson June 22, 2007

June 22, 2007

Hello all, I moved to lxml/elementtree from 4Suite and am loving it. However, it would be good to improve the documentation for us muddling our way through it - but of course the issue of who-will-do-it/who-has-the-time always comes up. To solve this - would it be possible to start a documentation wiki on the codespeak site (I use dokuwiki - which is excellent, and I hear moinmoin is good and python-powered)? This would disperse the documentation effort among those who use it (maybe only allow members of the mailing list, to prevent spam). I would be more than happy to add documentation as I figure things out. Not sure if this has already been discussed on the mailing list, as I don't seem to be able to perform searches on the mailing list archives. Cheers, -Roger Patterson

2 1

[lxml-dev] lxml 1.3 annotate() behaviour for empty string data elements
by jholg＠gmx.de June 21, 2007

June 21, 2007

Hi, I just noticed that annotate() does not add type information to empty string elements when parsed: >>> root = etree.fromstring(""" ... <root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ... xmlns:py="http://codespeak.net/lxml/objectify/pytype" ... xmlns:xsd="http://www.w3.org/2001/XMLSchema"> ... <s1>foobar</s1> ... <s2></s2> ... </root> ... """) >>> objectify.annotate(root) >>> >>> print etree.tostring(root, pretty_print=True)oot) <root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:py="http://codespeak.net/lxml/objectify/pytype" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <s1 py:pytype="str">foobar</s1> <s2/> </root> >>> Whereas type annotation happens when setting attributes manually: >>> root = objectify.Element("root") >>> root.s1 = "foobar" >>> root.s2 = "" >>> objectify.annotate(root) >>> print etree.tostring(root, pretty_print=True) <root xmlns:py="http://codespeak.net/lxml/objectify/pytype" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <s1 py:pytype="str">foobar</s1> <s2 py:pytype="str"></s2> </root> >>> I know this happens due to the .text of the node being None in the 1st case instead of '' in the second case (which is lxml/ElementTree/libxml2 behaviour that bites me once and again). Still, I'd prefer to have annotate() provide all data elements with type information; after all, the element in question is treated as a StringElement (the default emtpy_data_class) anyway. Objections? Holger -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail

2 1

Re: [lxml-dev] ElementInclude doesn't honour base URLs
by Stefan Behnel June 21, 2007

June 21, 2007

Mike Meyer wrote: > In <4679A865.3010706(a)behnel.de>, Stefan Behnel <stefan_ml(a)behnel.de> typed: >> Mike Meyer wrote: >>> The problem is that using ElementInclude.include doesn't resolve href >>> attributes relative to their containing document. >> Fixed on the trunk and for 1.3. > > Cool. Thank you. > > For the archives, my workaround for 1.2.x is: > > from os.path import join, split > from lxml.etree import parse > from lxml.ElementInclude import include, default_loader > > class Loader(object): > """Custom loader for XInclude processing to fix relative addressing for files.""" > > def __init__(self, name): > self.base = split(name)[0] > > def __call__(self, href, typ, encoding=None): > return default_loader(join(self.base, href), typ, encoding) This only works for local files. You can use "urlparse.urljoin" to make it work for more or less any file path or URL (remember that lxml supports ftp and http). You don't even need to do the split before hand, BTW. Ah, and I don't think this works for included files including files themselves from different directories. But it's still a good work around for most use cases. Stefan

1 0

Re: [lxml-dev] ElementInclude doesn't honour base URLs
by Stefan Behnel June 20, 2007

June 20, 2007

Hi, Mike Meyer wrote: > The problem is that using ElementInclude.include doesn't resolve href > attributes relative to their containing document. Fixed on the trunk and for 1.3. Stefan

1 0

Re: [lxml-dev] tests crash on libxml2 2.6.19 - drop support?
by Stefan Behnel June 20, 2007

June 20, 2007

Hi, Mike Meyer wrote: > In <46793138.4040701(a)weitershausen.de>, Philipp von Weitershausen <philipp(a)weitershausen.de> typed: >> I hear that others have lxml working on MacOS quite nicely. Not sure >> whether this is with MacPorts or self-compiled libxml/libxslt (last time >> I tried the latter I had no luck either). Martin Aspeli and Christian >> Zagrodnick were the last ones I heard report success in some way or >> another... > > I've had no trouble with lxml built from Macpors on OSX. Ok, so that sounds like people rather install new libs, which makes it rather unlikely that they end up trying to use something older than 2.6.20 anyway. So, if we drop the support, we explicitly tell people to install something newer, which is in their own interest. > Well, except > for the previously reported and otherwise uncommented on problem with > XInclude, but that's reproducible under FreeBSD, and with > cElementTree. An XInclude problem that's reproduceable with cElementTree (which has no real XInclude support) would most likely have been commented on. Any more hints on what you are talking about here? Maybe ElementInclude? Are there any issues with it? Stefan

1 0

[lxml-dev] tests crash on libxml2 2.6.19 - drop support?
by Stefan Behnel June 20, 2007

June 20, 2007

Hi all, when going through my usual little pre-release QA test scripts, I noticed that the tests on the 1.3 branch currently trigger a reproduceable crash under libxml2 2.6.19. Don't know about older ones as they don't even compile on my machine. Sadly, valgrind doesn't tell me enough about where the problems come from, so it's not a quick thing to fix. Now, there are two ways to tackle this: work around the problem or drop support for 2.6.19 and earlier. Given the fact that the release of 2.6.20 just had its second anniversary (and knowing that libxml2 still had a lot of bugs fixed since), I do not feel much in favour of the first solution. I would rather encourage people to upgrade their libs if they want to use lxml 1.3. This is not a problem for most Linux systems where installing new libs tends to be trivial (and is often done on a regular basis to fix security holes). Neither is it a problem for Windows as long as Sidnei keeps providing us with static builds against recent libs. But I remember that MacOS X usually ships with pretty old versions, and I think I recall one of them being 2.6.16 and the other being 2.6.19. So taking this decision would mean dropping out-of-the-box support for those MacOS versions. Can anyone please verify that these were the versions in question? What are the lib versions that current MacOS systems ship with? And which OS versions would we drop support for if lxml 1.3 stopped running under 2.6.19? Hoping for some insights, Stefan

3 4

[lxml-dev] XSLT crashing Zope
by Eric Tiffany June 19, 2007

June 19, 2007

I am having a really difficult time getting lxml etree.XSLT to work inside Zope. It will start up, but after a couple of requests (or sometimes immediately) python crashes with a stack trace that looks like this: Thread 2 Crashed: 0 libxml2.2.dylib 0x04504370 xmlDictFree + 265 1 libxml2.2.dylib 0x045042e2 xmlDictFree + 123 2 etree.so 0x04206d2a __pyx_f_5etree_14_ParserContext_initThreadDictRef + 63 (etree.c:20751) 3 etree.so 0x0426f761 __pyx_f_5etree_4XSLT___call__ + 2597 (etree.c:33610) 4 org.python.python 0x0020d87f PyObject_Call + 45 (abstract.c:1795) 5 org.python.python 0x0027e397 PyEval_EvalFrame + 16838 (ceval.c:3776) 6 org.python.python 0x0027ebaa PyEval_EvalFrame + 18905 (ceval.c:3651 And on for another 170 frames. This is running zope 2.9.6, python 2.4.4, on MacOS 10.4.9 (Intel) with this environment lxml.etree: (1, 3, 0, 44370) libxml used: (2, 6, 28) libxml compiled: (2, 6, 28) libxslt used: (1, 1, 20) libxslt compiled: (1, 1, 20) I have built lxml using --without-threading, and it seems to make no difference. If I test from the python shell, my example works fine. IF I RUN ZOPE INSIDE ECLIPSE, IT WORKS FINE. Same code, same setup. If I start zope using it's zopectrl or runzope scripts, it croaks. Here is the code: print >> sys.stderr, abs_execution_path("header.xsl") xsl_tree = etree.parse(abs_execution_path("header.xsl")) print >> sys.stderr, "creating transformer" transformer = etree.XSLT(xsl_tree) print >> sys.stderr, "parsing header" header = etree.parse(StringIO(str(context.response))) print >> sys.stderr, "transforming header" result = str(transformer(header)) print >> sys.stderr, "result done" And here is what I see in the log /Applications/Plone-2.5.2/Instance/Products/xtend/header.xsl creating transformer parsing header transforming header Segmentation fault This is costing me a lot of $$ and lost goodwill from my customer. I wish I had never heard of python or plone. ET

2 1

Re: [lxml-dev] lxml 1.3 coming up
by Stefan Behnel June 19, 2007

June 19, 2007

Stephan Richter wrote: > Did sourceline make it? :-) Right now z3c.rml depends on lxml trunk, since the > lxml 1.3 beta release does not contain the sourceline feature. It would be > nice to depend on a release. Well, you can check: http://codespeak.net/svn/lxml/branch/lxml-1.3/src/lxml/etree.pyx (hint: yes it did :) Have fun, Stefan

1 0

[lxml-dev] Website won't render on Camino
by Eric Tiffany June 18, 2007

June 18, 2007

The http://codespeak.net/lxml/ site doesn't render on Camino Version 2007050909 (1.5). The current contents of the browser just remains (as if you had not tried to follow a link or didn't put in a new URL). It's very weird. ET

2 3