lxml - The Python XML Toolkit June 2007

lxml@python.org

20 participants
42 discussions

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

[lxml-dev] Pickling objectified trees
by Christian Zagrodnick 09 Jun '08

09 Jun '08

Hi, the other day I had to pickle objectified trees. I just thought to share my findings. Pickling is about serialization. IMHO the natural serialization of an objectified tree is its XML representation. So the following basically does that: -------------------------- import copy_reg import lxml.etree import lxml.objectify def treeFactory(state): """Un-Pickle factory.""" return lxml.objectify.fromstring(state) copy_reg.constructor(treeFactory) def reduceObjectifiedElement(object): """Reduce function for lxml.objectify trees. See http://docs.python.org/lib/pickle-protocol.html for details. """ return (treeFactory, (lxml.etree.tostring(object), )) copy_reg.pickle(lxml.objectify.ObjectifiedElement, reduceObjectifiedElement, treeFactory) ----------------------------------------- You might consider just registering the reduce function in lxml itself. Shouldn't hurt, should it. -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891

2 4

[lxml-dev] lxml namespaces
by Maxim Sloyko 20 Feb '08

20 Feb '08

Hi All! I have a little problem with XML namespaces. In my application I have two XML processors, that process the same document, one after the other. The first one looks for nodes in 'ns1' namespace, and substitutes them, according to some algorithm. After this processor is finished, it is guaranteed that there are no more 'ns1' nodes left in the tree. 'ns1' namespace dclaration is still there, in the root node (well, I put it there manually). Now, when this namespace is no longer needed, I want to get rid of it, because it confuses some other processors (namely, my browser) So, the question is, how do I do that? del tree.getroot().nsmap['ns1'] does not seem to do the trick :( Thanks in advance and Happy Holidays! -- Maxim Sloyko

3 5

Re: [lxml-dev] Some XPath questions...
by Ian Bicking 03 Jul '07

03 Jul '07

Mike Meyer wrote: > In <468579E3.7010802(a)colorstudy.com>, Ian Bicking <ianb(a)colorstudy.com> typed: >> Thanks, very helpful. I'm guessing it was an oversight that you didn't >> copy the list... > > I wasn't sure which way to go. Without CC'ing people won't know you've already answered my questions. >>>> There's some other tricky ones I'm not sure about either, though they >>>> seem to be kind of working. Things like div:only-child (when it's a div >>> //*[name() = 'div' and last() = 1] >> This doesn't seem to be working for me: >> >>> xpath('span:only-child') >> *[name() = 'span' and (last() = 1)] >> But testing with <div><span></span></div> in the document, I don't get >> anything returned. > > Seems to work ok for me: > >>>> d = fromstring('<html><div><span></span></div><span /></html>') >>>> d.xpath('//span') > [<Element span at 918500>, <Element span at 9182d0>] >>>> d.xpath('//*[name() = "span" and last() = 1]') > [<Element span at 918500>] > > But I may be missing something. Got a full test case? Here's a small example: >>> from lxml.etree import HTML >>> h = '<html><body><div></div><div><span></span></div></body></html>' >>> h = HTML(h) >>> h.xpath("descendant-or-self::*[name() = 'span' and (last() = 1)]") [] >>> h.xpath("//*[name() = 'span' and (last() = 1)]") [<Element span at -4866f944>] So when I use // it works. Huh. I prefer descendant-or-self, because I find it peculiar to do a search from the root when you've called the method on some particular element (that may not be at the root). >>>> div:empty (no children, including text, maybe not including whitespace). >>> Ouch. let me think about that one. >> Yeah, I couldn't figure that one out. I thought this might work: >> >>> xpath('E:empty') >> e[count(./children::*) = 0 and string(.) = ''] >> But maybe I don't understand how count() works; this isn't a valid XPath >> expression. > > You want "child" not "children". Using normalize-space(.) instead of > string(.) will exclude whitespace. This does assume you are ignoring > comments and PIs; I believe that's the behavior you want. Cool, that seems to work right. One query I'm realizing might be really hard (maybe too hard in XPath) is *:first-of-type, *:last-of-type, and *:only-of-type, since they match in a funny sort of way. You can't really do: *[count(../*[name() = name()) = 1] But it's kind of what *:only-of-type means. Or: *[count(following-sibling::name()) = 0 and count(previous-sibling::name()) = 0] You just can't use name() that way. Hmm... well, it's not that important of a query to me, I guess, so maybe I'll just catch it and give an error. -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

2 2

[lxml-dev] Some XPath questions...
by Ian Bicking 02 Jul '07

02 Jul '07

I'm trying to implement CSS selectors, by translating them into XPath. There's some CSS expressions that I'm having a hard time with, so maybe someone can tell me how they might work. Expression: div:first-child -- means a div element when it is the first child of its parent. I.e.: <li> <div id="a">...</div> <div id="b">...</div> </li> It makes the first div and not the second. I thought this could be: descendant-or-self::*/div[0] or... descendant-or-self::*/div[position() = 0] Those two should be equivalent; the second is a bit easier to handle programmatically. But it doesn't work (doesn't match anything). Another expreesion: div.foo + div -- means a div element that is the immediately next sibling of a div element with the class .foo. I would translate this to: descendant-or-self::div[@class='foo']/following-sibling::div[0] (The class matching is actually a bit more complex, but it doesn't actually matter to this.) I'm (a) not sure if this is right, because maybe it means the next div after the matching div, even if there's another element in-between, and (b) it doesn't return any results regardless. Another expression: div:contains('celia') -- means a div where the textual content has the word 'celia' in it, case insensitive. At least, I think it's case insensitive -- the CSS spec is annoyingly vague, but implementations seem to work like this. I translate this to: descendant-or-self::div[contains(css:lower-case(string(.)), 'celia'] I added the lower-case function like: def _make_lower_case(context, s): return s.lower() etree.FunctionNamespace("css")['lower-case'] = _make_lower_case But XPath gives so few errors that it's hard to tell if it's really working. The XPath expression returns some elements, but not the correct number from what I can tell. Especially since when I had a bug and wasn't lowercasing the second argument (using 'CELIA') it still returned elements. There's some other tricky ones I'm not sure about either, though they seem to be kind of working. Things like div:only-child (when it's a div with no siblings), div:last-child (no next sibling), div:first-child (no previous sibling), div:first-of-type (no preceding siblings that are divs), div:last-of-type (no following siblings that are divs), div:only-of-type (you are probably getting the pattern), div:empty (no children, including text, maybe not including whitespace). There's also div:nth-child(matcher) and div:nth-of-type(matcher), which selects among siblings with patterns like "2" (second sibling), "3n" (every third element), "odd" (odd elements) and some other selections. I kind of see how to deal with this using position(), but I'm not sure how to do either nth-of-type or nth-child (and the ones I do understand I am also vague about). I've committed the incomplete code in lxml.html.css -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

2 2

[lxml-dev] lxml 1.3 coming up
by Stefan Behnel 01 Jul '07

01 Jul '07

Hi all, I finally found some time to work on lxml 1.3 again. Since the current trunk will become 2.0 (though not that soon :), I created a new branch "lxml-1.3" that will become the 1.3 release. I worked through the trunk commits to see which ones to copy over, found quite a few and rejected some bigger ones. I'd be glad if others could give this branch a final try before the release, see if it compiles (e.g. on MS Windows), if it contains the long awaited bug fixes, etc. I'm planning the release for the end of the week. http://codespeak.net/svn/lxml/branch/lxml-1.3/ Note that I tried to avoid behavioural changes, so some 'fixes' might have been left out for that reason. If unsure, it might be worth asking. Have fun, Stefan

5 29

[lxml-dev] Another (last?) take on proxy deallocation
by Stefan Behnel 30 Jun '07

30 Jun '07

Hi all, while Ian Bicking was working on the "lxml.html" trunk, I noticed that the way some of the modules were implemented could crash lxml.etree during garbage collection. I know, Martijn and I have already reimplemented the proxy code a couple of times, each time solving more and more of the encountered problems, but I really hope this is the last time we have to reimplement this. I already disliked the last way I had to rewrite it, as it required an additional document traversal step each time a document is deallocated. While we accept this behaviour for disconnected tree fragments (a trade-off between different overheads), it should not be necessary for the whole document (at least not more often than required by xmlFreeDoc()). But the problem is that Python's cyclic garbage collector gives no guarantees about the order in which the collected objects are cleaned up - and libxml2 requires access to the document node when cleaning up a tree fragment. So it is actually required that we clean up all _Element proxies first and *then* free the xmlDoc. So, _Document must really always be deallocated *after* all its _Element proxies have been garbage collected. I was thinking about a way to do this for a while and experimented with it on a "proxy-deallocation" branch - until I realised that the best way to control the garbage collector was the garbage collection mechanism itself - i.e. reference counting. So, I checked in a small patch (SVN revision 44623 on the trunk) that simply doubles the ref-counts that an _Element holds to its _Document so that we can control when the ref-count to the document is decreased. lxml.etree now does this explicitly in the tp_dealloc function of the _Element class, *after* cleaning up the proxy, so that the ref-count of the _Document never goes down to 0 before the last of its _Element proxies was deallocated. It is then safe to run xmlFreeDoc() on the libxml2 document from _Document.__dealloc__. https://codespeak.net/viewvc/?view=rev&revision=44623 I really like this approach and I also like that it removes the need for document traversal on _Document deallocation. And: it keeps the code on the lxml.html branch from crashing, which is a *really* good sign. :] So, I will also merge this into the 1.3 branch and release a 1.3.1 soon. We already had a couple of small fixes on the branch, so a bug-fix release next week should nicely improve the code quality of the official release. Have fun, Stefan

1 0

Re: [lxml-dev] Some XPath questions...
by Ian Bicking 29 Jun '07

29 Jun '07

Thanks, very helpful. I'm guessing it was an oversight that you didn't copy the list... Mike Meyer wrote: > In <468538ED.9060004(a)colorstudy.com>, Ian Bicking <ianb(a)colorstudy.com> typed: >> I'm trying to implement CSS selectors, by translating them into XPath. >> There's some CSS expressions that I'm having a hard time with, so maybe >> someone can tell me how they might work. >> >> Expression: >> >> div:first-child -- means a div element when it is the first child of its >> parent. I.e.: >> >> <li> >> <div id="a">...</div> >> <div id="b">...</div> >> </li> >> >> It makes the first div and not the second. >> >> I thought this could be: >> >> descendant-or-self::*/div[0] >> or... descendant-or-self::*/div[position() = 0] > >> Those two should be equivalent; the second is a bit easier to handle >> programmatically. But it doesn't work (doesn't match anything). > > XPath arrays are 1-indexed, not 0-indexed, so position() will never be > 0. I understand some version of IE get this wrong as well. > > To pick out all the div elements that are the first child of their > parent, use: > > //*[position() = 1 and name() = 'div'] > > or equivalently: > > descendant-or-self::*[position() = 1 and name() = 'div'] I don't know how I missed the fact they are 1-indexed... I guess it's become such an unusual choice these days. But handy anyway, since CSS is also 1-indexed. >> Another expreesion: >> >> div.foo + div -- means a div element that is the immediately next >> sibling of a div element with the class .foo. I would translate this to: >> >> descendant-or-self::div[@class='foo']/following-sibling::div[0] >> >> (The class matching is actually a bit more complex, but it doesn't >> actually matter to this.) I'm (a) not sure if this is right, because >> maybe it means the next div after the matching div, even if there's >> another element in-between, and (b) it doesn't return any results >> regardless. > > > Your maybe is right - it means the first div after, whether or not > there are following siblings. You then select the first element from > that list (or would, if you used a 1 instead of a 0). Same solution: > the last bit is following-sibling::*[position() = 1 and name() = 'div'] > >> Another expression: >> >> div:contains('celia') -- means a div where the textual content has the >> word 'celia' in it, case insensitive. At least, I think it's case >> insensitive -- the CSS spec is annoyingly vague, but implementations >> seem to work like this. I translate this to: >> >> descendant-or-self::div[contains(css:lower-case(string(.)), 'celia'] >> >> I added the lower-case function like: >> >> def _make_lower_case(context, s): >> return s.lower() >> etree.FunctionNamespace("css")['lower-case'] = _make_lower_case >> >> But XPath gives so few errors that it's hard to tell if it's really >> working. The XPath expression returns some elements, but not the >> correct number from what I can tell. Especially since when I had a bug >> and wasn't lowercasing the second argument (using 'CELIA') it still >> returned elements. > > I think you've got the parens in the wrong place - the last close goes > after 'celia', not the comma. That was just a typo in the email; copying and pasting directly: >>> xpath('E:contains("foo")') e[contains(css:lower-case(string(.)), 'foo')] However, now that I'm writing my own tests it seems fine (I was using someone else's tests, and I think they were wrong; though I'm not sure -- you'll always get all the parents of an element if you use that, since if a child contains text then necessarily all their parents contain the same text). >> There's some other tricky ones I'm not sure about either, though they >> seem to be kind of working. Things like div:only-child (when it's a div > > //*[name() = 'div' and last() = 1] This doesn't seem to be working for me: >>> xpath('span:only-child') *[name() = 'span' and (last() = 1)] But testing with <div><span></span></div> in the document, I don't get anything returned. These all work now... >> with no siblings), > >> div:last-child (no next sibling) > > //*[name() = 'div' and position() = last()] > >> div:first-child (no previous sibling) > > Didn't we just coer that one? >> div:first-of-type (no preceding siblings that are divs > > //div[position() = 1] > >> div:last-of-type (no following siblings that are divs), > > //div[position() = last()] > >> div:only-of-type (you are probably getting the pattern) > > //div[last() = 1] > >> div:empty (no children, including text, maybe not including whitespace). > > Ouch. let me think about that one. Yeah, I couldn't figure that one out. I thought this might work: >>> xpath('E:empty') e[count(./children::*) = 0 and string(.) = ''] But maybe I don't understand how count() works; this isn't a valid XPath expression. >> There's also >> div:nth-child(matcher) and div:nth-of-type(matcher), which selects among > > Those should be easy with the above examples. > >> siblings with patterns like "2" (second sibling), "3n" (every third >> element), "odd" (odd elements) and some other selections. I kind of see >> how to deal with this using position(), but I'm not sure how to do >> either nth-of-type or nth-child (and the ones I do understand I am also >> vague about). -- Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers

1 0

[lxml-dev] lxml.html.parse() should behave like lxml.etree.parse()
by Stefan Behnel 29 Jun '07

29 Jun '07

Hi, I noticed that lxml.html.parse() currently takes a string as argument that is interpreted as HTML code. However, lxml.etree.parse() also takes a string, which is interpreted as a filename. I think we should not divert from the other lxml APIs here. At least, it surprised me when I called etree.tostring(html.parse("doc/html/api.html")) and got "<p>doc/html/api.html</p>" as a result. I really want lxml to stay an integrated set of tools, things that work together smoothly. And a commen base API is very important here. I don't mind Elements having different APIs in different packages (that's the main idea after all), but I would like to keep functions and methods with similar names semantically close wherever possible. It's hard to come up with a good name for the functions, though, as the function that comes closest is called HTML(). Not really a perfect name for a function (but ok for a factory). What about "parse_chunk()" or "parse_string()"? Then the other functions would become something like "parse_string_element()" and "parse_string_elements()". I know, that's long, but I wouldn't mind that, since the meaning is clear and most of the time, you'd use "parse_string()" anyway. We could then add a "parse()" function that basically does what the current "parse()" function does, but for files as input. Would that be ok for you? Stefan

3 2

[lxml-dev] Building with distutils only
by Itamar Shtull-Trauring 29 Jun '07

29 Jun '07

Though it is possible to build lxml 1.2 with distutils only, no need to install anything extra, the try/except that used to be in 1.1 for this no longer exists. This is very frustrating since I don't want to use ezsetup or whatever it's called when building lxml; especially when at one point it tried to download egg from non-existent URL. At work we have the following patch; it could be easily replaced with a try/except that imports from distutils when setuptools is not installed: --- lxml-1.2.1/setup.py 2007-02-20 08:43:16.000000000 -0500 +++ lxml-1.2.1.new/setup.py 2007-04-06 15:02:13.000000000 -0400 @@ -1,7 +1,4 @@ -from ez_setup import use_setuptools -use_setuptools(version="0.5") - -from setuptools import setup +from distutils.core import setup import sys, os # need to insert this to python path so we're sure we can import --- lxml-1.2.1/setupinfo.py 2007-02-27 03:23:38.000000000 -0500 +++ lxml-1.2.1.new/setupinfo.py 2007-04-06 15:01:04.000000000 -0400 @@ -1,5 +1,5 @@ import sys, os -from setuptools.extension import Extension +from distutils.extension import Extension try: from Pyrex.Distutils import build_ext as build_pyx

2 3

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

lxml - The Python XML Toolkit June 2007