Mailman 3 June 2006 - lxml - The Python XML Toolkit

[lxml-dev] ANN: XPathEvaluator 0.1.0
by Andreas Pakulat 21 Jun '06

21 Jun '06

Hi, as per Stefan's request I'm announcing here (and on the PyQt mailing list) the first release of a small tool I "just" (i.e. the last 3 days) wrote. XPathEvaluator is a tool to test what results a XPath expression gives you when executed on a specific XML file. With the help of lxml it can also parse pretty broken (and of course correct) HTML files. It loads it's data from URLs if you want and highlights all nodes that an XPath evaluation returns so you can easily identify them. I couldn't use lxml for more than HTML parsing because there's unfortunately no easy way to find out the element to which an attribute result or text result belongs to. I might actually use lxml for the initial XML parsing too, because it's way faster than the PyXML parser I currently have. I hope somebody finds this useful, I might add new features in the future, however there's no priority at the moment for the development of XPathEvaluator. As always with open source software: Patches welcome. XPathEvaluator can be downloaded from: http://www.apaku.de/linux/xpathevaluator/index.php That page also mentions all required software. Hope somebody will find this useful. Andreas -- You will obey or molten silver will be poured into your ears.

2 8

[lxml-dev] Bug in lxml with StringIO?
by Andreas Pakulat 21 Jun '06

21 Jun '06

Hi, I'm not sure wether this is a bug or not, but somethings surely wrong here: >>> data=file('/home/andreas/temp/view.php') >>> html=data.read() >>> data.close() >>> uhtml=html.decode('iso-8859-1') >>> print uhtml[uhtml.index('tannien')-10:uhtml.index('tannien')+10] l">Großbritannien:&n >>> print type(uhtml[uhtml.index('tannien')-10:uhtml.index('tannien')+10]) <type 'unicode'> >>> print repr(uhtml[uhtml.index('tannien')-10:uhtml.index('tannien')+10]) u'l">Gro\xdfbritannien:&n' >>> sio=StringIO(uhtml) >>> tree=etree.parse(sio, etree.HTMLParser()) >>> tree.xpath('//font[contains(text(),'tannien')]/text()') File "<stdin>", line 1 tree.xpath('//font[contains(text(),'tannien')]/text()') ^ SyntaxError: invalid syntax >>> tree.xpath('//font[contains(text(),"tannien")]/text()') [u'Gro\xc3\x9fbritannien:\xa0\xa0'] It seems as if the return value from the xpath function gets encoded into utf-8 or something like that. This doesn't happen when using a much simpler file: >>> data=file('/home/andreas/temp/test.html') >>> html=data.read() >>> data.close() >>> uhtml=html.decode('iso-8859-1') >>> print uhtml <html> <body> Text=Großbritannien </body> </html> >>> sio=StringIO(uhtml) >>> tree=etree.parse(sio, etree.HTMLParser()) >>> row=tree.xpath('//body/text()') >>> print row [u'\nText=Gro\xdfbritannien\n'] The problematic "file" is downloaded from: http://www.ofdb.de/view.php?page=film&fid=1130 Any ideas welcome. I could probably also just handle to raw string to etree.parse but I atm don't have the time to test this (maybe later tonight) Andreas -- You like to form new friendships and make new acquaintances.

6 26

[lxml-dev] we got history
by Martijn Faassen 21 Jun '06

21 Jun '06

Hi there, I just noticed codespeak.net now has 'viewvc' available, which means we now have a better tool to look at past revisions and so on of lxml: https://codespeak.net/viewvc/lxml/ Regards, Martijn

1 0

[lxml-dev] missing functionality
by Nico Poppelier 21 Jun '06

21 Jun '06

This is my first post to lxml-dev, so apologies if this has been discussed before. Today, in an email discussion with Martijn Faassen, it became apparent that lxml.etree is missing functionality in the area of namespaces. The W3C specification for XML Namespaces allows names of the form prefix:local to be used for elements and attributes. The present version of lxml.etree only supports prefixes on elements. As attachment I'm sending a small Python script and sample file which will demonstrate the problem. Testing configuration: Ubuntu Linux 6.06, Python 2.4, libxml 2.6.24, libxslt 1.1.15, lxml 1.0.1 (the latter not from a .deb, but compiled myself). My suggestion: extend the API to preserve namespace prefixes on attributes. An attribute list should be a list of 3-tuples (prefix, name, value) with prefix=None if there's no prefix, instead of a list of 2-tuples (name, value). Regards, Nico Poppelier (ex-member of W3C Math WG, converted Perl hacker :-) #!/usr/bin/python # # Adapted from Listing 3 in article # http://www.xml.com/pub/a/2003/02/12/py-xml.html # by Uche Ogbuji import sys from lxml import etree parser = etree.XMLParser() root = etree.parse(sys.argv[1], parser) iter = root.getiterator() def trunc(text): return len(text)>40 and text[:40] + '...' or text for element in iter: print 'Element:', element.tag if element.keys(): print '\tPrefix:', element.prefix print '\tAttributes:' for name, value in element.items(): print "\t\t%s = '%s'" % (name, value) if element.text: content = trunc(element.text) print '\tText:', repr(content) print '\tChildren:' for child in element.getchildren(): print '\t\tElement:', child.tag if child.tail: content = trunc(child.tail) print '\t\tText:', repr(content)

4 16

[lxml-dev] Still getting crashes with lxml 1.0.1
by Piet van Oostrum 21 Jun '06

21 Jun '06

I am still getting crashes with lxml 1.0.1, compiled from the source tarball. Not as many as with 1.0, but it still occurs fairly often. Most of the time it is at a call to lxml.parse(), and not only when I reuse a variable. The problems that occur are Malloc errors, Bus errors and segmentation faults. -- Piet van Oostrum <piet(a)cs.uu.nl> URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4] Private email: piet(a)vanoostrum.org

3 4

[lxml-dev] parsing HTML from a StringIO
by Andreas Pakulat 21 Jun '06

21 Jun '06

Hi, I've got a small problem here parsing a html file from a StringIO object. I need to use a StringIO to cache the html (downloaded from a server) and then I'd like to feed it to lxml, however: >>> ustr=urlopen(url).read() >>> sio=StringIO(ustr) >>> t=etree.parse(sio,etree.HTMLParser()) Traceback (most recent call last): File "<stdin>", line 1, in ? File "etree.pyx", line 1588, in etree.parse File "parser.pxi", line 673, in etree._parseDocument File "parser.pxi", line 693, in etree._parseMemoryDocument File "apihelpers.pxi", line 332, in etree._utf8 AssertionError: All strings must be Unicode or ASCII I guess I'm doing something wrong, because: >>> data=urlopen(url) >>> f=file('/home/andreas/temp/urlfile.html','w') >>> f.write(data.read()) >>> f.close() >>> t=etree.parse('/home/andreas/temp/urlfile.html',etree.HTMLParser()) works as expected. I must be doing something wrong with the StringIO, but I don't know what? Do I really need to encode the html string into an unicode object? Andreas -- You will wish you hadn't.

4 6

[lxml-dev] New deallocation procedure
by Stefan Behnel 21 Jun '06

21 Jun '06

Hi all, I tried to resolve the bug that Petr found, but it seems to be a general problem with Python's garbage collector. Apparently, we cannot rely on the assumption that _Elements that hold a reference to _Documents are always freed *before* the _Document. This *may* be the case for the normal ref-count deallocator. However, whenever the cyclic GC is required to resolve the dependencies, it is free to deallocate objects in arbitrary order. The reason why this is a problem is that lxml previously freed C nodes when deallocating their proxy _Element (if it was not contained in a C document), and the entire C document when the _Document was deallocated. This works as long as _Documents are freed after all their proxy _Elements. It crashes when the GC decides to deallocate any of the _Elements after their _Document. It is not easy to solve this, since _Elements can't just free the entire C document at their own deallocation time. There may be other _Elements still sitting on it. Only the _Document can do that - after all proxies are disconnected. The way I resolved this is by adding a deep-traversal to the _Document deallocation code. It searches the entire tree for _Element proxies and clears their C node reference. This prevents these proxies from caring about their C representation when they are deallocated - no more access to already freed data, no more crashes. The down side is that this is costly. It doesn't matter too much for small documents, as traversal is very fast and the memory free() call is already costly enough. However, it does matter for large trees, especially if they do not fit into memory. If swapping is involved, we have to swap the tree twice now. It's sad we have to do this, but I really cannot see any way to force the Python memory management to always free _Document objects after all _Elements, regardless of the way and time they are freed. This is a very conservative approach, but it is the only way I can see that definitely prevents crashes. Stefan

2 3

[lxml-dev] [Half-OT] libxml2 creates new python objects?
by Andreas Pakulat 20 Jun '06

20 Jun '06

Hi, I know this is a bit OT, however the mailing list of libxml2 is rather quiet and so I'd like to ask here (if you think this is to OT, please just don't reply to my mail): Does anybody know wether libxml2 generates new python objects "on-the-fly"? I have a rather strange behaviour here where I get new objects any time I ask a node for it's children. At least thats the only way it can be. Let me explain a bit: I built a Qt4 model to represent a XML document, using the python bindings of libxml2 as implementation base. Now for lookup-speed I store all "already-seen" tree-items in a dictionary. But after opening the document root (html as root-element, 2 children) this dict already has 20 entries. Multiple times it contains a xml node for html. While "somehow" this still works for the model itself, it totally breaks the highlighting of nodes that match a given xpath expression. For that to work, I need to walk the already shown nodes (from the dictionary) and mark them for highlighting... I don't think there's another way to identify a node within a document, other than by the id I can obtain (or the reference to the object itself) - But if anybody has an idea for this, that would be great. If not, I'm probably filing a bugreport against libxml2's python bindings... Andreas -- Be free and open and breezy! Enjoy! Things won't get any better so get used to it.

1 1

[lxml-dev] Differences between lxml and elementtree output
by LC Rees 19 Jun '06

19 Jun '06

I modified a module that uses elementtree to work with lxml. Most of the changes required adjusting for the documented differences between elementtree and lxml. There was one difference in output, however, that was a little idiosyncratic. Using the following XML document: <rss version="2.0"> <channel> <title>Example</title> <link>http://www.example.org/</link> <description>RSS Example</description> <language>en-us</language> <pubDate id="cpubdate">$month $day, $year</pubDate> <lastBuildDate id="lastbuilddate">%(month)s %(day)s, %(year)s</lastBuildDate> <item class="item"> <title id="title" /> <link id="link" /> <guid id="guid" isPermaLink="true" /> <description id="description" /> <pubDate id="ipubdate" /> </item> </channel> </rss> I modified the element: <guid isPermaLink="true" id="guid"/> by adding an attribute consisting of name: '{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource' and value 'http://www.example.com/rss/5423093' through (internally) a call to the guid element's set method. Prior to this call, there were no namespaces declared in the document. After the call, in both the elementtree and lxml implementations, the guid element is modified to look like this: <guid xmlns:ns0="http://www.w3.org/1999/02/22-rdf-syntax-ns#" isPermaLink="true" id="guid" ns0:resource="http://www.example.com/rss/5423094"/> When I delete the attribute using (internally) the statement: del guid.attrib['{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource'] with the elementtree version the result looks like this: <guid id="guid" isPermaLink="true" /> while with the lxml version the result looks like this: <guid xmlns:ns0="http://www.w3.org/1999/02/22-rdf-syntax-ns#" isPermaLink="true" id="guid"/> Any ideas on why the namespace declaration lingers in the lxml output? Thanks, L.C. Rees lcrees(a)gmail.com

2 1

[lxml-dev] Some problem with an xpath
by Andreas Pakulat 16 Jun '06

16 Jun '06

Hi, I'd like to extract a string from an html document without caring where it is in the tree. However somehow my xpath expression returns all nodes :-( I tried to reproduce the problem with a mini script, but >>> xml=etree.XML('<a>TestTestTf<e />Testdgfd</a>') >>> xml.xpath('//b[fn:contains(self::text(),\'Test\')]') [<Element b at -4833fe1c>, <Element b at -4833fdf4>, <Element b at -4833fdcc>, <Element b at -4833fda4>] works as expected, however for my html page I get >>> html = etree.parse('/home/andreas/public_html/batman_dvd.html',etree.HTMLParser()) >>> html.xpath('//font[fn:contains(self::text(),\'Minuten\')]/text()') ['\n ', '\n ', '\n ', '\n ', u' Die folgenden Daten wurden noch nicht redaktionell \xfcberpr\xfcft.\n ', '\n ', '\n ', '\n ', '\n ', 'Erscheinungsart:\n ', '\n ', '\n ', 'Label:\n ', '\n ', '\n ', u'V\xd6-Termin:\n The page can be seen at http://www.ofdb.de/view.php?page=fassung&fid=1130&vid=148784 Is this a problem of lxml or my xpath expression? Even if I provide a more apropriate "start path", i.e. select a table deep in the hierarchy that contains the looked for element I get a lot of text nodes back. Andreas -- You definitely intend to start living sometime soon.

2 2