Mailman 3 May 2008 - lxml - The Python XML Toolkit

[lxml-dev] Regarding the ElementTree API
by Deepali H. Dhabade 30 May '08

30 May '08

ElementTree API does not support namespace prefixes.Is there any latest ElementTree API which gives full support for namespace prefixes. I need this because I am working on openxml+python support

2 1

[lxml-dev] DOM tree intersection/comparison?
by Viksit Gaur 27 May '08

27 May '08

Hi all, I was wondering if there's a currently implemented way to find out the common elements between 2 DOM trees? If not (I couldn't find any obvious classes or functions) - what would you recommend as the best method to do so? Use iterparse/iterwalk on two pages, and then do a side by side comparison looks like a naive method.. Btw, when I say comparison, the basic aim is to figure out dom element sequences or subtrees that are common. For instance, a page with <html> <div> <a> </div> </html> and <html> <div> <a> <hr> </div> <div> <b> </div> </html> would have div->a as a common element, amongst other things.. Cheers, Viksit

3 7

Re: [lxml-dev] Trouble parsing large XML document with ElementTree
by Sam Kuper 25 May '08

25 May '08

Stefan, My apologies and thanks - my eyes obviously weren't working so well yesterday! The program's running smoothly now (aside from some "'ascii' codec can't encode character" exceptions that I'm "try/except"ing until I've understood how to handle them with lxml; I realise the docs cover this but I (still) don't quite understand them all yet). Thanks again. Gratefully, Sam 2008/5/25 Stefan Behnel <stefan_ml(a)behnel.de>: > > result = etree.XML("content.example.xml", parser) > > result = etree.parse("thefile.xml", parser) > > See the difference? >

1 0

Re: [lxml-dev] problem\bug in xpath compare() with text in tail
by Stefan Behnel 25 May '08

25 May '08

Hi, please keep the list involved. matan ninio wrote: > Is there some good place to look for information about XPath? Search for "xpath tutorial" ? Stefan

4 5

[lxml-dev] Trouble parsing large XML document with ElementTree
by Sam Kuper 25 May '08

25 May '08

Dear lovely lxmlves, Yesterday I tried to parse a large file, the Open Directory Project's links document, available here <http://rdf.dmoz.org/rdf/content.rdf.u8.gz>. The process went like this: 1) Unzipped the file using 7-zip. No errors reported. 2) Renamed the file by adding a .xml extension, mainly so Windows (see my spec below) would recognise it as an XML file. 3) Had a look at the file in Oxygen's large document viewer. It took a few minutes to load, but everything looked shipshape. 4) Opened a command prompt, navigated to the directory containing the file, and started Python. 5) Entered: from lxml import etree 6) Entered: doc = open ('content.rdf.u8.xml', 'r') 7) Entered: docParsed = etree.parse(doc) Steps 4, 5 and 6 all went smoothly, but after step 7, the RAM usage went up to around 96% (fair enough, it's a big document) and the Windows UI became sluggish. It didn't crash, and the RAM usage stabilised around that amount, with Windows Task Manager showing well under 10% CPU load from Python. Still, I figured it might take a while to parse, so I left it overnight. In the morning, I found the following error message immediately underneath the command I'd entered in step 7: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "lxml.etree.pyx", line 2520, in lxml.etree.parse File "parser.pxi", line 1331, in lxml.etree._parseDocument File "parser.pxi", line 1361, in lxml.etree._parseFilelikeDocument File "parser.pxi", line 1254, in lxml.etree._parseDocFromFilelike File "parser.pxi", line 850, in lxml.etree._BaseParser._parseDocFromFilelike File "parser.pxi", line 452, in lxml.etree._ParserContext._handleParseResultDoc File "parser.pxi", line 536, in lxml.etree._handleParseResult File "parser.pxi", line 478, in lxml.etree._raiseParseError lxml.etree.XMLSyntaxError: Memory allocation failed : building node I hope that's meaningful to someone, and that perhaps I might be able to get some suggestions about how to parse the file on my PC. Also, I was thinking of trying to parse the file on a virtual server that only has 64M of RAM. I don't mind if the VPS takes a day or two, as long as the code to make it work is fairly straightforward. So any suggestions about that option would be helpful too. Many thanks, Sam --- Macbook 2.13GHz with 2GB RAM Windows Vista Home Premium via Leopard BootCamp ActivePython 2.5.1 lxml installed via lxml-2.0.3-py2.5-win32.egg (this was the most up-to-date egg that was available last time I checked, which was about a week or two ago)

2 8

Re: [lxml-dev] Trouble parsing large XML document with ElementTree
by Sam Kuper 24 May '08

24 May '08

To add to the message below, I've just tried running a much simpler program that doesn't call lxml to see if the memory error is a Python/environment one rather than being due to lxml. It turns out to be: >>> infile = open("content.rdf.u8.xml", "r") >>> print infile.read() Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError Ok, so clearly Python isn't happy to read content.rdf.u8.xml in one go. The normal workaround for processing large text files piece by piece seems to be either to set a byte limit on how much is read at once, or to read the file line by line. However, neither of those will work in this case because they won't produce well-formed XML that the target parser interface can handle (correct me if I'm wrong). I'm sure there must be a fairly easy solution to this, but it's eluding me. All assistance greatly appreciated! Sam 2008/5/24 Sam Kuper <sam.kuper(a)uclmail.net>: > Dear Stefan, > > I've tried the method you've suggested below, but it isn't quite working > for me. It may be that I've misunderstood your suggestion. I'll explain what > I've tried. Here is my python program, extract_links_dmoz.py: > > from lxml import etree > infile = open("content.example.xml", "r") > infile.seek(0) > outfile = open("output_test001.txt", "w") > class EchoTarget(): > def start(self, tag, attrib): > if tag.endswith("xternalPage"): > line = attrib["about"] > if line != "": > outfile.write(line+"\n") > print line > def close(self): > return "closed!" > parser = etree.XMLParser(target = EchoTarget()) > result = etree.XML(infile.read(), parser) > > This uses the short, example RDF file at > http://rdf.dmoz.org/rdf/content.example.txt (which I have renamed > content.example.xml), and works fine. When I view the output_test001.txt > file, it contains one URL per line, which is exactly what I want for now. > > However, if I change the program to read content.rdf.u8.xml (i.e. the > full-length DMOZ links file from http://rdf.dmoz.org/rdf/content.rdf.u8.gz) > instead of content.example.xml , then when I run the program I get the > following error: > > Traceback (most recent call last): > File "extract_links_dmoz.py", line 26, in <module> > result = etree.XML(infile.read(), parser) > MemoryError > > Any help you (or others) can offer would be greatly appreciated. > > Many thanks, > > Sam > > 2008/5/22 Stefan Behnel <stefan_ml(a)behnel.de>: > >> Hi, >> >> Sam Kuper wrote: >> > Gosh, this is turning into a really fragmented post; apologies. I meant >> to >> > add to the first post that once parsed, my intention was to run a fairly >> > simple XSL transform on the document, to extract a copy of each of the >> URLs >> > it contains. Probably something like this: >> > <?xml version="1.0" encoding="UTF-8"?> >> > <xsl:stylesheet version="1.0" xmlns:xsl=" >> > http://www.w3.org/1999/XSL/Transform"> >> > <xsl:template match="/"> >> > <html> >> > <body> >> > <h2>ODP URLs</h2> >> > <xsl:for-each select="Topic/link"> >> > <p><xsl:value-of select="@r:resource"/></p> >> > </xsl:for-each> >> > </body> >> > </html> >> > </xsl:template> >> > </xsl:stylesheet> >> >> That is a problem that can be solved with extremely little memory. Take a >> look >> at the (SAX-like) target parser interface, which will not build a tree and >> instead just receive callbacks while parsing: >> >> http://codespeak.net/lxml/dev/parsing.html#the-target-parser-interface >> >> Write a parser target class that keeps track of being inside or outside >> the >> "Topic" tag (start/end), and whenever you find a "link" tag while inside a >> "Topic" tag, look for a "{whatever-namespace}resource" attribute in the >> attrib >> dictionary and and write it into a hand-generated HTML stream like the one >> you >> used above. >> >> Stefan >> > > >

1 1

[lxml-dev] Efficient way to search element attribs?
by Viksit Gaur 24 May '08

24 May '08

Hi all, When building a tree, I add a particular attribute to each element (eg: ID), and was wondering if there are any methods I could use to search the tree for tags that have this attribute? I looked at find(), findall() and the xpath functions - but they seem to help only when you know the tag name itself. Of course, one way might be to do a manual depth first or breadth first search from the root downwards, recursively.. But that would become expensive pretty soon. Thanks, Viksit

2 1

[lxml-dev] problem\bug in xpath compare() with text in tail
by Matan Ninio 24 May '08

24 May '08

This may be a just my (limited) understanding of Xpath and XML, but i'm getting a strange problem when I try to use xpath to search for specific strings in a file. specifically, when I use "\\*[compare(text(),"needle")]" to look for elements with "needle" in their text, it only works when the strings appears in the "text" part, but not when its in the "tail" part. So: <prompt> e=etree.HTML("<html><body>inbody<h5>text</h5>tail</body></html>") <prompt> e.xpath("//text()") ['inbody', 'text', 'tail'] <prompt> e.xpath("//*[contains(text(),'text')]//text()") ['text'] ---- works fine, but <prompt> e.xpath("//*[contains(text(),'tail')]//text()") [] ---- does not. is it just that I need to use a different function/attribute for the tail (instead of text())? Is this a bug? Is there a workaround? using lxml 2.0.5, via mac "port" thanks!

2 1

[lxml-dev] bug: objectify removes text on replace()?
by Christian Zagrodnick 16 May '08

16 May '08

Hi, with lxml 2.0.4 I get text removed when I replace a node. The text after the replaced node vanishes.... ----------------------- import lxml.objectify import lxml.etree xml = lxml.objectify.fromstring( '<foo><bar>before baz<baz/>after baz</bar><bar/></foo>') print lxml.etree.tostring(xml, pretty_print=True) print 50*'-' baz = xml['bar']['baz'] xml['bar'].replace(baz, lxml.objectify.E.holler()) print lxml.etree.tostring(xml, pretty_print=True) ----------------- Prints out: <foo> <bar>before baz<baz/>after baz</bar> <bar/> </foo> -------------------------------------------------- <foo> <bar>before baz<holler xmlns:py="http://codespeak.net/lxml/objectify/pytype" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/></bar> <bar/> </foo> Thanks, -- Christian Zagrodnick · cz(a)gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 4 · fax +49 345 1229889 1 Zope and Plone consulting and development

2 2

[lxml-dev] Efficient methods to build a tree out of HTML structure?
by Viksit Gaur 16 May '08

16 May '08

Hi all, I was wondering - what would be the most efficient method to access all the elements in the DOM tree, in some order, using lxml.etree? The methods I currently see in the docs return a class like ElementDepthfirstIterator or iterwalk, which have 2 issues - 1) The first has a flat representation of the tree, so I lose child/parent structure 2) Things like iterwalk do return "start" and "end" actions - but instead of first doing an iterwalk and then parsing the results, is there a better way to construct the tree when iterwalk itself is running? Or perhaps there is some method I've missed completely? Quick note on what I'm trying to do - graphically represent the DOM structure of a page using a library like networkX.. Cheers, Viksit

3 5