Mailman 3 January 2008 - lxml - The Python XML Toolkit

[lxml-dev] Many thanks to...
by Gilles Lenfant 30 Jan '08

30 Jan '08

The lxml developers great team. I just released some days ago openxmllib, a Python library that extracts text and meta-data from OpenXML documents (MS Office 2007, Apple iWork, and some others) for full text indexing purpose. Perhaps more features in the future. http://code.google.com/p/openxmllib/ Got headaches reading and understanding OpenXML docs. Hopefully, lxml is so easy to work with and so fast... The words of a 60 pages Word .docx document is now extracted in 0.2 seconds instead of 8 seconds on my MacBook and I removed 60% of the code volume since I switched from the standard XML libs that come with Python 2.4. lxml rocks and grooves -- Gilles Lenfant gilles.lenfant(a)gmail.com

2 1

[lxml-dev] Message from a happy user!
by Charles Brossollet 29 Jan '08

29 Jan '08

Hi all, This is not a question, nor a bug report, but a message to thank the lxml team. I had a script using ElementTree, that was taking 10s. I tried to switch to cElementTree, but this broke my code. So I gave a try to lxml, changed nothing to my code except the import statement, and zooom, it runs in 1 second. Thank you very much for you work, and I hope you will get more visibility in the xml python libs, because you deserve it! Charles Brossollet

3 3

[lxml-dev] removing nested element from html wipes text
by iShTa dAsaH 29 Jan '08

29 Jan '08

Hi I got missing text after removing last element.. I've tried 2.0 beta but still no luck. txt=""" '<p> startPara <strong>str0ng</strong> MiddleP <u>UUUUUUUUU</u> <b>Bold</b> EndPara </p>""" ee=etree.HTML(txt) pp=ee.xpath('//p')[0] >>> pp.tail >>> pp.xpath('./text()') ['startPara', 'MiddleP', 'EndPara'] >>> bold=pp.xpath('./b')[0] >>> bold.clear() >>> pp.xpath('./text()') ['startPara', 'MiddleP'] as at seen EndPara text is missing

2 1

Re: [lxml-dev] [XML-SIG] Converting XML Schema to data struture and then to XML
by Stefan Behnel 28 Jan '08

28 Jan '08

Hi, Stefan Behnel wrote: > Good Z wrote: >>>> I need to write an application to convert XML schema (.xsd) to a data >>>> structure then using this data structure and the data from some other >>>> files, generate an XML file. >>>> >>>> I have seen generateDS and pyfo library that can be combined to write >>> this >>>> application but my problem is that i somehow do not like DOM and >>> would like >>>> to write this in ElementTree. I wonder if anyone can point me how to >>>> write >>>> this quickly in ElementTree or someone has already done this effort. >> In case you can do without the schema, lxml.objectify might be worth a >> look: >> >> http://codespeak.net/lxml/dev/objectify.html >> >> Thanks Stefan, >> >> But my primary requirement is converting XML Schema. > > Sure, no problem. But may I ask what the reasoning behind this requirement is? > I mean, code generation isn't really a value by itself. Just in case you're interested (and since similar topics have come up a couple of times on lxml's mailing list), I added a doc section on parse-time schema validation in lxml 2.0 here: http://codespeak.net/svn/lxml/trunk/doc/objectify.txt Look out for section 2, "Asserting a Schema". It's not yet included in the online docs at http://codespeak.net/lxml/dev/objectify.html but it will appear there by the end of the week. Stefan

1 0

[lxml-dev] RESOLVED Re: document('') and custom resolver not working again/still
by Brad Clements 27 Jan '08

27 Jan '08

Ok this is definitely not an lxml problem. Here's how you can recreate the same problem I had.. duh! If you (accidentally) use the same base_url for when loading both the xsl and xml source, there's a 50% chance that document('') will return the xml source, not the xsl source object. class Tester(object): def test(self): xsl_resolver = Resolver(fake_resolve_function) xsl_parser = etree.XMLParser(load_dtd=True) xsl_parser.resolvers.add(Resolver(resolver=xsl_resolver)) stylesheet_doc = etree.fromstring(xsl_src, xsl_parser, base_url='/carriers/view.htm') stylesheet = etree.XSLT(stylesheet_doc) xml_resolver = Resolver(fake_resolve_function) xml_parser = etree.XMLParser(load_dtd=True) xml_parser.resolvers.add(Resolver(resolver=xml_resolver)) xml_doc = etree.fromstring(xml_src, xml_parser, base_url='/carriers/view.htm') return str(stylesheet(xml_doc)) -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM: BKClements

2 1

[lxml-dev] document('') and custom resolver not working again/still
by Brad Clements 27 Jan '08

27 Jan '08

I am stumped. Probably doing something stupid but I just cannot get document('') to work with lxml, but it does work when the .xsl is sent to the browser (ff) and it does work with xsltproc I think there's been a change in libxslt at some point, but I don't know exactly what changed. This deployment works: RHEL4 linux with libxml2 2.6.27 and libxslt 1.1.18, with lxml 2 alpha4 this deployment does not work: Centos 4 with libxml2 2.6.28 and libxslt 1.1.19 with lxml 2 alpha 3 So I upgraded to libxml2 2.6.30, libxslt 1.1.22 and lxml 2 beta 2 But it still doesn't work correctly. I am passing a base_url when creating documents using etree.fromstring I've traced the execution of document('') via gdb, both in libxslt/functions.c and libxslt/documents.c For example, in libxslt/functions.c 142 idoc = xsltLoadDocument(tctxt, URI); (gdb) print URI $7 = (xmlChar *) 0x90e7c00 "/carrier_payables/view.htm" the correct uri (what I passed for base_url) is shown and Now in documents.c line 315:: 317 if ((ret->doc != NULL) && (ret->doc->URL != NULL) && (gdb) print ret->doc $11 = 0x90d6ef8 (gdb) print ret->doc->URL $12 = (const xmlChar *) 0x90dca08 "/carrier_payables/view.htm" (gdb) print URI $13 = (const xmlChar *) 0x90e7c00 "/carrier_payables/view.htm" so all seems to be correct, it is returning the expected document back to the xpath evaluator. I'm stumped that it works with older versions of libxslt, works with client-side transform, and works with xsltproc, but not through lxml with "newer libxslt" Can anyone suggest some other steps I can use to diagnose this problem? I'm sure I've done something wrong with how I am using lxml, but I can't figure it out. My .xsl looks (in part) like this. Its loaded using etree.fromstring with a base_url: <?xml version="1.0"?> <xsl_:stylesheet xmlns:xsl_="http://www.w3.org/1999/XSL/Transform" xmlns:metal="http://xml.zope.org/namespaces/metal" xmlns:tal="http://xml.zope.org/namespaces/tal" xmlns:const="const.uri" version="1.0" exclude-result-prefixes="tal metal const"> <xsl_:output encoding="utf-8" method="xml" omit-xml-declaration="no" cdata-section-elements="" doctype-public="-//W3C//DTD XHTML 1.0 Transitional//EN" doctype-system="http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"/> <const:javascript id="number_pad_javascript"> <const:file>MochiKit/MochiKit.js</const:file> <const:file>jstal/jstal.js</const:file> <const:file>global.js</const:file> <const:file>view.js</const:file> </const:javascript> <xsl_:template match="/"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title> View Carrier Payables </title> <link rel="stylesheet" type="text/css" href="/css/default.css"/> <link rel="stylesheet" media="print" type="text/css" href="/css/print.css"/> </head> <body> <div id="ajax-indicator" style="display:none" class="no-print"> <img title="some title" width="24" height="24" src="/images/ajax_indicator.gif"/> </div> <xsl_:for-each select="document('')//const:javascript/const:file"> <xsl_:variable name="file" select="."/> <script type="text/javascript"> <xsl_:attribute name="src"> <xsl_:value-of select="concat('/scripts/', $file)"/> </xsl_:attribute> </script> </xsl_:for-each> <div>more stuff</div> </body> </html> </xsl_:template> </xsl_:stylesheet> using xsltproc against a source xml file: <root /> produces this output <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title> View Carrier Payables </title> <link rel="stylesheet" type="text/css" href="/css/default.css" /> <link rel="stylesheet" media="print" type="text/css" href="/css/print.css" /> </head> <body> <div id="ajax-indicator" style="display:none" class="no-print"> <img title="some title" width="24" height="24" src="/images/ajax_indicator.gif" /> </div> <script type="text/javascript" src="/scripts/MochiKit/MochiKit.js"></script> <script type="text/javascript" src="/scripts/jstal/jstal.js"></script> <script type="text/javascript" src="/scripts/global.js"></script> <script type="text/javascript" src="/scripts/view.js"></script> <div>more stuff</div> But with lxml, I get: <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title> View Carrier Payables </title> <link rel="stylesheet" type="text/css" href="/css/default.css" /> <link rel="stylesheet" media="print" type="text/css" href="/css/print.css" /> </head> <body> <div id="ajax-indicator" style="display:none" class="no-print"> <img title="some title" width="24" height="24" src="/images/ajax_indicator.gif" /> </div> <div>more stuff</div> I am using a custom resolver. The resolver is not asked to resolve the .xsl stylesheet, nor is it asked to resolve '' p-code is like this: parser = etree.XMLParser(load_dtd=True) parser.resolvers.add(Resolver(resolver=my_resolver)) stylesheet_doc = etree.fromstring(xslt_src, parser, base_url=xsl_uri) stylesheet = etree.XSLT(stylesheet_doc) likewise the source xml is loaded in the same way parser = etree.XMLParser(load_dtd=True) parser.resolvers.add(Resolver(resolver=my_resolver)) xml_doc = etree.fromstring(xml_src, parser, base_url=xml_uri) finally return stylesheet(xml_doc, **params) -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM: BKClements

2 1

[lxml-dev] lxml 2.0beta2 on PyPI
by Stefan Behnel 26 Jan '08

26 Jan '08

Hi, I just released lxml 2.0beta2 to PyPI. This is the last beta before the final release, which is to be expected on February 1st. So, last chance to find bugs. :) Changelog follows, as usual. Have fun, Stefan 2.0beta2 (2008-01-25) Features added * Plain ASCII XPath string results are no longer forced into unicode objects as in 2.0beta1, but are returned as plain strings as before. * All XPath string results are 'smart' objects that have a getparent() method to retrieve their parent Element. * with_tail option in serialiser functions. * More accurate exception messages in validator creation. Bugs fixed * Missing import in lxml.html.clean. * Some Python 2.4-isms prevented lxml from building/running under Python 2.3. Other changes * Exceptions carry only the part of the error log that is related to the operation that caused the error. * XMLSchema() and RelaxNG() now enforce passing the source file/filename through the file keyyword argument. * The test suite now skips most doctests under Python 2.3. * make clean no longer removes the .c files (use make realclean instead)

1 0

[lxml-dev] elements in gc.garbage - ok ?
by Jeroen van Hilst 26 Jan '08

26 Jan '08

Hi, While using lxml (which is a great tool!), i am experiencing some memory issues. I have made a small piece of code that makes elements go into in gc.garbage. I am suspecting this to be the problem of my program cunsuming lots of memory. Can someone tell me if it is ok that this happens - or what the reason is ? Thanks in advance. - Jeroen #====================== from lxml import etree import gc gc.set_debug(gc.DEBUG_LEAK) for x in range(1,19): r = etree.Element('div') s = str(x) #if the attr is not touched there are no messages from gc r.attrib[s] = s gc.collect() if gc.garbage: print gc.garbage #====================== I am using: Windows XP SP2 Python 2.5.1.1 lxml 1.3.6 libml 2.6,28 begin 666 lxml_mem.py M(U5S:6YG.@HC4'ET:&]N(#(N-2XQ+C$*(U=I;F1O=W,@6% @4U @,@HC;'AM M;" Q+C,N-@HC;&EB>&UL(#(N-BXR. H*9G)O;2!L>&UL(&EM<&]R="!E=')E M90II;7!O<G0@9V,*"F=C+G-E=%]D96)U9RAG8RY$14)51U],14%+*0H*9F]R M('@@:6X@<F%N9V4H,2PQ.2DZ"B @("!R(#T@971R964N16QE;65N="@G9&EV M)RD*(" @(',@/2!S='(H>"D*(" @("-I9B!T:&4@871T<B!I<R!N;W0@=&]U M8VAE9"!T:&5R92!A<F4@;F\@;65S<V%G97,@9G)O;2!G8PH@(" @<BYA='1R M:6);<UT@/2!S"@IG8RYC;VQL96-T*"D*:68@9V,N9V%R8F%G93H@<')I;G0@ +9V,N9V%R8F%G90H` ` end

2 2

Re: [lxml-dev] cssselect and cssutils
by Ian Bicking 25 Jan '08

25 Jan '08

Höke, Christof wrote: >> ::first-letter is hard because it doesn't match any object in lxml. >> If it returned a string like "A" it would be very much out of >> context (e.g., no parent pointer), and it would be hard to do >> anything useful with it. To make it useful I think it would >> require some new stringish object that also looked nodeish (e.g., >> had a .getparent() method). Though maybe an object like that should >> exist; something similar would be needed for representing ranges. > > What came to my mind was the DOM range spec stuff, but it is not > really finished, is it? I was reading about it some years (!) ago I > think in the Javascript Definitive Guide but I think it never went > anywhere really. It doesn't really matter too much, since lxml isn't that much like the DOM. But the same use cases for the DOM range can apply to lxml. > :first-letter should actually be element.text[0] I guess (which would > be a string in lxml currently?), I don't really know the lxml API but > would it be possible to define a subtype for element.text for this > case? But you are right, a more general approach would certainly be > better. element.text is just a unicode string. Maybe we could have a method like element.text_range(0, 1) that returns a subclass of unicode that also happens to know something about its location. E.g.: class ElementText(unicode): def __new__(cls, text, is_tail, range, parent): self = unicode.__new__(cls, text) self.is_tail = is_tail self.range = range self._parent = parent def getparent(self): return self._parent def enclose_in_tag(self, el): """ Enclose this text range in an element, like:: span = Element('span') el.text_range(0, 1).enclose_in_tag(span) """ parent = self.getparent() el.text = unicode(self) if self.is_tail: el.tail = parent.tail[self.range[1]:] parent.tail = parent.tail[:self.range[0]] index = parent.getparent().index(parent) parent.getparent().insert(index+1, el) else: el.tail = parent.text[self.range[1]:] parent.text = parent.text[:self.range[0]] parent.insert(0, el) self._parent = el self.range = (0, len(self)) self.is_tail = False All untested, of course. A real sense of a range would be a bit more difficult, as it involves lots of partial elements. But something like this would be necessary to do that work. Upon further thought, maybe subclassing unicode isn't the right thing -- perhaps it should really just wrap a string. Then perhaps you could just have do, say, el.text_range[:5], where e.text_range is a range object for all of its text, and you could slice range objects to further break them down. But dealing with changes to the element are tricky. It's all a bit tricky ;) -- Ian Bicking : ianb(a)colorstudy.com : http://blog.ianbicking.org

2 7

[lxml-dev] About objectify
by David Soulayrol 25 Jan '08

25 Jan '08

Hello, I wanted to give a try to objectify, and while reading documentation, I've seen that what I'd like to do involves using a schema for my XML file, and the xsi namespace to specify element types. Since I do discover at the same time the objectify particularities and what is exactly a schema in details, I have some troubles to handle the whole thing, and I'm not sure I do understand well everything from documentation. The main problem is that if I do an error, for now I do not know if I should try correct my schema, my XML file or my python code. Does anyone have a simple example with a short XML file containing some elements, its schema, and the python code parsing the file using objectify with the xsi namespace to do type association ? Thanks, -- David.

3 7