Mailman 3 September 2007 - lxml - The Python XML Toolkit

[lxml-dev] non-ascii characters get garbled
by js 27 Sep '07

27 Sep '07

Hello, list. The lxml doc [*1] says that "You should generally avoid converting XML/HTML data to unicode before passing it into the parsers. It is both slower and error prone." [*1] http://codespeak.net/lxml/parsing.html#python-unicode-strings but my experience is different from that. For example, the following code doesn't bother encoding things and leave the work to lxml.etree. According to the doc, this is right way, but it does't work and you'll got garbled characters. (give it a try) -------------------------------------------------------------------- # -*- coding: utf-8 -*- from lxml import html as etree url='http://apple.com/kr' tree = etree.parse(url) from pprint import pformat for t in tree.xpath('//a[text()]'): print t.text_content() -------------------------------------------------------------------- The next one break the rule and doing all charset conversion for oneself. This one works great and all charset conversion will succeed. -------------------------------------------------------------------- # -*- coding: utf-8 -*- from lxml import html as etree from urllib2 import urlopen from StringIO import StringIO url='http://apple.com/kr' res = urlopen(url) html = res.read().decode(res.headers.getparam('charset')) tree = etree.parse(StringIO(html)) from pprint import pformat for t in tree.xpath('//a[text()]'): print t.text_content() -------------------------------------------------------------------- But the latter doesn't always work. Sometimes I got "ValueError: Unicode strings with encoding declaration are not supported. " Is this a known issue? If so, how can I get out of this problem? Are there any workarounds? I tried to figure out the cause of these and looked over the lxml and libxml2 's code but could not find a clue. (To me this appeared to be not a lxml's problem but libxml2's ,though) Any information would be greatly appriciated. Thanks you in advance.

2 9

[lxml-dev] lxml 2.0alpha3 released
by Stefan Behnel 26 Sep '07

26 Sep '07

Hi all, I'm happy to announce the third alpha release of lxml 2.0. It features a number of clean-ups and another large refactoring of the parser code. The feed parser and the normal parsers now use separate contexts, so that you can use both concurrently on the same parser instance. Have fun, Stefan 2.0alpha3 (2007-09-26) Features added * Separate feed_error_log property for the feed parser interface. The normal parser interface and iterparse continue to use error_log. * The normal parsers and the feed parser interface are now separated and can be used concurrently on the same parser instance. * fromstringlist() and tostringlist() functions as in ElementTree 1.3 * iterparse() accepts an 'html' boolean keyword argument for parsing with the HTML parser (note that this interface may be subject to change) * Parsers accept an 'encoding' keyword argument that overrides the encoding of the parsed documents. * New C-API function hasChild() to test for children * annotate() function in objectify can annotate with Python types and XSI types in one step. Accompanied by xsiannotate() and pyannotate(). Bugs fixed * XML feed parser setup problem * Type annotation for unicode strings in DataElement() Other changes * lxml.etree now emits a warning if you use XPath with libxml2 2.6.27 (which can crash on certain XPath errors) * Type annotation in objectify now preserves the already annotated type be default to prevent loosing type information that is already there.

1 0

[lxml-dev] Python script to optimize XML text
by Robert Dailey 25 Sep '07

25 Sep '07

Hi, I'm currently seeking a python script that provides a way of optimizing out useless characters in an XML document to provide the optimal size for the file. For example, assume the following XML script: <root> <Test></Test>   </root> By running this through an XML optimizer, the file would appear as: <root><Test/></root> Note that the following were changed: - All comments were stripped from the XML - All spaces, tabs, carriage returns, and other forms of unimportant whitespace are removed - Elements that contain no text or children that are in the form of <Empty></Empty> use the short-hand method for ending an element body: <Empty/> Anyone know of a tool or python script that can perform optimizations like explained above? I realize I could probably do this with regular expressions in python, however I was hoping someone already did this work.

5 6

Re: [lxml-dev] 2.0alpha docs?
by Frederik Elwert 21 Sep '07

21 Sep '07

Am Freitag, den 21.09.2007, 09:31 -0500 schrieb Ian Bicking: > Frederik Elwert wrote: > > Am Freitag, den 21.09.2007, 09:25 -0500 schrieb Ian Bicking: > >> Are the docs for 2.0 built and up anywhere online? > > > > Yes, they are: > > > > http://codespeak.net/lxml/dev/ > > The front page doesn't have any links to > http://codespeak.net/lxml/dev/lxmlhtml.html -- but I'm not sure how the > doc index is created, so could you link that in? Sorry, forgot to answer to the list instead of you personally. I am not involved in the development of lxml, but somebody else on the list might look into this. Cheers, Frederik

2 1

[lxml-dev] 2.0alpha docs?
by Ian Bicking 21 Sep '07

21 Sep '07

Are the docs for 2.0 built and up anywhere online? -- Ian Bicking : ianb(a)colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers

1 0

[lxml-dev] time to ideas: a good API for iterparse() on HTML ?
by Stefan Behnel 21 Sep '07

21 Sep '07

Hi all, I wonder what a good API for iterparse() on HTML would be. I'm kinda tempted to change the iterparse class into a function like parse(), remove the existing keyword arguments and replace them with a standard "parser" argument as in parse() and fromstring(): >>> iterator = etree.iterparse(f, parser=etree.HTMLParser()) I'm not sure this works, as we can't support a parser target object ("target" keyword of parsers) or the feed parser interface with iterparse (which both the XMLParser and the HTMLParser currently support), but it wouldn't be obvious from the API that you can't pass a target parser into iterparse(). So it's not quite the perfect interface, as this would need to raise an error: >>> parser = etree.HTMLParser(target=SomeTarget()) >>> iterator = etree.iterparse(f, parser=parser) The alternatives would be an "html" keyword option to iterparse (the straight forward, simple solution, but which we use nowhere else in the API): >>> iterator = etree.iterparse(f, html=True) or a "method" argument like in the serialisers: >>> iterator = etree.iterparse(f, method="html") or maybe: >>> iterator = etree.iterparse(f, input_type="html") or an "iterparsehtml" function/class (which would be the worst thing to do IMHO): >>> iterator = etree.iterparsehtml(f) I feel that there should be some symmetry between iterparse(), the other parse functions and the parser classes, but I'm not sure which. Any comments? Stefan

1 1

[lxml-dev] CSS and lxml
by Frederik Elwert 17 Sep '07

17 Sep '07

Hello! I am currently looking into the possibilities to work with CSS in lxml or Python in general. I hope it's not too off-topic, but since it's related to lxml, I thought I might post it here. The specific use case I have in mind would take an element and look for all applying CSS rules. Let's say we have a style element: <style type="text/css"> p { font-size: 16pt; } p.strong { font-weight: bold; } </style> and in the tree, there are two elements: <p style="color:red;">Some text</p> <p class="strong">Some other text</p> Then I'd like to get something like >>> el1.getstyle() {'font-size': '16pt', 'color': 'red'} for the first element and >>> el2.getstyle() {'font-size': '16pt', 'font-weight': 'bold'} for the second one. I know that this is currently not possible. The only true CSS library for Python that I found were cssutils <http://cthedot.de/cssutils/>. They have quite sophisticated support for CSS parsing, but I think the library itself is quite DOM-centric and so it's not very pythonic / doesn't fit well to lxml. But more important, it has no real XML bindings. So it's possible to query stylesheets to get properties that match a selector: >>> stylesheet.props('p.strong') {'font-size': '16pt', 'font-weight': 'bold'} but not to query true elements to get the applying properties. On the other hand, lxml now has cssselect, which works the other way around: It takes a selector and returns all the elements that match that selector. >>> sel = CSSSelector('p.strong') >>> [e.text for e in sel(tree)] ['Some other text'] So I just wanted to ask if somebody already had thought about this, or if somebody has any ideas in which direction to head to solve this problem. Maybe one could write a module, that combines cssutils and lxml.cssselect to match css style properties and actual elements. But maybe a completely different approach would be needed. Regards, Frederik

2 5

[lxml-dev] XML files starting with BOM
by Gilles Lenfant 17 Sep '07

17 Sep '07

Hi from an lxml newbie, A first, many thanks for lxml that's the easiest XML lib for Python. lxml doesnt't like XML files starting with a BOM (See http:// www.w3.org/TR/2000/REC-xml-20001006#sec-guessing-no-ext-info). M$Office 2007 documents use such notation in their inner xml files. And I need to skip all chars from the file until I get a "<" before passing the stream to lxml. Hopefully, the files are UTF-8. Is it a bug or a feature ? -- Gilles Lenfant gilles.lenfant(a)gmail.com

2 1

[lxml-dev] Potential bug in trunk
by Anders Bruun Olsen 16 Sep '07

16 Sep '07

Hi, I've run into a weird problem. I am making a CherryPy-based application which uses lxml to do XSLT conversion of XML before sending it to the browser. It worked fine before switching to trunk (which has the namespace-patch that I need). Here is the output: [13/Sep/2007:13:08:37] HTTP Serving HTTP on http://0.0.0.0:8080/ [13/Sep/2007:13:08:44] Traceback (most recent call last): File "/usr/lib64/python2.4/site-packages/cherrypy/_cprequest.py", line 90, in run hook() File "/usr/lib64/python2.4/site-packages/cherrypy/_cprequest.py", line 58, in __call__ return self.callback(**self.kwargs) File "/home/abo/workspace/xmldict/src/xmldict/__init__.py", line 42, in transform_output xsltdoc = etree.parse(open(xslfile)) File "etree.pyx", line 2189, in etree.parse File "parser.pxi", line 1183, in etree._parseDocument File "parser.pxi", line 1217, in etree._parseFilelikeDocument File "parser.pxi", line 1126, in etree._parseDocFromFilelike File "parser.pxi", line 83, in etree._ParserDictionaryContext.getDefaultParser File "parser.pxi", line 585, in etree._BaseParser._copy AttributeError: 'lxml.etree._ResolverRegistry' object has no attribute '_copy' The really weird part is that when I start up the interactive interpreter and do the exact same operation it works: >>> from lxml import etree >>> xsltfile = "/home/abo/workspace/dicts/svda/svda.xsl" >>> xsltdoc = etree.parse(open(xsltfile)) Anybody able to venture a guess as to where this bug might lie? Is it in lxml, cherrypy or my code? -- Anders

2 1

[lxml-dev] lxml 2.0alpha2 released
by Stefan Behnel 15 Sep '07

15 Sep '07

Hi all, I just released lxml 2.0alpha2 to PyPI. http://pypi.python.org/pypi/lxml/2.0alpha2 http://codespeak.net/lxml/dev/ It features a number of major API additions that follow the ElementTree library and the future API changes in ElementTree 1.3. The main new features are HTML serialisation support, a feed interface to the parsers, a SAX-like target parser interface, and iterfind() as an iterator version of findall(). All of these are currently more or less experimental, so feedback is warmly welcome. The mailing list is always open for discussion, not only on the new features. The complete changelog is below. Have fun Stefan 2.0alpha2 (2007-09-15) Features added * ET.write(), tostring() and tounicode() now accept a keyword argument "method" that can be one of 'xml' (or None), 'html' or 'text' to serialise as XML, HTML or plain text content. * iterfind() method on Elements returns an iterator equivalent to findall() * itertext() method on Elements * Setting a QName object as value of the .text property or as an attribute will resolve its prefix in the respective context * ElementTree-like parser target interface as described in http://effbot.org/elementtree/elementtree-xmlparser.htm * ElementTree-like feed parser interface on XMLParser and HTMLParser (feed() and close() methods) Bugs fixed * lxml failed to serialise namespace declarations of elements other than the root node of a tree * Race condition in XSLT where the resolver context leaked between concurrent XSLT calls Other changes * element.getiterator() returns a list, use element.iter() to retrieve an iterator (ElementTree 1.3 compatible behaviour)

1 0