Mailman 3 February 2010 - lxml - The Python XML Toolkit

[lxml-dev] Problem writing special HTML characters &#...
by roby.brunelli＠gmail.com 26 Feb '10

26 Feb '10

I'm trying to write an RSS file (extracting information from an html page) using etree.ElementTree(..).write(..) When I create the description part of a news I insert text with special characters such as: È and when I print (or write to file) the corresponding element, I get &#200 which I do not want (I want the original special char): is there a way to prevent this kind of mapping?? Thanks a lot,

1 0

[lxml-dev] Why <!DOCTYPE implies xmlns="http://www.w3.org/1999/xhtml" and .tags like {http://www.w3.org/1999/xhtml}head ?
by Mateusz Korniak 26 Feb '10

26 Feb '10

Hi ! I am trying to perform simple task, "fix" any (X)HTML page to valid XML (XHTML) page. I am parsing source with lxml.html.HTMLParser(remove_comments=True,remove_pis=True) parser, than generating XML: lxml.etree.tostring(html_doc,pretty_print=True,with_tail=False,method="xml") To verify results I parse result but now using recovering_xml_parser = lxml.etree.XMLParser(recover=True) Whole code is attached. I works great except case when source has <!DOCTYPE declaration, then instead of expected tags like 'body' I end up with '{http://www.w3.org/1999/xhtml}body' :/ My questions are: 1) why it happens ? 2) how one should parse HTML of any kind to always end up with XML which after parsing contains tags values without any namespace ? Thanks in advance for any help, hints Regards, -- Mateusz Korniak

2 1

Re: [lxml-dev] Question about newlines
by fantasai 24 Feb '10

24 Feb '10

Stefan Behnel <stefan_ml <at> behnel.de> writes: > > Noah Slater wrote: > > On Sun, Dec 09, 2007 at 08:48:17AM +0100, Stefan Behnel wrote: > >> Serialisation will never alter content. > > [snip] > >>> 1) When adding a PI via the element.addprevious method and PI has > >>> it's tail trimmed and so when serialising the PI runs into the > >>> root element. > > > > Well, this is well and good but lxml REMOVES the PI tail so I cannot > > insert a newline even if I want to. > > Ah, got it. Thanks for insisting. :) > > lxml.etree does this on purpose. If you allow character data around the > processing instructions that you add as siblings of the root node, you need to > make sure it's only whitespace (not 'real' data) to keep the in-memory tree > well-formed and to serialise well-formed XML. So the behaviour would be: strip > the tail, but keep it if it's whitespace. Sounds a bit ugly to me... > > I also noted that libxml2's parser drops whitespace at the root level, which > is perfectly fine, as it is the most definitely ignorable whitespace there is. > I personally prefer having lxml add a line break when serialising processing > instructions and comments at the root level, and cosistently dropping all tail > text of PIs and comments appended/prepended to a root node. So the behaviour > for the root level would be: drop all whitespace when parsing, and add line > breaks around PIs and comments on serialisation. > > There's also the document ending issue. The document serialiser of libxml2 > does append a newline, and one day, lxml may switch to using it. So I added > this behaviour now - and had to adapt tons of test cases that compare > serialised XML between ET and lxml. But I don't mind having white-space > differences in the serialisation as long as it's well-formed, equivalent XML. I have a related problem, more like this one: http://stackoverflow.com/questions/973079/how-can-i-make-lxmls-parser-prese… Maybe you want to make it a parse-time option, but I think white space outside the root should be preserved. And stripping the addition to the document root context only if it's not all whitespace seems to me the right thing to do. ~fantasai

3 5

[lxml-dev] ElementTree.iterparse segfault
by Cserna, Zsolt 23 Feb '10

23 Feb '10

Hi all, There's a test in selftest2.py which fails (results segfault) on my 64-bit linux host: def iterparse(): """ Test iterparse interface. >>> iterparse = ElementTree.iterparse >>> context = iterparse("samples/simple.xml") >>> for action, elem in context: ... print("%s %s" % (action, elem.tag)) end element end element end empty-element end root I've created a standalone script which contains: from lxml import etree as ElementTree iterparse = ElementTree.iterparse context = iterparse("samples/simple.xml") for i in context: print i ...and it also results segmentation fault. Where samples/simple.xml is the xml bundled with lxml: <root> <element key='value'>text</element> <element>text</element>tail <empty-element/> </root> I've tested it with python 2.6.4, 2.5.4, and lxml 2.2.2 and 2.2.4 (2.2.2 is linked with libxml2 2.7.3, libxslt 1.1.24, 2.2.4 is linked with libxml2 2.7.6). It only happens on 64-bit linux (RHEL 4), my 32-bit linux systems (RHEL 3 and RHEL 4) are fine. Any advise on this bug? Could you check that it also happens on you 64-bit linux system? Thanks for advance, Zsolt -------------------------------------------------------------------------- NOTICE: If received in error, please destroy, and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error. We may monitor and store emails to the extent permitted by applicable law.

2 5

[lxml-dev] Custom XSLT functions and text nodes
by Ross Burton 20 Feb '10

20 Feb '10

Hi, I'm trying to perform some string manipulation on text nodes via a custom XSLT function. My XSLT has this: <xsl:template match="text()"> <my:stringsplit/> </xsl:template> But no matter what I do the context node doesn't appear to have any text content. Using pdb: (Pdb) print input_node <Element text at 7fbfc4f10e68> (Pdb) print input_node.tag text (Pdb) print input_node.tail None (Pdb) print input_node.text None I expect I'm doing something rather stupid, but how can I get the text so that I can manipulate it and then put it in the result tree? Cheers, Ross

2 1

[lxml-dev] is it ok to move lxml build after running setup.py?
by John Byrne 18 Feb '10

18 Feb '10

Hi, Complete newbie to lxml here. I have just built the version 2.2.4 using the setup.py script and I was wondering if it actually installs/copies anything anywhere. The reason is, I built it in my home directory, but now I'd like to move it. I'm guessing that I just need to have the "build/lib.linux-i686-2.4/lxml" directory on my PYTHONPATH, but I want to be sure I don't mess anything up. Is it OK to move the above directory, and then include that new location on PYTHONPATH? Thanks (and sorry if this is an obvious one, I am not too familiar with setup.py etc.) -John

2 1

[lxml-dev] lxml.html.tostring & CDATA
by Jordan Baker 17 Feb '10

17 Feb '10

Trying to embed a script that needs to be inside a CDATA and it seems to be escaping the CDATA itself: mydoc = """ <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 4.01//EN"> <html> <head> <title></title> </head> <body> <script type="text/javascript"> <![CDATA[ ... unescaped script content ... ]]> </script> <img src="http://foo.jpg"/> </body> </html> """ >>> print tostring(document_fromstring(mydoc), method="xml") <html xmlns="http://www.w3.org/1999/xhtml "><head><title></title></head><body> <script type="text/javascript"> <![CDATA[ ... unescaped script content ... ]]> </script><img src="http://foo.jpg" /></body></html> Is this is a bug? -- Jordan Baker Scryent Clearly Open Source Plone, Zope, Python, Linux & more +1 416 871-3810 www.scryent.com

2 6

Re: [lxml-dev] lxml.html.tostring & CDATA
by Chuck Bearden 17 Feb '10

17 Feb '10

On Tue, Feb 16, 2010 at 2:46 PM, Jordan Baker <jbb(a)scryent.com> wrote: > On Tue, Feb 16, 2010 at 3:44 PM, Jordan Baker <jbb(a)scryent.com> wrote: >> >> Trying to embed a script that needs to be inside a CDATA and it seems to >> be escaping the CDATA itself > > Noticed that this is lxml 2.1.2 ... will try with a more recent version Does HTML 4.01 Transitional define CDATA sections? I can't find any reference to them in the specification at the W3C. I'm not a standards guru, though, and I may well have missed something. Chuck

2 1

[lxml-dev] Building an ESI tag with lxml
by Martin Aspeli 09 Feb '10

09 Feb '10

Hi, I'm trying to use lxml to conditionally insert an <esi:include /> tag into an HTML document. The document is parsed with the HTML parser and manipulated in various ways. At one point, I search for a node ('placeholder') and want to replace it with something that renders to: <esi:include src="http://..." /> The code I used looks like this: nsmap = {'esi': 'http://www.edge-delivery.org/esi/1.0'} root.nsmap.update(nsmap) # root is the <html /> element esiNode = etree.Element("{%s}include" % nsmap['esi'], nsmap=nsmap) esiNode.set('src', url) placeholder.addnext(esiNode) # placeholder is later removed There are two problems with this: - The xmlns:esi ends up on the <esi:include /> tag instead of the HTML root. Varnish doesn't like this apparently. - The <esi:include /> tag is not self-closing when rendered with the html.tostring (using etree.tostring is not really an option as other things are going on which want html rendering). Varnish likes this even less. Thus: <esi:include src="http://..." xmlns:esi="http://www.edge-delivery.org/esi/1.0"></esi:include> What can I do to push the namespace declaration up to the top node ('root') and make the tag self-closing? Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book

3 21

[lxml-dev] Custom element classes: lost my proxy
by Nathan Reynolds 08 Feb '10

08 Feb '10

Hi all, As soon as I insert my custom elements into a regular lxml.etree.Element, they revert to the standard Element interface. Is it possible to get my proxy back for these elements? Thanks, Nath

2 1