Mailman 3 December 2010 - lxml - The Python XML Toolkit

Re: [lxml-dev] xpath: retrieve attribute names
by jholg＠gmx.de 29 Dec '10

29 Dec '10

(cc-ing the list) Hi, > hi, > maybe not too convenient but this works: > > >>> x = etree.XML('<a test="1" a2="2"/>') > >>> x.xpath('@*') > ['1', '2'] > >>> x.xpath('name(@*)') > 'test' > >>> x.xpath('name(@*[1])') > 'test' > >>> x.xpath('name(@*[2])') > 'a2' As I don't know the number of attributes up front that'll lead to a solution similar to what I posted originally: > >>>> [ (elt.xpath('name(@*[$i])', i=i+1), elt.xpath('@*[$i]', i=i+1)[0]) > for i in range(elt.xpath('count(@*)')) ] > > [('py:pytype', 'int'), ('foo', 'bar'), ('doo', 'woop')] > > Best regards, Holger -- Neu: GMX De-Mail - Einfach wie E-Mail, sicher wie ein Brief! Jetzt De-Mail-Adresse reservieren: http://portal.gmx.net/de/go/demail

1 0

[lxml-dev] lxml.html text_content() returns html for escaped html?
by Art Gibson 21 Dec '10

21 Dec '10

In the code snippet below, the "text_content{}" call returns html instead of the escaped html. Is this the desired behaviour? from lxml.html import fromstring code_block = '<code><a href="" title=""></code>' doc = fromstring(code_block) doc.text_content() #returns <a href="" title=""> , instead of the escaped version Thanks for any help, Art

2 1

Re: [lxml-dev] Using cssselect with XHTMLParser
by Henrik 17 Dec '10

17 Dec '10

On 14 dec 2010, at 15:59, Stefan Behnel wrote: > Henrik, 14.12.2010 14:18: >> I have a problem with using cssselect method on my xhtml elements. >> >> xhtml = u"""<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> >> <html xmlns="http://www.w3.org/1999/xhtml"> >> <head> >> <title>Foo</title> >> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> >> <meta http-equiv="Content-Language" content="sv-se" /> >> <meta http-equiv="imagetoolbar" content="no" /> >> <meta name="MSSmartTagsPreventParsing" content="true" /> >> </head> >> <body id="start"> >> <div id="wrap"> >> <div id="site"> >> <div id="header"> >> <h1>Header</h1> >> <h2>Header 2</h2> >> <img id="logotype" src="/site/images/baker-tilly-sydost.jpg" width="210" height="64" alt="Baker Tilly Sydost" /> >> </div> >> <p>Test©</p> >> </div> >> </div> >> </body> >> </html> >> """ >> >> parser = lxml.html.XHTMLParser(recover=True) >> soup = lxml.html.fromstring(xhtml, parser=parser) >> h1s = soup.body.cssselect("h1") >> print h1s " => This returns an empty list [] > > CSSSelect is namespace aware. Since XHTML documents live in a namespace, you need to use fully qualified tag names. > > http://codespeak.net/lxml/cssselect.html#namespaces > > Alternatively, use the xhtml_to_html() function in lxml.html to remove the namespaces from the tag names in your document. > > Stefan Hello, I've been using the xhtml_to_html() to resolve some of my issues but I can figure out how to write a cssselect() query with a namespace-prefix? How should I write to find the <h1> tag in a xhtml document without using xhtml_to_html()? I've tried the full cssselect('http://www.w3.org/1999/xhtml | h1') but nothing works :) Can I set this default namespace as a class variable or something? Cheers, Henrik

1 0

[lxml-dev] Using cssselect with XHTMLParser
by Henrik 14 Dec '10

14 Dec '10

Hello list, I have a problem with using cssselect method on my xhtml elements. I.e xhtml = u"""<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title>Foo</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <meta http-equiv="Content-Language" content="sv-se" /> <meta http-equiv="imagetoolbar" content="no" /> <meta name="MSSmartTagsPreventParsing" content="true" /> </head> <body id="start"> <div id="wrap"> <div id="site"> <div id="header"> <h1>Header</h1> <h2>Header 2</h2> <img id="logotype" src="/site/images/baker-tilly-sydost.jpg" width="210" height="64" alt="Baker Tilly Sydost" /> </div> <p>Test ©</p> </div> </div> </body> </html> """ parser = lxml.html.XHTMLParser(recover=True) soup = lxml.html.fromstring(xhtml, parser=parser) h1s = soup.body.cssselect("h1") print h1s " => This returns an empty list [] Changing from XHTMLParser(recover=True) to HTMLParser(recover=True) Makes the code work again and h1s returns a list with an Element. Who do I get the cssselect to work when parsing with XHTMLParser? Cheers, Henrik

2 1

[lxml-dev] Test coverage and profiling for XSL(T) code
by Christian Heimes 10 Dec '10

10 Dec '10

Hello! We are using Ned Batchelder's excellent coverage tool [1] to measure the code coverage of our unit test suite. Now I'm looking for a way to get a coverage report for our large XSL(T) code base. Is there any way to see which lines of a XSL file are executed with lxml? The word "coverage" is mentioned at [2] for version 2.7.0 but I can't find any additional information. In a slightly related matter, how can I profile the speed of our templates? I like to search for hot spots and optimize them. In my experience with Python optimization, only a profiler is able to detect hot spots during runtime. Now I like to do the same with XSL code. Thanks, Christian [1] http://pypi.python.org/pypi/coverage [2] http://xmlsoft.org/xml.html

3 2

Re: [lxml-dev] URL extraction from HTML documents
by Dimitrios Pritsos 02 Dec '10

02 Dec '10

On 02/12/10 20:49, Graff, Marc wrote: > I have only leveraged lxml for one project so I am no expert. If your > code is small enough and you are not prohibited from doing so, can you > provide it so others can understand what you are doing and how to make a > better determination of the memory leak you encountered and possible > cause? > > Thanks. I think that I sent a sample case, as a response to Stefan's sample code for memory leakage replication. regards Dimitrios > -----Original Message----- > From: lxml-dev-bounces(a)codespeak.net > [mailto:lxml-dev-bounces@codespeak.net] On Behalf Of Dimitrios Pritsos > Sent: Thursday, December 02, 2010 10:35 AM > To: lxml-dev(a)codespeak.net > Subject: Re: [lxml-dev] URL extraction from HTML documents > > On 12/02/2010 02:53 PM, Stefan Behnel wrote: > > > On 12/02/2010 02:53 PM, Stefan Behnel wrote: >> Dimitrios Pritsos, 02.12.2010 12:17: >>> module re (regular expression) is just fine for >>> URL extraction, however I would prefer the use of XPath for > extracting a >>> variate of links more easily in Coding point of view. >> Sure, lxml.html has specific support for extracting URLs from parsed >> documents. >> > But what about the Memory Leakage, I am sorry if there is a solution > already. However, I believe that this is not intuitive at all (I mean > the all tree to stay in Memory like a Garbage and not to be replaced). I > > don't think that I am experienced enough to fix this. > >>> Plus I think that the overhead of Tree Building is not so much (I > dont >>> know for sure thought). >> Likely slower than re, but also likely fast enough. >> >> >>> Speaking of XPath for url extraction, I think that lxml.html has some >>> issues in url extraction (this is what I think reading the Code of > this >>> module). >> Such as ... ? >> > I just think it is harder to have all the definition of HTML 4.0 (XTHML > 1.0 , 1.1 e.t.c.) and have the code up-to-date. XPath (I think) it will > be more general. just that :) >>> And the question is why not to use the XPath for making the >>> code twice smaller and twice neater (I cleaner and well formed - I > hope >>> my vocabulary is correct), maybe faster too. >> Maybe. If you want to provide a patch that simplifies the code and >> back it with sufficient evidence that it's at least as fast as before >> and doesn't degrade functionality, I'll be happy to accept it. > As for lxml.html I think I can send something Xpath based and to be > multiprocessing/multi-threaded too (optionally). But still need some > work and I don't have enough time to finish it right now, because this > is a critical phase for my PhD and Job. >> Stefan > Thank you very much for your fast response! > > Dimitrios > > _______________________________________________ > lxml-dev mailing list > lxml-dev(a)codespeak.net > http://codespeak.net/mailman/listinfo/lxml-dev >

1 0

Re: [lxml-dev] URL extraction from HTML documents
by Graff, Marc 02 Dec '10

02 Dec '10

I have only leveraged lxml for one project so I am no expert. If your code is small enough and you are not prohibited from doing so, can you provide it so others can understand what you are doing and how to make a better determination of the memory leak you encountered and possible cause? Thanks. -----Original Message----- From: lxml-dev-bounces(a)codespeak.net [mailto:lxml-dev-bounces@codespeak.net] On Behalf Of Dimitrios Pritsos Sent: Thursday, December 02, 2010 10:35 AM To: lxml-dev(a)codespeak.net Subject: Re: [lxml-dev] URL extraction from HTML documents On 12/02/2010 02:53 PM, Stefan Behnel wrote: On 12/02/2010 02:53 PM, Stefan Behnel wrote: > Dimitrios Pritsos, 02.12.2010 12:17: >> module re (regular expression) is just fine for >> URL extraction, however I would prefer the use of XPath for extracting a >> variate of links more easily in Coding point of view. > > Sure, lxml.html has specific support for extracting URLs from parsed > documents. > But what about the Memory Leakage, I am sorry if there is a solution already. However, I believe that this is not intuitive at all (I mean the all tree to stay in Memory like a Garbage and not to be replaced). I don't think that I am experienced enough to fix this. > >> Plus I think that the overhead of Tree Building is not so much (I dont >> know for sure thought). > > Likely slower than re, but also likely fast enough. > > >> Speaking of XPath for url extraction, I think that lxml.html has some >> issues in url extraction (this is what I think reading the Code of this >> module). > > Such as ... ? > I just think it is harder to have all the definition of HTML 4.0 (XTHML 1.0 , 1.1 e.t.c.) and have the code up-to-date. XPath (I think) it will be more general. just that :) > >> And the question is why not to use the XPath for making the >> code twice smaller and twice neater (I cleaner and well formed - I hope >> my vocabulary is correct), maybe faster too. > > Maybe. If you want to provide a patch that simplifies the code and > back it with sufficient evidence that it's at least as fast as before > and doesn't degrade functionality, I'll be happy to accept it. As for lxml.html I think I can send something Xpath based and to be multiprocessing/multi-threaded too (optionally). But still need some work and I don't have enough time to finish it right now, because this is a critical phase for my PhD and Job. > > Stefan Thank you very much for your fast response! Dimitrios _______________________________________________ lxml-dev mailing list lxml-dev(a)codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev

1 0

Re: [lxml-dev] How to register extension functions for XSL transformations
by Shaung 02 Dec '10

02 Dec '10

On Thu, Dec 2, 2010 at 6:35 PM, <jholg(a)gmx.de> wrote: > Hi, > > You're confusing namespaces and namespace prefixes. lxml generally uses namespaces, not prefixes, which is The Right Way. This should help you on: > > ext_mod = MyControlAdapter() > funcs = ('GetDescFor',) > exts = etree.Extension(ext_mod, funcs, ns='urn:MyControlAdapter') > transform = etree.XSLT(xslt_doc, extensions=exts) > rslt = transform(input_doc) > > Holger > This solved my problem. Thanks a lot! Shaung

1 0

[lxml-dev] How to register extension functions for XSL transformations
by Shaung 02 Dec '10

02 Dec '10

Hello all, I am trying to convert a xml document into html. Input: <MyControl type="Text" caption="#SomeVar"/> Expected output: <span> SomeVar's Description </span> The xsl file I am using: <?xml version='1.0'?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:myctl="urn:MyControlAdapter"> <xsl:output method ="html"/> <xsl:template match="/" > <xsl:apply-templates select="/MyControl"/> </xsl:template> <xsl:template match="MyControl"> <xsl:choose> <xsl:when test="@type = 'Text'"> <xsl:call-template name="RenderText"/> </xsl:when> <xsl:otherwise> <span>nothing</span> </xsl:otherwise> </xsl:choose> </xsl:template> <xsl:template name="RenderText"> <span> <xsl:value-of select="myctl:GetDescFor(@caption)"/> </span> </xsl:template> </xsl:stylesheet> and in python: class MyControlAdapter: def GetDescFor(self, _, name): return foo.get_variable(name).get_desc() # foo is the module holding a list of variables ext_mod = MyControlAdapter() funcs = ('GetDescFor',) exts = etree.Extension(ext_mod, funcs, ns='myctl') transform = etree.XSLT(xslt_doc, extensions=exts) rslt = transform(input_doc) But always got an exception lxml.etree.XSLTApplyError: XPath evaluation returned no result. So am I doing it wrong? If so how to get it done without modifying the xslt contents? I am pretty new to xlst, so I might be totally thinking it wrong but just have too little knowledge to know how wrong I am... Any help would be greatly appreciated. (Sorry for my English) Shaung

2 1