Mailman 3 May 2009 - lxml - The Python XML Toolkit

Re: [lxml-dev] problem about lxml encoding
by Stefan Behnel 31 May '09

31 May '09

Hi, please keep the list involved. Thanks. qhlonline wrote: > Since I am processing Chinese Webs. There are instances that some Webs > are not regular. When it said <meta encoding="GB2312">, we can't decode > the HTML string with GB2312 decoder, we found that this content is > encoded with GBK or GB18030 in fact. But the lxml parser will process it > according to the meta declaration. I have read the source of libxml2, > there are no "encoding" string match "GB2312","GBK" or "GB18030", but > some other encodings like ENCODING_2022_JP which may be a super set of > GB2312 I just don't know where in libxml2 or in lxml the <meta> declared > GB2312 encoding is converted to some other encodings that apparently > supported by libxml2's XML's xmlCharEncoding structure, and How? Most encodings are handled by libiconv. libxml2 only handles the 'normal' (or most common) XML encodings in its own code. > I think > since "GB18030" is a super set of "GB2312", if we change the lxml source > to let all <meta charset = 'GB2312'> strings were decoded with GB18030 > codec, then there will be no error even if the HTML file is not regular. No need to play with lxml's sources here. What should work is to read the HTML page into a byte string, decode it manually into a unicode string, and then let lxml parse that. That way, you also have full control over the decoding and can handle any decoding errors yourself. Stefan

2 1

Re: [lxml-dev] xpath on text nodes
by Stefan Behnel 31 May '09

31 May '09

Hi, Jamie Norrish wrote: > I've included at the end of this message an example of the XML I'm > operating over, where the aim is to get a rough number of characters of > textual content preceding and following a name or rs element. Given the > highly multiform nature of the markup, I *think* that the simplest way > of going about this is to go from text node to text node, forward and > back, accumulating the text as it goes, and stopping once a certain > amount has been reached. > > The way I'm currently doing this is by simply selecting a certain number > of text nodes preceding and following the name or rs element > (name_node.xpath('following::text()[position()<15]'), for example), and > iterating through those and stopping when the right amount of text has > been accumulated. Obviously this has the problem that too many or too > few text nodes (in the XPath result sense) may be selected, which is > either inefficient or leads to too little context. > > Selecting an ancestor and then splitting the textual content of that > isn't, I think, a better option, given the nature of the XML I'm dealing > with. A name/rs element may be at almost any level of the tree, and its > textual content may well be repeated multiple times within any given > chunk. Ok, I now see where you are coming from. Something like the above XPath expression or the respective lxml.etree API code would have been my first attempt, too. I actually doubt that you can do much better in this case. It's actually a more general problem. Imagine you select a text node that has a certain length and contains the found text multiple times. How would you find a good context here? Is it the context of the first occurrence, which may include a lot of preceding text but not the last occurrence within the text node itself (if it is long enough) - or is it the last occurrence that is interesting here, with all the text that follows the matching text node? So the underlying problem is even independent of the API you use, it's more that substrings do not match nicely with the granularity of a text node. > I totally understand that it's problematic to change lxml to have a > different model for text, and I'm either going to continue with my > current method, or else use a modified form of my ideal solution, which > is to get the parent element of the text, and then use XPath again to > get the appropriate next text node in the sequence from that. This is a > little more cumbersome than I'd like, obviously, since the expression > changes not just by the direction of the context (preceding or > following) but also whether the current text is the text or tail of the > element. I'd have to run some tests to see whether the extra processing > slowed things down too much - this process is one that operates over > (often) thousands of name elements within each of over a thousand > documents. Maybe you should try the same thing without XPath, just using the API. XPath is fast when you are very selective or when you grab the aggregated text content of an element. It's less great when you do things iteratively. The API based algorithm may not even be that complex as you can use tree iteration and stuff. (Did I mention that readability counts? :) > (The point of getting this context is to give people some idea of who a > name element might be referring to, for when it is being keyed to an > entity in our authority control system. So the markup doesn't matter > particularly, but the textual content does.) > >Stefan Behnel wrote: >> I still do not have a clear idea of what you consider "text context" >> actually. Does that take the tree structure into account (e.g. only within >> a certain parent element), or is it just any text content that precedes the >> XPath result in reverse document order, wherever it occurs in the tree? > > Just any, though there are some cases where the markup could be used to > usefully limit the context (so, for example, the name may occur within a > bibliographic entry in a list of citations, and it's unlikely that any > textual content from before or after that entry will be relevant. That's > typically going to be the exception, however; even staying within a > paragraph element is not necessarily helpful (named things are often > introduced at the end of a paragraph and given more context in the > following paragraph, for example). This sounds like your algorithm is already more complex than a simple "any text node preceding the one that matches". That convinces me that an API based solution will be a lot more flexible than anything you could scratch out of XPath. It would allow you to special case certain tag types, for example, or to notice when you cross parent boundaries. > Here's the example of a small piece of a document, in case it helps. I'll leave it in the reply, just in case others have ideas, too. > But really, I'm happy enough with the way lxml works (it's great software - > thank you and everyone else who has made it what it is!). Not being > familiar with its inner workings I didn't know whether it would be > feasible or practical to add XPath to text results. Now I know, and I'll > continue on without complaint. :) Stefan > <lb/>give my love to everybody including <name > key="name-110011" type="person">Peter</name>, hoping he is > <lb/>finding his way around the house better now, & that > this > <lb/> > <pb xml:id="n12" n="12" corresp="#JCB-001l"/> > finds you as it leaves me, in the best of health & very > <lb/>much in love with you. > </p> > <closer> > <salute><choice><abbr>Yr</abbr><expan>Your</expan></choice> > <choice><abbr>affect.</abbr><expan>affectionate</expan></choice> son > </salute> > <lb/> > <signed> > <name key="name-207379" type="person">J.C. Ulysses > Beaglehole</name> > </signed> > <seg type="postscript">P.S. You might tell yourself, <name > key="name-110417" type="person">Auntie</name> & <name > key="name-034628" type="person">Christine</name>, that > <lb/>I have struck nobody yet with so swish a > > <choice><orig>dressing-<lb/>gown</orig><reg>dressing-gown</reg></choice> > as mine. > <lb/>I had now better get on to some other letters > <lb/>of thanks, greeting, business, etc.</seg> > <signed><name key="name-207379" > type="person">J.</name></signed> > <seg type="postscript">P.P.S. You might send me the date of > Auntie <unclear>Sis'</unclear> > <lb/>birthday. I hope Auntie's had a fitting celebration.</seg> > <signed><name key="name-207379" type="person">J.</name> > <lb/> > </signed> > <seg type="postscript">P.P.P.S. I have been writing all the > morning & it is now > <lb/>¼ to 1. If you pass the letter round it will save > <lb/>much exhaustion to my dexter hand.</seg> > <salute> > <choice><abbr>Yrs</abbr><expan>Yours</expan></choice> > <del>finally</del> penultimately, > <lb/> > </salute> > <signed><name key="name-207379" > type="person">J.C.B.</name></signed>

2 1

[lxml-dev] problem about lxml encoding
by qhlonline 31 May '09

31 May '09

Hi, all I have a question about lxml encoding and html encoding. What is the relationship between the encoding which lxml parser will be using and the encoding of an HTML file? if the HTML file encoding is not the same with it's <meta charset="some encoding">, What the choice of lxml's parser will be? the 'charset' encoding or the HTML file coding?

2 1

[lxml-dev] UTF-8 not supported
by Ovnicraft 26 May '09

26 May '09

Hi folks, when do this, encoding='iso-8859-1' write xml enconding ok, but when try the same thing with 'UTF-8', not appears in my file, i have 2.2 version. How i can encoding my file with utf-8? regards, -- http://twitter.com/ovnicraft

3 7

[lxml-dev] lxml 2.2 validation question
by James Slagle 20 May '09

20 May '09

Hello, I'm having some trouble getting lxml (v. 2.2) to validate an ElementTree object that I'm building and was hoping someone on the list could help and maybe tell me what I'm doing wrong. If I create an ElementTree object directly from xml and an associated schema, it will validate fine. If I then construct a similar ElementTree object by just instantianting ElementTree, it will not validate. The odd thing is that the resulting xml from etree.tostring for both objects is identical. I've attached a python script that shows the problem I'm having. The validation error is: *** DocumentInvalid: Element 'Foo': No matching global declaration available for the validation root. I can get the second ElementTree object (etree2) to validate if I put the long explicit namesplace in front of the tag value (Foo) when I create etree2 in the script. So, if I change line 25 in the script to: rootelem = etree.Element('{http://example.com}Foo', {}, nsmap) , it will validate. However, the 2 resulting xml outputs are no longer equal b/c the output from etree2 is output with explict namespaces. Any ideas? Thanks for any help. -- James Slagle

4 8

[lxml-dev] could not find doctest
by Mary Lei 20 May '09

20 May '09

I copied the following from the lxml document on the web: #! /stage/irsa-sw-dev/cm/env_1.4_mnt/python/bin/python import urllib import urllib2 import urlparse import os import popen2 import StringIO import sys, getopt import re import string import doctest version = sys.version_info print "python version: ", version print os.environ['PYTHONPATH'] ## test with lxml import lxml.html from lxml.html import builder as E from lxml.html import usedoctest html = E.HTML( E.HEAD( E.LINK(rel="stylesheet", href="great.css", type="text/css"), E.TITLE("Best Page Ever") ), E.BODY( E.H1(E.CLASS("heading"), "Top News"), E.P("World News only on this page", style="font-size: 200%"), "Ah, and here's some more text, by the way.", lxml.html.fromstring("<p> and this is a parsed fragment </p>") ) ) print lxml.html.tostring(html) And got the following error: python version: (2, 5, 1, 'final', 0) .:/home/lei/python-stuff/BeautifulSoup-3.1.0.1:./lib:/home/lei/lxml-2.2/src Traceback (most recent call last): File "/home/lei/python-stuff/test_lxml_htmlout.py", line 22, in <module> from lxml.html import usedoctest File "/home/lei/lxml-2.2/src/lxml/html/usedoctest.py", line 13, in <module> doctestcompare.temp_install(html=True, del_module=__name__) File "/home/lei/lxml-2.2/src/lxml/doctestcompare.py", line 395, in temp_install frame = _find_doctest_frame() File "/home/lei/lxml-2.2/src/lxml/doctestcompare.py", line 486, in _find_doctest_frame "Could not find doctest (only use this function *inside* a doctest)") LookupError: Could not find doctest (only use this function *inside* a doctest) Any clues to this problem ? -- Mary Lei Software Testing IPAC-NExScl Rm: KS-233 MS: 220-6 Phone: 395-1998

2 2

[lxml-dev] parse unable to load url
by Mary Lei 20 May '09

20 May '09

I copied sample code from lxml documentation #! /stage/irsa-sw-dev/cm/env_1.4_mnt/python/bin/python import urllib import urllib2 import urlparse import os import popen2 import StringIO import sys, getopt import re import string version = sys.version_info print "python version: ", version print os.environ['PYTHONPATH'] ## test with lxml from lxml import etree from lxml.html import fromstring, tostring, parse, submit_form #page = parse('http://tinyurl.com').getroot() page = parse('/home/lei/python-stuff/tmp_CoRoT_exo_index.html').getroot() print page page = parse('http://tinyurl.com').getroot() And got errors when parsing url, local file is ok dodo:lei > test_lxml.py python version: (2, 5, 1, 'final', 0) .:/home/lei/python-stuff/BeautifulSoup-3.1.0.1:./lib:/home/lei/lxml-2.2/src <Element html at 3f9a40> Traceback (most recent call last): File "test_lxml.py", line 27, in <module> page = parse('http://tinyurl.com').getroot() File "/home/lei/lxml-2.2/src/lxml/html/__init__.py", line 661, in parse return etree.parse(filename_or_url, parser, base_url=base_url, **kw) File "lxml.etree.pyx", line 2693, in lxml.etree.parse (src/lxml/lxml.etree.c:52591) File "parser.pxi", line 1478, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:75665) File "parser.pxi", line 1507, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:75993) File "parser.pxi", line 1407, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:75002) File "parser.pxi", line 965, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:72023) File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:67830) File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:68877) File "parser.pxi", line 563, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:68093) IOError: Error reading file 'http://tinyurl.com': failed to load HTTP resource -- Mary Lei Software Testing IPAC-NExScl Rm: KS-233 MS: 220-6 Phone: 395-1998

2 2

Re: [lxml-dev] Ask for help about lxml usage
by Stefan Behnel 13 May '09

13 May '09

qhlonline wrote: > My code of multi-thread parsing test is as follows, May be there are some > problems, and I am very glad to get your suggestion. I have created eight > threads in this program and they collaborate to parse a single HTML file > 1000 times, the result I want is the time used. > from lxml import etree > import time > > import threading > class TargetParser(object): > # The Target Parser Def: > def __init__(self): > self.ImgTag = 0 > self.StyleTag = 0 > self.ScriptTag= 0 > def start(self, tag, attrib): > if tag == 'img': > self.ImgTag = self.ImgTag+1 > elif tag == 'style': > self.StyleTag = self.StyleTag+1 > elif tag == 'script': > self.ScriptTag = self.ScriptTag+1 > else: > pass > def end(self, tag): > pass > def close(self): > return self I'd strip this all together and go with separate parse/search phases. > class MultiThread: > def __init__(self): > self.circle = 0 > self.timeres = 0.0 > > def CircleParse(self, webpath, circles=1000): > self.webpath=webpath > self.circles=circles > self.lock=thread.allocate_lock() > self.lock2=thread.allocate_lock() > starttime = time.time() > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) I'd use a for-loop here. > self.lock2.acquire() > self.lock2.acquire() This bothers me. Not only that it appears twice, but also that you need an explicit lock in your program. > timeres = time.time()-starttime # total time > print 'In MultiThread Parser, the time Consume is : ',timeres, ' > Seconds!\n' > > def NewParse(self): > > MyParser = TargetParser() > Parser=etree.HTMLParser(target=MyParser) #The parser used in this > thread A general note on naming in Python programs: you should skim through PEP 8. Your code is rather hard to read as it uses AllCamelCase names for class names, method names and some local variables, i.e. things that are semantically very different. It's common to use it for class names, but everything else should use lower-case separated by underscores. > while (self.circle < self.circles): > if(self.circle >= self.circles): > break > > res = etree.parse(self.webpath,Parser) > > if(self.lock.acquire()): > self.circle = self.circle + 1 > if(self.circle >= self.circles and self.lock2.locked()): > > self.lock2.release() > break > self.lock.release() Ok, my take would be this (although it's completely(!) untested): from Queue import Queue from threading import Thread from lxml import etree tag_counters = [ (tagname, etree.XPath('count(//%s)' % tagname)) for tagname in ('img', 'style', 'script') ] def start_threads(func, thread_count, *args): for _ in range(thread_count): thread = Thread(target=func, args=args) thread.setDaemon() thread.start() def handle_urls(url_queue, result_queue): parser = etree.HTMLParser() while True: # I'm a deamon, so I don't care try: url = url_queue.get() doc = etree.parse(url, parser) result = [ (tagname, count(doc)) for tagname, count in tag_counters ] result_queue.put(result) doc = None # free space while we wait except Exception, e: # catch-all to make sure we report all 'normal' exceptions result_queue.put(e) e = None # free space while we wait # run benchmark in_queue = Queue() out_queue = Queue() start_threads(handle_urls, 10, in_queue, out_queue) from time import time t = time() for _ in range(100): in_queue.put("file://tmp/somefile.html") for _ in range(100): print out_queue.get() print time() - t ... minus some glitches, but I bet you can fix them and post a better version. Have fun, Stefan

1 0

Re: [lxml-dev] Ask for help about lxml usage
by Stefan Behnel 13 May '09

13 May '09

[forwarding this to the list] qhlonline wrote: > Thank you very much for your kindly help. I have tried to trace the lxml > source code. I have attempted to find out whether the lxml is simply a > python-wrapped interface for HTML/XML parsing. if so, after my program > have reached the C/Python interface(eg. xmlDoc* htmlCtxtReadMemory in > htmlparser.pxd), the following job should have run parallelled on > different CPU cores, becausing it was using libxml2. I have also naively > thought that the elimination of DOM tree creation process would save time, > but it seems that this job is done by SAX event processers of libxml2 if > we are using the default parser. It's not naive, the thing is just that this is a lot more efficient for single-threaded programs than for multi-threaded ones. That's not an obvious difference, and it's also not advertised in the docs. Most users don't care too much about the exact high performance characteristics and it's not like there's an obvious bottleneck in lxml.etree that you have to warn people about. It's more the general "Python code is single-threaded" kind of thing and it all depends on how you use it, so you have to benchmark your own code anyway. > I don't know the Cython language, some > new types and declarations of variables and functions in .pxi files and > .pxd files makes me feel headache. You can ignore most of the little stuff that you don't understand. The main idea is just that you can switch freely between C and Python in the code, so depending on how firm you are in both, some code sections may be less obvious than others. > but my instructor said that since the > multi-thread parsing program with lxml can save time on two-core CPU > machine(It is true, it can run nearly 20% faseter with 2 thread on > two-core CPU machine), it should run better with more thread on a better > machine. now, with your help, I know maybe multi-process program will do > better. thank you again! Usually a lot faster, as you avoid any implicit concurrency issues. In multi-process programs, synchronisation only happens where you explicitly do it. Things are a lot more implicit and subtle in multi-threading. I'll comment on your program in your other mail. Please keep the list involved next time. Stefan

1 0

[lxml-dev] ElementTree.write default_namespace argument
by Richard Lewis 12 May '09

12 May '09

Hi there, I have lxml version 2.1.5, Python 2.5 both installed from Debian packages. I'd like to be able to use the default_namespace argument to ElementTree's write method. However, I get the following error: Traceback (most recent call last): File "./join-html-pages.py", line 62, in <module> main() File "./join-html-pages.py", line 59, in main document.write(file(new_filename, 'w'), encoding='utf-8', method='html', default_namespace='html') File "lxml.etree.pyx", line 1576, in lxml.etree._ElementTree.write (src/lxml/lxml.etree.c:16699) TypeError: write() got an unexpected keyword argument 'default_namespace' Is this expected behaviour? My application is extracting nodes from numerous XHTML documents and merging them into one HTML document. The bit that's going wrong is writing the resultant HTML to a file. I'm not really interested in it being correct XHTML or in using the correct namespace. So any alternative solutions would also be of interest. Cheers, Richard

2 1