Mailman 3 May 2009 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 11 Apr '23

11 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 10 Apr '23

10 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

[lxml-dev] docinfo.doctype doesn't include internal entities?
by Sidnei da Silva 24 Jul '09

24 Jul '09

Hi there, I am looking for a way to output internal entities that have been parsed from the original document when writing out a tree, but apparently this is not exposed in any attribute. Here's an example: {{{ import lxml.etree document = """<?xml version="1.0"?> <!DOCTYPE application [ <!ENTITY nbsp "\ "> ]> <application> </application> """ tree = lxml.etree.fromstring(document) print tree.getroottree().docinfo.doctype }}} I would expect this to output: {{{ <!DOCTYPE application [ <!ENTITY nbsp "\ "> ]> }}} But instead it gives me: {{{ <!DOCTYPE application> }}} Is it a bug or I'm not looking at the right place? -- Sidnei da Silva

3 2

[lxml-dev] Compilation of lxml on Windows 64-bit
by Hanni Ali 17 Jul '09

17 Jul '09

Hi All, Came across this little gem of a Python module which seems able to translate some XML reports for me nicely. I was very pleased to see a Python 2.6 installer for windows, made checking it out much easier than many modules I use. So thanks for that. However although my testing seems to confirm this is the ideal module I do need to deploy to our production environment which is a 64-bit Windows environment running Python 2.6 (I code on 32-bit box for simplicity/resource reasons). Has anyone compiled and used lxml on this platform combination? If not does anyone foresee me having any issues compiling it for this platform? Thanks, Hanni

2 7

[lxml-dev] Binary egg for Mac OS X
by Alexander Limi 15 Jun '09

15 Jun '09

Hi, I'm working on documenting Deliverance / xdv for use with Plone. It has lxml as a dependency, and I have run into a serious issue: On Mac OS X, we can't assume that people have Xcode (ie. gcc and friends) installed, thus we can't really compile lxml on those computers, not even using the staticlxml[1] recipe. I see that there are binary eggs for Windows, is there a special reason why there are no binary eggs for OS X, or is it just a matter of not having the infrastructure to make it available? Happy to help find a solution if it's just a matter of locating a reliable way to get it compiled every time there is a new release. [1] http://pypi.python.org/pypi/z3c.recipe.staticlxml/ -- Alexander Limi · http://limi.net

6 11

[lxml-dev] Possible bug, libxml segfault?
by Avleen Vig 04 Jun '09

04 Jun '09

Hi folks :) I sent this message to the xml@gnome list, but that place is seems very quiet lately. I'm hoping someone here might have some insight. Indeed this might be a bug in lxml, we're not sure. Background: We use libxml and libxslt in one of our applications (specifically, in Python via lxml). Recently we've seen our application dying at strange times for no apparent reason. We managed to get a core file out of one crash, and the results of some of our debugging are here: http://xml.pastebin.com/m70c259d6 (I'd be happy to poke more in a particular direction on there, I'm a bit new to gdb :) To me, it seems the parser is complaining while trying to parse the namespaces in the <stylesheet> node in transforms/_base.xslt The node for that opens like this: <xsl:stylesheet version="1.0" xmlns="http://www.w3.org/1999/xhtml"; xmlns:xsl="http://www.w3.org/1999/XSL/Transform"; xmlns:tfxslt="http://tfnet.co.uk/ns/tfxslt"; xmlns:fb="http://www.facebook.com/2008/fbml"; extension-element-prefixes="tfxslt str exsl" exclude-result-prefixes="str exsl tfxslt fb" xmlns:error="http://www.woome.com/error/";> I dug a little deeper and found a bunch of the "address out of bounds" errors and thought I should ask here as I'm drawing a blank on where to go next. The problem happens intermittently, but usually several times a day. I could probably reproduce it. I also see the 'exclude-result-prefixes' mentioned in the backtrace, could that be involved here? Any suggestions you have would be much appreciated!

2 3

[lxml-dev] XML Documents & I18N (the way Cocoon does it)
by Alexis Georges 01 Jun '09

01 Jun '09

Hello everyone, I am maintaining a multilingual website which works with XML, XSLT to generate XHTML. I am working with Apache Cocoon (http://cocoon.apache.org/2.1/) using (among other things) their I18NTransformer. Basically I can use elements in the I18N (http://apache.org/cocoon/i18n/2.1) namespace, and then tell Cocoon to apply the I18NTransfomer to the document; this replaces the I18N elements with a localized value (eg. a formatted date/number, a translated label/attribute, etc...). I have been looking at lxml a little bit to see if I could move to a Python-based framework for the website. I am not quite sure how to go about the I18N part though. Using the Babel library (http://babel.edgewall.org/) along with request headers to generate localized data, I have everything I need. What is missing is the "parser" for the I18N elements. All I can think of right now is to implement a SAX parser, the way Cocoon does (in Java). Does anyone have suggestions? Am I making this too complicated? Thanks! Alexis

2 2

Re: [lxml-dev] problem about lxml encoding
by Stefan Behnel 31 May '09

31 May '09

Hi, please keep the list involved. Thanks. qhlonline wrote: > Since I am processing Chinese Webs. There are instances that some Webs > are not regular. When it said <meta encoding="GB2312">, we can't decode > the HTML string with GB2312 decoder, we found that this content is > encoded with GBK or GB18030 in fact. But the lxml parser will process it > according to the meta declaration. I have read the source of libxml2, > there are no "encoding" string match "GB2312","GBK" or "GB18030", but > some other encodings like ENCODING_2022_JP which may be a super set of > GB2312 I just don't know where in libxml2 or in lxml the <meta> declared > GB2312 encoding is converted to some other encodings that apparently > supported by libxml2's XML's xmlCharEncoding structure, and How? Most encodings are handled by libiconv. libxml2 only handles the 'normal' (or most common) XML encodings in its own code. > I think > since "GB18030" is a super set of "GB2312", if we change the lxml source > to let all <meta charset = 'GB2312'> strings were decoded with GB18030 > codec, then there will be no error even if the HTML file is not regular. No need to play with lxml's sources here. What should work is to read the HTML page into a byte string, decode it manually into a unicode string, and then let lxml parse that. That way, you also have full control over the decoding and can handle any decoding errors yourself. Stefan

2 1

Re: [lxml-dev] xpath on text nodes
by Stefan Behnel 31 May '09

31 May '09

Hi, Jamie Norrish wrote: > I've included at the end of this message an example of the XML I'm > operating over, where the aim is to get a rough number of characters of > textual content preceding and following a name or rs element. Given the > highly multiform nature of the markup, I *think* that the simplest way > of going about this is to go from text node to text node, forward and > back, accumulating the text as it goes, and stopping once a certain > amount has been reached. > > The way I'm currently doing this is by simply selecting a certain number > of text nodes preceding and following the name or rs element > (name_node.xpath('following::text()[position()<15]'), for example), and > iterating through those and stopping when the right amount of text has > been accumulated. Obviously this has the problem that too many or too > few text nodes (in the XPath result sense) may be selected, which is > either inefficient or leads to too little context. > > Selecting an ancestor and then splitting the textual content of that > isn't, I think, a better option, given the nature of the XML I'm dealing > with. A name/rs element may be at almost any level of the tree, and its > textual content may well be repeated multiple times within any given > chunk. Ok, I now see where you are coming from. Something like the above XPath expression or the respective lxml.etree API code would have been my first attempt, too. I actually doubt that you can do much better in this case. It's actually a more general problem. Imagine you select a text node that has a certain length and contains the found text multiple times. How would you find a good context here? Is it the context of the first occurrence, which may include a lot of preceding text but not the last occurrence within the text node itself (if it is long enough) - or is it the last occurrence that is interesting here, with all the text that follows the matching text node? So the underlying problem is even independent of the API you use, it's more that substrings do not match nicely with the granularity of a text node. > I totally understand that it's problematic to change lxml to have a > different model for text, and I'm either going to continue with my > current method, or else use a modified form of my ideal solution, which > is to get the parent element of the text, and then use XPath again to > get the appropriate next text node in the sequence from that. This is a > little more cumbersome than I'd like, obviously, since the expression > changes not just by the direction of the context (preceding or > following) but also whether the current text is the text or tail of the > element. I'd have to run some tests to see whether the extra processing > slowed things down too much - this process is one that operates over > (often) thousands of name elements within each of over a thousand > documents. Maybe you should try the same thing without XPath, just using the API. XPath is fast when you are very selective or when you grab the aggregated text content of an element. It's less great when you do things iteratively. The API based algorithm may not even be that complex as you can use tree iteration and stuff. (Did I mention that readability counts? :) > (The point of getting this context is to give people some idea of who a > name element might be referring to, for when it is being keyed to an > entity in our authority control system. So the markup doesn't matter > particularly, but the textual content does.) > >Stefan Behnel wrote: >> I still do not have a clear idea of what you consider "text context" >> actually. Does that take the tree structure into account (e.g. only within >> a certain parent element), or is it just any text content that precedes the >> XPath result in reverse document order, wherever it occurs in the tree? > > Just any, though there are some cases where the markup could be used to > usefully limit the context (so, for example, the name may occur within a > bibliographic entry in a list of citations, and it's unlikely that any > textual content from before or after that entry will be relevant. That's > typically going to be the exception, however; even staying within a > paragraph element is not necessarily helpful (named things are often > introduced at the end of a paragraph and given more context in the > following paragraph, for example). This sounds like your algorithm is already more complex than a simple "any text node preceding the one that matches". That convinces me that an API based solution will be a lot more flexible than anything you could scratch out of XPath. It would allow you to special case certain tag types, for example, or to notice when you cross parent boundaries. > Here's the example of a small piece of a document, in case it helps. I'll leave it in the reply, just in case others have ideas, too. > But really, I'm happy enough with the way lxml works (it's great software - > thank you and everyone else who has made it what it is!). Not being > familiar with its inner workings I didn't know whether it would be > feasible or practical to add XPath to text results. Now I know, and I'll > continue on without complaint. :) Stefan > <lb/>give my love to everybody including <name > key="name-110011" type="person">Peter</name>, hoping he is > <lb/>finding his way around the house better now, & that > this > <lb/> > <pb xml:id="n12" n="12" corresp="#JCB-001l"/> > finds you as it leaves me, in the best of health & very > <lb/>much in love with you. > </p> > <closer> > <salute><choice><abbr>Yr</abbr><expan>Your</expan></choice> > <choice><abbr>affect.</abbr><expan>affectionate</expan></choice> son > </salute> > <lb/> > <signed> > <name key="name-207379" type="person">J.C. Ulysses > Beaglehole</name> > </signed> > <seg type="postscript">P.S. You might tell yourself, <name > key="name-110417" type="person">Auntie</name> & <name > key="name-034628" type="person">Christine</name>, that > <lb/>I have struck nobody yet with so swish a > > <choice><orig>dressing-<lb/>gown</orig><reg>dressing-gown</reg></choice> > as mine. > <lb/>I had now better get on to some other letters > <lb/>of thanks, greeting, business, etc.</seg> > <signed><name key="name-207379" > type="person">J.</name></signed> > <seg type="postscript">P.P.S. You might send me the date of > Auntie <unclear>Sis'</unclear> > <lb/>birthday. I hope Auntie's had a fitting celebration.</seg> > <signed><name key="name-207379" type="person">J.</name> > <lb/> > </signed> > <seg type="postscript">P.P.P.S. I have been writing all the > morning & it is now > <lb/>¼ to 1. If you pass the letter round it will save > <lb/>much exhaustion to my dexter hand.</seg> > <salute> > <choice><abbr>Yrs</abbr><expan>Yours</expan></choice> > <del>finally</del> penultimately, > <lb/> > </salute> > <signed><name key="name-207379" > type="person">J.C.B.</name></signed>

2 1

[lxml-dev] problem about lxml encoding
by qhlonline 31 May '09

31 May '09

Hi, all I have a question about lxml encoding and html encoding. What is the relationship between the encoding which lxml parser will be using and the encoding of an HTML file? if the HTML file encoding is not the same with it's <meta charset="some encoding">, What the choice of lxml's parser will be? the 'charset' encoding or the HTML file coding?

2 1

[lxml-dev] UTF-8 not supported
by Ovnicraft 26 May '09

26 May '09

Hi folks, when do this, encoding='iso-8859-1' write xml enconding ok, but when try the same thing with 'UTF-8', not appears in my file, i have 2.2 version. How i can encoding my file with utf-8? regards, -- http://twitter.com/ovnicraft

3 7

[lxml-dev] lxml 2.2 validation question
by James Slagle 20 May '09

20 May '09

Hello, I'm having some trouble getting lxml (v. 2.2) to validate an ElementTree object that I'm building and was hoping someone on the list could help and maybe tell me what I'm doing wrong. If I create an ElementTree object directly from xml and an associated schema, it will validate fine. If I then construct a similar ElementTree object by just instantianting ElementTree, it will not validate. The odd thing is that the resulting xml from etree.tostring for both objects is identical. I've attached a python script that shows the problem I'm having. The validation error is: *** DocumentInvalid: Element 'Foo': No matching global declaration available for the validation root. I can get the second ElementTree object (etree2) to validate if I put the long explicit namesplace in front of the tag value (Foo) when I create etree2 in the script. So, if I change line 25 in the script to: rootelem = etree.Element('{http://example.com}Foo', {}, nsmap) , it will validate. However, the 2 resulting xml outputs are no longer equal b/c the output from etree2 is output with explict namespaces. Any ideas? Thanks for any help. -- James Slagle

4 8

[lxml-dev] could not find doctest
by Mary Lei 20 May '09

20 May '09

I copied the following from the lxml document on the web: #! /stage/irsa-sw-dev/cm/env_1.4_mnt/python/bin/python import urllib import urllib2 import urlparse import os import popen2 import StringIO import sys, getopt import re import string import doctest version = sys.version_info print "python version: ", version print os.environ['PYTHONPATH'] ## test with lxml import lxml.html from lxml.html import builder as E from lxml.html import usedoctest html = E.HTML( E.HEAD( E.LINK(rel="stylesheet", href="great.css", type="text/css"), E.TITLE("Best Page Ever") ), E.BODY( E.H1(E.CLASS("heading"), "Top News"), E.P("World News only on this page", style="font-size: 200%"), "Ah, and here's some more text, by the way.", lxml.html.fromstring("<p> and this is a parsed fragment </p>") ) ) print lxml.html.tostring(html) And got the following error: python version: (2, 5, 1, 'final', 0) .:/home/lei/python-stuff/BeautifulSoup-3.1.0.1:./lib:/home/lei/lxml-2.2/src Traceback (most recent call last): File "/home/lei/python-stuff/test_lxml_htmlout.py", line 22, in <module> from lxml.html import usedoctest File "/home/lei/lxml-2.2/src/lxml/html/usedoctest.py", line 13, in <module> doctestcompare.temp_install(html=True, del_module=__name__) File "/home/lei/lxml-2.2/src/lxml/doctestcompare.py", line 395, in temp_install frame = _find_doctest_frame() File "/home/lei/lxml-2.2/src/lxml/doctestcompare.py", line 486, in _find_doctest_frame "Could not find doctest (only use this function *inside* a doctest)") LookupError: Could not find doctest (only use this function *inside* a doctest) Any clues to this problem ? -- Mary Lei Software Testing IPAC-NExScl Rm: KS-233 MS: 220-6 Phone: 395-1998

2 2

[lxml-dev] parse unable to load url
by Mary Lei 20 May '09

20 May '09

I copied sample code from lxml documentation #! /stage/irsa-sw-dev/cm/env_1.4_mnt/python/bin/python import urllib import urllib2 import urlparse import os import popen2 import StringIO import sys, getopt import re import string version = sys.version_info print "python version: ", version print os.environ['PYTHONPATH'] ## test with lxml from lxml import etree from lxml.html import fromstring, tostring, parse, submit_form #page = parse('http://tinyurl.com').getroot() page = parse('/home/lei/python-stuff/tmp_CoRoT_exo_index.html').getroot() print page page = parse('http://tinyurl.com').getroot() And got errors when parsing url, local file is ok dodo:lei > test_lxml.py python version: (2, 5, 1, 'final', 0) .:/home/lei/python-stuff/BeautifulSoup-3.1.0.1:./lib:/home/lei/lxml-2.2/src <Element html at 3f9a40> Traceback (most recent call last): File "test_lxml.py", line 27, in <module> page = parse('http://tinyurl.com').getroot() File "/home/lei/lxml-2.2/src/lxml/html/__init__.py", line 661, in parse return etree.parse(filename_or_url, parser, base_url=base_url, **kw) File "lxml.etree.pyx", line 2693, in lxml.etree.parse (src/lxml/lxml.etree.c:52591) File "parser.pxi", line 1478, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:75665) File "parser.pxi", line 1507, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:75993) File "parser.pxi", line 1407, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:75002) File "parser.pxi", line 965, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:72023) File "parser.pxi", line 539, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:67830) File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:68877) File "parser.pxi", line 563, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:68093) IOError: Error reading file 'http://tinyurl.com': failed to load HTTP resource -- Mary Lei Software Testing IPAC-NExScl Rm: KS-233 MS: 220-6 Phone: 395-1998

2 2

Re: [lxml-dev] Ask for help about lxml usage
by Stefan Behnel 13 May '09

13 May '09

qhlonline wrote: > My code of multi-thread parsing test is as follows, May be there are some > problems, and I am very glad to get your suggestion. I have created eight > threads in this program and they collaborate to parse a single HTML file > 1000 times, the result I want is the time used. > from lxml import etree > import time > > import threading > class TargetParser(object): > # The Target Parser Def: > def __init__(self): > self.ImgTag = 0 > self.StyleTag = 0 > self.ScriptTag= 0 > def start(self, tag, attrib): > if tag == 'img': > self.ImgTag = self.ImgTag+1 > elif tag == 'style': > self.StyleTag = self.StyleTag+1 > elif tag == 'script': > self.ScriptTag = self.ScriptTag+1 > else: > pass > def end(self, tag): > pass > def close(self): > return self I'd strip this all together and go with separate parse/search phases. > class MultiThread: > def __init__(self): > self.circle = 0 > self.timeres = 0.0 > > def CircleParse(self, webpath, circles=1000): > self.webpath=webpath > self.circles=circles > self.lock=thread.allocate_lock() > self.lock2=thread.allocate_lock() > starttime = time.time() > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) > thread.start_new_thread(self.NewParse,()) I'd use a for-loop here. > self.lock2.acquire() > self.lock2.acquire() This bothers me. Not only that it appears twice, but also that you need an explicit lock in your program. > timeres = time.time()-starttime # total time > print 'In MultiThread Parser, the time Consume is : ',timeres, ' > Seconds!\n' > > def NewParse(self): > > MyParser = TargetParser() > Parser=etree.HTMLParser(target=MyParser) #The parser used in this > thread A general note on naming in Python programs: you should skim through PEP 8. Your code is rather hard to read as it uses AllCamelCase names for class names, method names and some local variables, i.e. things that are semantically very different. It's common to use it for class names, but everything else should use lower-case separated by underscores. > while (self.circle < self.circles): > if(self.circle >= self.circles): > break > > res = etree.parse(self.webpath,Parser) > > if(self.lock.acquire()): > self.circle = self.circle + 1 > if(self.circle >= self.circles and self.lock2.locked()): > > self.lock2.release() > break > self.lock.release() Ok, my take would be this (although it's completely(!) untested): from Queue import Queue from threading import Thread from lxml import etree tag_counters = [ (tagname, etree.XPath('count(//%s)' % tagname)) for tagname in ('img', 'style', 'script') ] def start_threads(func, thread_count, *args): for _ in range(thread_count): thread = Thread(target=func, args=args) thread.setDaemon() thread.start() def handle_urls(url_queue, result_queue): parser = etree.HTMLParser() while True: # I'm a deamon, so I don't care try: url = url_queue.get() doc = etree.parse(url, parser) result = [ (tagname, count(doc)) for tagname, count in tag_counters ] result_queue.put(result) doc = None # free space while we wait except Exception, e: # catch-all to make sure we report all 'normal' exceptions result_queue.put(e) e = None # free space while we wait # run benchmark in_queue = Queue() out_queue = Queue() start_threads(handle_urls, 10, in_queue, out_queue) from time import time t = time() for _ in range(100): in_queue.put("file://tmp/somefile.html") for _ in range(100): print out_queue.get() print time() - t ... minus some glitches, but I bet you can fix them and post a better version. Have fun, Stefan

1 0

Re: [lxml-dev] Ask for help about lxml usage
by Stefan Behnel 13 May '09

13 May '09

[forwarding this to the list] qhlonline wrote: > Thank you very much for your kindly help. I have tried to trace the lxml > source code. I have attempted to find out whether the lxml is simply a > python-wrapped interface for HTML/XML parsing. if so, after my program > have reached the C/Python interface(eg. xmlDoc* htmlCtxtReadMemory in > htmlparser.pxd), the following job should have run parallelled on > different CPU cores, becausing it was using libxml2. I have also naively > thought that the elimination of DOM tree creation process would save time, > but it seems that this job is done by SAX event processers of libxml2 if > we are using the default parser. It's not naive, the thing is just that this is a lot more efficient for single-threaded programs than for multi-threaded ones. That's not an obvious difference, and it's also not advertised in the docs. Most users don't care too much about the exact high performance characteristics and it's not like there's an obvious bottleneck in lxml.etree that you have to warn people about. It's more the general "Python code is single-threaded" kind of thing and it all depends on how you use it, so you have to benchmark your own code anyway. > I don't know the Cython language, some > new types and declarations of variables and functions in .pxi files and > .pxd files makes me feel headache. You can ignore most of the little stuff that you don't understand. The main idea is just that you can switch freely between C and Python in the code, so depending on how firm you are in both, some code sections may be less obvious than others. > but my instructor said that since the > multi-thread parsing program with lxml can save time on two-core CPU > machine(It is true, it can run nearly 20% faseter with 2 thread on > two-core CPU machine), it should run better with more thread on a better > machine. now, with your help, I know maybe multi-process program will do > better. thank you again! Usually a lot faster, as you avoid any implicit concurrency issues. In multi-process programs, synchronisation only happens where you explicitly do it. Things are a lot more implicit and subtle in multi-threading. I'll comment on your program in your other mail. Please keep the list involved next time. Stefan

1 0

[lxml-dev] ElementTree.write default_namespace argument
by Richard Lewis 12 May '09

12 May '09

Hi there, I have lxml version 2.1.5, Python 2.5 both installed from Debian packages. I'd like to be able to use the default_namespace argument to ElementTree's write method. However, I get the following error: Traceback (most recent call last): File "./join-html-pages.py", line 62, in <module> main() File "./join-html-pages.py", line 59, in main document.write(file(new_filename, 'w'), encoding='utf-8', method='html', default_namespace='html') File "lxml.etree.pyx", line 1576, in lxml.etree._ElementTree.write (src/lxml/lxml.etree.c:16699) TypeError: write() got an unexpected keyword argument 'default_namespace' Is this expected behaviour? My application is extracting nodes from numerous XHTML documents and merging them into one HTML document. The bit that's going wrong is writing the resultant HTML to a file. I'm not really interested in it being correct XHTML or in using the correct namespace. So any alternative solutions would also be of interest. Cheers, Richard

2 1

[lxml-dev] Ask for help about lxml usage
by qhlonline 12 May '09

12 May '09

Hi, all I am a lxml experimental user. from site http://codespeak.net/lxml/FAQ.html I know that python can support multithread parsing without GIL. I have tried to write multi-thread parsing program to run on a eight-core CPU computer, but the total CPU used was only 180%, only about 20% of each core had been used.but when I tried the libxml2 directly, It run much faster, and more then 50% of each CPU core were used. My goal is to parse a HTML file on a disk to get special HTML tags and their relative data, like attributes and texts. I will not use DOM tree creation, renew, delete, or XPath operations. then how can my HTML-Parsing program run faster? I have used the Target SAX parser to parse a HTML file ,but the speed is not good enough. the Iterparse can't parse HTML file eigher(I have set the "html=True" parameter), the parser said my HTML file had misplaced the DOCTYPE declaration,but this web page is caught from a popular website and is truly subject to the HTML protocal. Now there are more HTML files to process, so now I wan't to speed up the parsing by multi-thread process. My question are whether the LXML had freed GIL completely on memery and disk file prasing? how can my multi-thread program run faster on multi-core CPU computer? Can I make some change on lxml source to jump some unwanted operation to Improve my program? Thanks a lot Yours Sincere

2 1

[lxml-dev] xpath on text nodes
by Jamie Norrish 09 May '09

09 May '09

The xpath method is currently available only for ElementTree and Element objects. Is it possible for it to be available to text nodes also? My current use case is getting a certain length text context for a particular element node, and I'd like to implement that through a recursive call to a function that returns the content of a supplied text node appended to the content of the next text node in sequence (provided the required length has not been passed). Jamie

2 5

[lxml-dev] how to get line,col position
by Mary Lei 07 May '09

07 May '09

How can I get dtd.validate to return the line, column number for the xhtml in error? Here is my code to validate an xhtml doc against the dtd using lxml: # no need to write a temp html file CoRotHomeFile = open ( 'CoRoTHome.html', 'r' ) contents = CoRotHomeFile.read() CoRotHomeFile.close() dtd1 = etree.DTD(file='xhtml1-transitional.dtd') (the ent files are present) etree.clear_error_log() root1 = etree.HTML(contents) try: rc = dtd1.validate(root1) except (DTDValidateError,DTDError),e: print "e ", e print "dtd errors" len = len(dtd1.error_log) error = dtd1.error_log[0] print "line", (error.line) print "column", (error.column) print dtd1.error_log If I use xmllint, I got column /home/lei/lxml-2.2/src:.:/home/lei/python-stuff/BeautifulSoup-3.1.0.1 <lxml.etree.DTD object at 0x2ede10> dtd errors without any position info line 0 column 0 <string>:0:0:ERROR:VALID:DTD_STANDALONE_WHITE_SPACE: standalone: tr declared in the external subset contains white spaces nodes <string>:0:0:ERROR:VALID:DTD_STANDALONE_WHITE_SPACE: standalone: tr declared in the external subset contains white spaces nodes <string>:0:0:ERROR:VALID:DTD_STANDALONE_WHITE_SPACE: standalone: tr declared in the external subset contains white spaces nodes <string>:0:0:ERROR:VALID:DTD_STANDALONE_WHITE_SPACE: standalone: tr declared in the external subset contains white spaces nodes But if I apply xmllint, it gives the same messages but with positional info: /home/lei/python-stuff/CoRoTHome.html:83: HTML parser error : htmlParseStartTag: invalid element name dedicated to asteroseismology of bright stars (typically V<10mag) and ^ /home/lei/python-stuff/CoRoTHome.html:23: element tr: validity error : standalone: tr declared in the external subset contains white spaces nodes /home/lei/python-stuff/CoRoTHome.html:93: element tr: validity error : standalone: tr declared in the external subset contains white spaces nodes /home/lei/python-stuff/CoRoTHome.html:99: element tr: validity error : standalone: tr declared in the external subset contains white spaces nodes ... Document /home/lei/python-stuff/CoRoTHome.html does not validate against xhtml1-transitional.dtd Thanks. -- Mary Lei Software Testing IPAC-NExScl Rm: KS-233 MS: 220-6 Phone: 395-1998

2 3

[lxml-dev] Ingore namespace when parsing
by Aaron Maxwell 06 May '09

06 May '09

Hi all, When using python lxml to parse an XML document whose root element defines a namespace, is there some way the library can allow me to not explicitly invoke that namespace in queries? Consider an XML document with this content: {{{ <?xml version="1.0" ?> <Root xmlns="http://redsymbol.net/SomeNamespace"> <Child1></Child1> <Child2></Child2> </Root> }}} If I parse it like this: {{{ def ignore_ns(path_to_file): x = etree.parse(open(path_to_file)) for kid in x.getroot(): print kid.tag }}} ... where the path_to_file contains the above xml document, then this output is produced: {{{ {http://redsymbol.net/SomeNamespace}Child1 {http://redsymbol.net/SomeNamespace}Child2 }}} Alternatively, I can define a namespace-string stripping function dynamically, and apply it as needed: {{{ def strip_out_ns(): x = etree.parse(open(path_to_file)) ns = x.getroot().nsmap[None] def no_ns(s): return s.split('{'+ns+'}')[-1] for kid in x.getroot(): print no_ns(kid.tag) }}} The output of this is simpler: {{{ Child1 Child2 }}} More commonly, I will want to search for a child element of some root, using a query like {{{ rootElement.find('Child1') }}} (where rootElement is an Element object). In the namespaced xml document above, this call to .find() will return None, but {{{ # ns found from rootElement.nsmap as above rootElement.find('{' + ns + '}' + 'Child1') }}} will correctly find the child element. In this kind of situation, where I just want to parse the document and really don't care about the namespace, is there some way to construct a parser that will ignore it in a more automated way? Is there a simpler, better approach, or some insight I'm missing? Thanks everyone in advance. Cheers, Aaron

4 6