Mailman 3 June 2009 - lxml - The Python XML Toolkit

[lxml-dev] lxml 2.2.2 released
by Stefan Behnel June 21, 2009

June 21, 2009

Hi all, I just released lxml 2.2.2 to PyPI. This is (mainly) a bug fix release that contains a fix for a potential tree corruption in namespace handling during tree manipulations. Since this can be triggered by input data (depending on your code), you might want to upgrade if you do not control your input. Changelog follows below, as usual. Stefan 2.2.2 (2009-06-21) Features added * New helper functions strip_attributes(), strip_elements(), strip_tags() in lxml.etree to remove attributes/subtrees/tags from a subtree. Bugs fixed * Namespace cleanup on subtree insertions could result in missing namespace declarations (and potentially crashes) if the element defining a namespace was deleted and the namespace was not used by the top element of the inserted subtree but only in deeper subtrees. * Raising an exception from a parser target callback didn't always terminate the parser. * Only {true, false, 1, 0} are accepted as the lexical representation for BoolElement ({True, False, T, F, t, f} not any more), restoring lxml <= 2.0 behaviour.

1 0

[lxml-dev] can validation yield multiple errors?
by az＠svilendobrev.com June 18, 2009

June 18, 2009

hi i'm new to lxml/libxml. as i see, the validation (i use schemas) stops at first error. is it possible to get all errors that parser is able to recover from? ciao svilen www.svilendobrev.com

2 1

[lxml-dev] parser target exception recovery bug?
by D.Hendriks (Dennis) June 17, 2009

June 17, 2009

Hello all, Using lxml 2.2 with a custom parser target (tree builder), I've run into a problem when the parser target raises an exception. In this case, parsing continues, although only for 'data' (not for 'start' and 'end'). I used recover=False when creating the XMLParser. Using the following code: import sys from lxml import etree # Parser target without exceptions. class MyTreeBuilder1(object): def close(self): print 'close' def start(self, tag, attrs): print 'start', tag, attrs def data(self, data): if len(data.strip()) > 0: print 'data: data=', repr(data) def end(self, tag): print 'end', tag # Parser target with exceptions. class MyTreeBuilder2(MyTreeBuilder1): def close(self): print 'close' def start(self, tag, attrs): print 'start', tag, attrs def data(self, data): if len(data.strip()) > 0: print 'data: data=', repr(data) def end(self, tag): print 'end', tag if tag=='b': print 'ERROR' raise ValueError('error') xml_data='''<a> <b>test</b> <d>test2</d> <d>test2</d> </a>''' # Successfull parsing. print '---' builder = MyTreeBuilder1() parser = etree.XMLParser(target=builder, recover=False) rslt = etree.fromstring(xml_data, parser) # Unsuccessfull parsing. print '---' builder = MyTreeBuilder2() parser = etree.XMLParser(target=builder, recover=False) rslt = etree.fromstring(xml_data, parser) I get this output: --- start a {} start b {} data: data= u'test' end b start d {} data: data= u'test2' end d start d {} data: data= u'test2' end d end a close --- start a {} start b {} data: data= u'test' end b ERROR data: data= u'test2' data: data= u'test2' Traceback (most recent call last): File "lxml_parser_target_bug.py", line 49, in ? rslt = etree.fromstring(xml_data, parser) File "lxml.etree.pyx", line 2534, in lxml.etree.fromstring (src/lxml/lxml.etree.c:51135) File "parser.pxi", line 1523, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:76176) File "parser.pxi", line 1402, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:74927) File "parser.pxi", line 928, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:71707) File "parsertarget.pxi", line 135, in lxml.etree._TargetParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:82586) File "lxml.etree.pyx", line 230, in lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:6813) File "saxparser.pxi", line 227, in lxml.etree._handleSaxEnd (src/lxml/lxml.etree.c:78230) File "parsertarget.pxi", line 78, in lxml.etree._PythonSaxParserTarget._handleSaxEnd (src/lxml/lxml.etree.c:81918) File "lxml_parser_target_bug.py", line 33, in end raise ValueError('error') ValueError: error The first output (between --- and ---) is ok, since it is for the non-exception parser target. The second output (after the second ---) is not ok for me. You can see 'ERROR' at the point where the exception is raised. After that, two 'data' events are generated in the parser target. Clearly, parsing continued. Also, the 'close' is never called. After the entire input is parsed, the exception is finally re-raised. Two questions: - Is the continued parsing ('data' function calls) a bug? - Is the not calling 'close' a bug? Any replies would be greatly appreciated. Dennis

2 5

[lxml-dev] objectify parses 't' and 'f' as BoolElement?
by jholg＠gmx.de June 17, 2009

June 17, 2009

Hi, somewhere on the way between lxml 2.0 and 2.1.5 objectify changed to accept 't' and 'f' as values recognized as BoolElements: 2.0.alpha4: 0 $ python2.4 -i -c 'from lxml import etree, objectify; objectify.enableRecursiveStr(); print etree.__version__; print etree.LIBXML_VERSION, etree.LIBXSLT_VERSION' 2.0.alpha4 (2, 6, 27) (1, 1, 20) >>> root = objectify.fromstring('<root><bool>f</bool></root>') >>> print root root = None [ObjectifiedElement] bool = 'f' [StringElement] >>> root.bool.text 'f' 2.1.5: 0 $ python2.4 -i -c 'from lxml import etree, objectify; objectify.enable_recursive_str(); print etree.__version__; print etree.LIBXML_VERSION, etree.LIBXSLT_VERSION' 2.1.5 (2, 6, 32) (1, 1, 23) >>> root = objectify.fromstring('<root><bool>f</bool></root>') >>> print root root = None [ObjectifiedElement] bool = False [BoolElement] I consider this a bug, XML Schema datatypes spec says " 3.2.2 boolean [Definition:] boolean has the ·value space· required to support the mathematical concept of binary-valued logic: {true, false}. 3.2.2.1 Lexical representation An instance of a datatype that is defined as ·boolean· can have the following legal literals {true, false, 1, 0}. ... " Objections to restricting this to said literals, again? Holger -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01

2 3

[lxml-dev] Using, or building, lxml in Windows with Cygwin
by David Antliff June 16, 2009

June 16, 2009

Hello, I am trying to get the Windows lxml egg to work with Cygwin, Python 2.5.2 on WindowsXP, 32bit. I wasn't able to find a Cygwin package for lxml so I felt I needed to obtain or build a win32 version of lxml. I want to use the xpath functionality that lxml provides in python scripts, running in Cygwin. *** All commands below are issued in a Cygwin bash shell *** I downloaded lxml-2.2.1-py2.5-win32.egg<http://pypi.python.org/packages/2.5/l/lxml/lxml-2.2.1-py2.5-win32.egg#md5=e…>from http://pypi.python.org/pypi/lxml/2.2.1 and placed it in a directory, setting PYTHONPATH to point here too. However pkg_resources.require() can't find it, so instead I unzipped it into a directory called 'lxml'. Then I do this: $ python Python 2.5.2 (r252:60911, Dec 2 2008, 09:26:14) [GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree Traceback (most recent call last): File "<stdin>", line 1, in <module> File "lxml/etree.py", line 7, in <module> __bootstrap__() File "lxml/etree.py", line 6, in __bootstrap__ imp.load_dynamic(__name__,__file__) ImportError: Permission denied Ok, I think I know why this is happening, so I do this: $ chmod +x lxml/*.pyd $ python Python 2.5.2 (r252:60911, Dec 2 2008, 09:26:14) [GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree Segmentation fault (core dumped) Ok, so that didn't work. The next thing I tried was the instructions found here: http://www.scribd.com/doc/16158375/LXML-221-documentation in section 23.7 - "Static linking on Windows". I downloaded the latest versions of iconv, libxml2, libxslt, zlib and lxml-2.2.1.tar.gz. I unpacked them as per the instructions and edited lxml-2.2.1/setup.py to fix the STATIC_INCLUDE_DIRS and STATIC_LIBRARY_DIRS to point back to those newly extracted libraries. Now I run: $ cd lxml-2.2.1 $ python setup.py bdist_wininst --static Building lxml version 2.2.1. NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' needs to be available. ERROR: /bin/sh: xslt-config: command not found ** make sure the development packages of libxml2 and libxslt are installed ** Using build configuration of libxslt Building against libxml2/libxslt in one of the following directories: ..\libxml2-2.7.3.win32\lib ..\libxslt-1.1.24.win32\lib ..\zlib-1.2.3.win32\lib ..\iconv-1.9.2.win32\lib running bdist_wininst error: distribution contains extensions and/or C libraries; must be compiled on a Windows 32 platform So I figure this is because I'm trying to build inside Cygwin. I tried inside a cmd.exe window, and it gets a bit further but then complains about not having VisualC++2003 installed... I tried -c mingw32 but setup.py didn't understand that option. Seemed like a dead end. I think it might be picking up my ActiveState Python installation too, rather than the Cygwin one. I also tried the actual versions of libxml, libxslt etc that the document actually mentions, same result. Anyway, my understanding is that all this should be unnecessary - the .egg should contain everything I need, shouldn't it? But the segmentation fault at import time is bad. I'd really like to use lxml in Cygwin if possible, because all the other options for xpath processing seem very limited. I like the idea of the egg because I want to be able to deploy my XML-processing python scripts with the .egg, so that I don't need to install any additional software (such as Cygwin packages). Any help would be greatly appreciated, please. Regards, -- David.

2 5

Re: [lxml-dev] Difference between xhtml etrees
by Stefan Behnel June 16, 2009

June 16, 2009

Hi, please CC the list on replies. D wrote: > 2009/6/11 Stefan Behnel: >> D wrote: >>> I have two xhtml documents which I would like to compare. They are >>> available as etrees. >>> Ideally I would like to have a resulting tree, where the appropriate >>> changes are marked with ins and del tags. I don't need anything fancy >>> like a detection of moves. >>> >>> I had a look at lxml.html.diff >>> http://codespeak.net/lxml/lxmlhtml.html#html-diff >>> but it operates on html strings only, and not on my parsed tree. >> >> Did you try passing the root elements of the trees? > > passing the root objects was a good idea, it can generates the > difference the way I want it. I just don't manage to get the data back > to xhtml. Maybe you could have a look: > > Here is my code: > def expandFiles(filename): > """open the file named filename, return an etree""" > document = "".join(open(filename).readlines()) > px = lxml.etree.XMLParser(load_dtd=True, no_network=False) > px.feed(document) > rx=px.close() > docx=lxml.etree.ElementTree(rx) > return docx Note that "load_dtd" does not imply validation, just that a DTD will be loaded if referenced. Also, it is a *lot* more efficient to do this: parser = lxml.etree.XMLParser(load_dtd=True, no_network=False) def expandFiles(filename): """open the file named filename, return an etree""" return lxml.etree.parse(filename, parser) ... and I'd actually rename the function (or drop it completely). > r1=expandFiles(r"1.xhtml") > r2=expandFiles(r"2.xhtml") > diff = lxml.html.diff.htmldiff(r1.getroot(),r2.getroot()) > # diff is now an html fragment, parse it > pdiff = lxml.html.document_fromstring(diff) > lxml.html.html_to_xhtml(pdiff) > pe = lxml.etree.ElementTree(pdiff) So far, so good. > # this gives me an xhtml file that is parsed without errors by > firefox, but does not contain any markup > # it looks like this in firefox: {http://www.w3.org/1999/xhtml}meta> > Resist SPR3012 Preparation{http://www.w3.org Not sure how this can happen. I'll give it a try later today. > # in addition, all character entities apper in the form > and not > like they should: #62; Would you have a 'real' example here? > I don't manage to transform pdiff to the same form r1 and r2 are in. > > I am sure this is due to a basic misunderstanding of lxml, maybe you > directly see what I am doing wrong? Not direcly, no. Maybe others have an idea? Stefan

2 1

[lxml-dev] Binary egg for Mac OS X
by Alexander Limi June 15, 2009

June 15, 2009

Hi, I'm working on documenting Deliverance / xdv for use with Plone. It has lxml as a dependency, and I have run into a serious issue: On Mac OS X, we can't assume that people have Xcode (ie. gcc and friends) installed, thus we can't really compile lxml on those computers, not even using the staticlxml[1] recipe. I see that there are binary eggs for Windows, is there a special reason why there are no binary eggs for OS X, or is it just a matter of not having the infrastructure to make it available? Happy to help find a solution if it's just a matter of locating a reliable way to get it compiled every time there is a new release. [1] http://pypi.python.org/pypi/z3c.recipe.staticlxml/ -- Alexander Limi · http://limi.net

6 11

Re: [lxml-dev] Difference between xhtml etrees
by D June 10, 2009

June 10, 2009

Hello, I have two xhtml documents which I would like to compare. They are available as etrees. Ideally I would like to have a resulting tree, where the appropriate changes are marked with ins and del tags. I don't need anything fancy like a detection of moves. I had a look at lxml.html.diff http://codespeak.net/lxml/lxmlhtml.html#html-diff but it operates on html strings only, and not on my parsed tree. That solution would mean that I have to dump the xhtml to html, diff, reparse the string as html and transform it to xml. I like the way daisydiff operates, but again only on files, and the output is either html, or xml, which I would need to merge into my tree. http://code.google.com/p/daisydiff/wiki/Examples Is there any way to compare two trees directly and interpret the differences the way daisydiff does, i.e. look only at text? Daniel

2 1

Re: [lxml-dev] Binary egg for Mac OS X
by Stefan Behnel June 8, 2009

June 8, 2009

Hi, Stefan Eletzhofer wrote: > On Mon, Jun 8, 2009 at 7:54 AM, Stefan Behnel wrote: >> What I would like to see is a really fat egg, i.e. one that works on all >> hardware platforms supported by MacOS-X. I have no idea how to do that, >> so I can't provide any hints. But since there seem to be some MacOS users >> on the list by now, I hope we can get away with some web digging plus >> trial and error. > > Well, I don't know if that's possible -- AFAIK there's only "binary > eggs" and that's it. That would be a compiler option, if I'm not mistaken. >> Also, the build must not rely on any macports libraries, as not everyone >> uses macports. So we'd have to make sure it's only built against system >> libraries (plus the statically built libxml2/libxslt). > > objdump says that this is true for libs built by the staticlxml recipe. Ok, although I'd prefer getting the eggs created using lxml's own static build. >> What about the "system Python vs. macports Python" question? Would such >> an >> egg work on both? Or does it make more sense to support only the system >> Python installation, and to leave macports specific builds to the >> macports maintainers? > > Well, we'd probly need to support all the python **versions** (2.4, > 2.5, ...). And > being statically built, it does per definition not interfer with other > libs, so we should be pretty safe here. IMO, it should only be statically built against libxml2 and libxslt, the rest would be taken from the system libs. Static building is evil, unless it *really* solves a problem. > So all in all, I think we should identify: > > - the LXML versions which are "important enough" to support (1.2.6 > anyone?) Since everything before 2.2 is out of maintenance (and the API is mostly compatible across versions anyway), I'd prefer having eggs for 2.2+ only. Providing older eggs will just let people run into ancient bugs. > - the python versions which we support (surely 2.4 and 2.5. what other > versions?) We currently have Windows eggs for 2.4/5/6 and 3.0, so having the same span for MacOS eggs would be nice. > I could perhaps set up a semi-automated system on my mac server to build > these. > I can also put them on a FTP / Web Server somewhere, and I could (given > access) also upload the static eggs to PyPi. Sure. Do you have a PyPI account? > Do we want to give the static versions a dedicated name? > "lxml-x.y.z-static" or smth like > that? Maybe not everyone wants static eggs (even not on OSX boxen)? I would want them to be the default download on MacOS (i.e. no special naming), but only after some testing by some MacOS users on differently configured systems, i.e. using different MacOS versions, different package distributions, different Python versions, etc. So maybe you could provide them separately first? Thanks for volunteering! Stefan

1 0

[lxml-dev] lxml 2.2.1 released
by Stefan Behnel June 5, 2009

June 5, 2009

Hi all, I just pushed lxml 2.2.1 to PyPI as a minor maintenance release. Changelog follows below. This release was built with Cython 0.11.2. Have fun, Stefan 2.2.1 (2009-06-02) Features added * Injecting default attributes into a document during XML Schema validation (also at parse time). * Pass huge_tree parser option to disable parser security restrictions imposed by libxml2 2.7. Bugs fixed * The script for statically building libxml2 and libxslt didn't work in Py3. * XMLSchema() also passes invalid schema documents on to libxml2 for parsing (which could lead to a crash before release 2.6.24).

3 3