Mailman 3 November 2008 - lxml - The Python XML Toolkit

[lxml-dev] Parsing this document
by Ross Vandegrift 25 Nov '08

25 Nov '08

Hi everyone, Sorry to as a -users question on a -dev mailing-list, but I don't see a -users resource for lxml. If one exists, I'd be happy to head over there. I have an XML document from a network device that I want to pull into my python code. The example document I'm working with is attached as vlan.xml. I find the results of this to be utterly perplexing: >>> import lxml.etree as ET >>> vtree = ET.parse("vlan.xml") >>> vtree.xpath("vlan") [] >>> vtree.xpath("//vlan") [] >>> vtree.xpath("rpc-reply") [] >>> vtree.xpath("/rpc-reply") [] >>> vtree.xpath("*") [<Element {http://xml.juniper.net/junos/9.2R2/junos-esw}vlan-information at 95a8a04>] >>> vtree.xpath("vlan-information") [] Based on the examples in the lxml XPath documentation, I'd expect the first to give me a list of Elements, one for each <vlan>...</vlan> in the file. Second should be the same based on XPath stuff I've read in Python and XML. My next two are shots in the dark: <rpc-reply> is the top-level element, but I can't retrieve it relatively or absolutely. Next is another shot in the dark, and somehow what I get the element one under the top-level. Huh? Also, I can't retrieve it by name. -- Ross Vandegrift ross(a)kallisti.us "If the fight gets hot, the songs get hotter. If the going gets tough, the songs get tougher." --Woody Guthrie

2 2

[lxml-dev] lxml is loading dtd from w3.org but I don't want it to
by Brad Clements 23 Nov '08

23 Nov '08

I cannot seem to disable loading DTD from w3.org when transforming a file. I suppose I am doing something wrong, but I can't see what that could be. I am using lxml inside a wsgi application, and every network request causes my wsgi server to go back to w3 to get the dtd (I am not using local catalogs). I am on ubuntu 7.xx (can't recall, 2 versions old) x86_64, using python 2.5 >>> sys.version '2.5.1 (r251:54863, Mar 7 2008, 03:39:23) \n[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)]' >>> lxml.etree.__version__ u'2.1.3' >>> lxml.etree.LIBXSLT_COMPILED_VERSION (1, 1, 21) >>> lxml.etree.LIBXML_COMPILED_VERSION (2, 6, 30) my code looks like this (I added all the =False keywords to see if they helped, they did not): parser = etree.XMLParser(load_dtd=False, attribute_defaults=False, dtd_validation=False) parser.resolvers.add(Resolver(resolver=xml_src_object.resolve)) xml_doc = etree.fromstring(xml_src_object.get_source(), parser, base_url=document_uri) also the xml or stylesheet source might be loaded this way: parser = etree.XMLParser(load_dtd=False, attribute_defaults=False, dtd_validation=False, no_network=True) stylesheet_doc = etree.parse(xslt_src_object, parser) where xslt_src_object is a string containing the filepath to parse. I was running an older lxml previously, but it was also downloading catalogs. I don't know what version that was, sorry. I upgraded to 2.1.3 today. If I do the transform with xsltproc, it goes fast and does not download the dtd. I have been working on this for a couple of hours, so I'm likely to have made a mistake on this. However this is code I've been using for more than a year and I doubt very much that it's always been downloading the dtd. I think I did an apt-get update/upgrade some weeks back, I might have gotten a newer libxml2. Perhaps it's not honoring the load_dtd=False ? any ideas? -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM: BKClements

2 1

Re: [lxml-dev] lxml Mac installation idea
by Chris Withers 21 Nov '08

21 Nov '08

Paul wrote: >>>>> Stefan Behnel wrote: >>>>> but I'd like it to work without buildout, and there's also several >>>>> buildout recipes and configurations out there and not one clear >>>>> canonical way to build lxml. Out of interest, what's the canonical way to build lxml when you're developing it? What's the canonical way to run the unit tests? I seem to be moving more and more to using buildout for each of the packages I maintain as it provides a simple answer to both these questions. > Here's the buildout.cfg that we've used: > > [libxml2] > recipe = zc.recipe.cmmi > url = http://somecustomer.agendaless.com/indexes/karl/cmmi/libxml2-2.6.32.tar.gz > extra_options = --without-python > > [libxslt] > recipe = zc.recipe.cmmi > url = http://somecustomer.agendaless.com/indexes/karl/cmmi/libxslt-1.1.24.tar.gz > extra_options = --with-libxml-prefix=${libxml2:location} > --without-python > > [lxml-environment] > XSLT_CONFIG=${buildout:directory}/parts/libxslt/bin/xslt-config > XML2_CONFIG=${buildout:directory}/parts/libxml2/bin/xml2-config > > [lxml] > recipe = zc.recipe.egg:custom > egg = lxml > include-dirs = ${libxml2:location}/include/libxml2 > ${libxslt:location}/include > library-dirs = ${libxml2:location}/lib > ${libxslt:location}/lib > rpath = ${libxml2:location}/lib > ${libxslt:location}/lib > environment = lxml-environment This looks quite similar to the buildout Martijn originally came up with back in 2005. Is it related to that? >> One problem with buildout recipes is that they aren't, AFAIK, very >> compatible with other systems. That is, they can build the egg, but >> the egg isn't on the path of any interpreter, so unless you use >> buildout entirely you'll have to do some further manipulation to >> make lxml available. Well, that's true, but surely lxml should or is ending up as a binary egg or few up on PyPI? http://pypi.python.org/pypi/lxml/2.1.2 ...would seem to suggest that this is the case. In which case, I guess buildout would help for building libxml2 and libxslt on environments other than Windows, but other than that? > Right. On one hand, I'm reluctant to put all lxml's eggs in the > buildout basket [wink] for that reason. ...I honestly don't think there's anything better in the python world. It's just a shame buildout's docs suck so much. Sometimes doctests have a *lot* to answer for! > lxml is much bigger than the > world of Zope. So is buildout ;-) > Being confronted with something you didn't know > (buildout) just to get to the thing you wanted (lxml) is a pain. Yes, I agree that simply for using lxml that shouldn't be the case. But, what about developing lxml? And what about if your project is using buildout already? > OTOH, I wonder if the solution is to simply provide a number of > options. We're really only talking about Mac users, so it isn't as > big of a decision. What's special about Mac users? (sorry for being dumb...) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

8 36

Re: [lxml-dev] [Lxml-checkins] r60038 - lxml/trunk - make static build the default
by Stefan Behnel 20 Nov '08

20 Nov '08

Hi Ian, ianb wrote: > Modified: lxml/trunk/setupinfo.py > ============================================================================== > --- lxml/trunk/setupinfo.py (original) > +++ lxml/trunk/setupinfo.py Thu Nov 20 20:25:47 2008 > @@ -317,7 +317,7 @@ > OPTION_STATIC = has_option('static') > OPTION_DEBUG_GCC = has_option('debug-gcc') > OPTION_AUTO_RPATH = has_option('auto-rpath') > -OPTION_BUILD_LIBXML2XSLT = has_option('static-deps') > +OPTION_BUILD_LIBXML2XSLT = has_option('static-deps', 'true') > if OPTION_BUILD_LIBXML2XSLT: > OPTION_STATIC = True > OPTION_LIBXML2_VERSION = option_value('libxml2-version') I'm not sure this should become the default. A dynamic build is generally the right thing and works very well on most platforms, except for Windows and MacOS-X. I wouldn't mind defaulting it to 'true' explicitly on those two platforms, though. MacOS-X packagers can just pass the respective env var to disable it. Stefan

2 1

[lxml-dev] lxml 2.0.10 and 2.1.3 released
by Stefan Behnel 19 Nov '08

19 Nov '08

Hi, I just released lxml 2.0.10 and lxml 2.1.3. Both releases are pure bug-fix releases for the stable 2.1 series and the mature 2.0 series. The ChangeLog follows below. As there appears to be a problem with PyPI's upload currently, you can get them from the homepage for now: http://codespeak.net/lxml/lxml-2.1.3.tgz http://codespeak.net/lxml/2.0/lxml-2.0.10.tgz Have fun, Stefan 2.0.10 (2008-11-17) Bugs fixed * Ref-count leaks when lxml enters a try-except statement while an outside exception lives in sys.exc_*(). This was due to a problem in Cython, not lxml itself. 2.1.3 (2008-11-17) Bugs fixed * Ref-count leaks when lxml enters a try-except statement while an outside exception lives in sys.exc_*(). This was due to a problem in Cython, not lxml itself. * Parser Unicode decoding errors could get swallowed by other exceptions. * Name/import errors in some Python modules. * Internal DTD subsets that did not specify a system or public ID were not serialised and did not appear in the docinfo property of ElementTrees. * Fix a pre-Py3k warning when parsing from a gzip file in Py2.6. * Test suite fixes for libxml2 2.7. * Resolver.resolve_string() did not work for non-ASCII byte strings. * Resolver.resolve_file() was broken. * Overriding the parser encoding didn't work for many encodings.

1 1

[lxml-dev] XmlTextReader support
by Roy, Remy 17 Nov '08

17 Nov '08

Hello, Is there any roadmap to provide support for the XmlTextReader interface as specified on this page http://www.xmlsoft.org/xmlreader.html in lxml? Regards, Rémy Roy

2 3

Re: [lxml-dev] lxml Mac installation idea
by Michael Guntsche 17 Nov '08

17 Nov '08

On Nov 17, 2008, at 18:18, Paul Everitt wrote: > > Unfortunately no change. > > --Paul > > On Nov 17, 2008, at 11:36 AM, Michael Guntsche wrote: > >> >> Hello Paul, >> >> Please try with the attached patch applied.... >> I get the same problems if I build libxml on the command-line, do not >> understand why it is working when I am compiling with python and >> subprocess. >> Tell me if the attached patch works for you. >> >> Kind regards, >> Michael >> >> >> Index: buildlibxml.py >> =================================================================== >> --- buildlibxml.py (revision 59948) >> +++ buildlibxml.py (working copy) >> @@ -188,7 +188,7 @@ >> ] >> call_subprocess(configure_cmd, cwd=libxml2_dir, **call_setup) >> call_subprocess( >> - ['make'], cwd=libxml2_dir) >> + ['make'], cwd=libxml2_dir, **call_setup) >> call_subprocess( >> ['make', 'install'], cwd=libxml2_dir) >> >> @@ -202,7 +202,7 @@ >> >> call_subprocess(libxslt_configure_cmd, cwd=libxslt_dir, >> **call_setup) >> call_subprocess( >> - ['make'], cwd=libxslt_dir) >> + ['make'], cwd=libxslt_dir, **call_setup) >> call_subprocess( >> ['make', 'install'], cwd=libxslt_dir) >> Paul, please completely clean up everything from the previous build (best just make a clean checkout) and try again. If this still does not work set MACOSX_DEPLOYMENT_TARGET=10.3 before starting the compilation. $ export MACOSX_DEPLOYMENT_TARGET=10.3 /Mike

2 3

[lxml-dev] Leaking tracebacks
by Christian Heimes 17 Nov '08

17 Nov '08

Hello! A college run into a memory leak on one of our servers a couple of days ago. With the help of Dowser - a CherryPy plugin - we were able to nail down the leak to traceback and frame objects. Some code in Martijn Faassen's oaipmh package kept references to an exception traceback. Today I was able to track 'n nail down the issue. I highly suspect lxml to be the culprit. Whenever the code reaches lxml with an exception in sys.exc_info the exception leaks. My dowser patch shows a refcnt of 4 references to the root traceback object that can't be traced to objects in the gc.get_objects() list. A minor change to oaipmh.server.XMLTreeServer gets rid of the leaking tracebacks: def handleException(self, exception): if isinstance(exception, error.ErrorBase): sys.exc_clear() ## <- REF LEAK FIX envelope = self._outputErrors( [(exception.oainame(), str(exception))]) return envelope # unhandled exception, so raise again raise We are using lxml 2.1.2 on Python 2.5.2. I think lxml 2.0.7 didn't have the problem. Christian

2 6

[lxml-dev] specifying lxml as a dependency?
by Chris Withers 17 Nov '08

17 Nov '08

Hi All, I'm just about to start moving Twiddler from ElementTree to lxml and I'm wondering how to specify lxml as a requirement in Twiddler's setup.py? Do I just do the normal setuptools dance or is there more to it than that? If I specify lxml as a requirement, how do lxml's requirements for libxml2 and libxlst get satisfied? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

4 7

[lxml-dev] Fwd: lxml Mac installation idea
by Michael Guntsche 17 Nov '08

17 Nov '08

CC'ing the list. Begin forwarded message: > From: Michael Guntsche <mike(a)it-loops.com> > Date: November 17, 2008 17:08:48 GMT+01:00 > To: Paul Everitt <paul(a)agendaless.com> > Subject: Re: [lxml-dev] lxml Mac installation idea > > > On Nov 17, 2008, at 16:13, Paul Everitt wrote: > >> >> I got: >> >> $ python setup.py build --static-deps >> [.........snip.......] >> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk - >> O2 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch >> -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized - >> Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings - >> Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested- >> externs -Winline -Wredundant-decls -arch ppc -arch i386 -isysroot / >> Developer/SDKs/MacOSX10.4u.sdk -o xmllint xmllint.o ./.libs/ >> libxml2.a -lpthread -lz /usr/lib/libiconv.dylib -lm >> ld: library not found for -lcrt1.10.5.o >> collect2: ld returned 1 exit status >> ld: library not found for -lcrt1.10.5.o >> collect2: ld returned 1 exit status >> lipo: can't open input file: /var/folders/i4/i4lSVTRWHsaTXG7UXMfGbE+ >> ++TI/-Tmp-//ccDzwBUG.out (No such file or directory) >> make[2]: *** [xmllint] Error 1 >> make[1]: *** [all-recursive] Error 1 >> make: *** [all] Error 2 >> Traceback (most recent call last): >> File "setup.py", line 107, in ? >> STATIC_CFLAGS, STATIC_BINARIES), >> File "/Users/paul/sandboxes/lxml/trunk/setupinfo.py", line 46, in >> ext_modules >> libxslt_version=OPTION_LIBXSLT_VERSION) >> File "/Users/paul/sandboxes/lxml/trunk/buildlibxml.py", line 190, >> in build_libxml2xslt >> call_subprocess( >> File "/Users/paul/sandboxes/lxml/trunk/buildlibxml.py", line 151, >> in call_subprocess >> raise Exception('Command "%s" returned code %s' % (cmd_desc, >> returncode)) >> Exception: Command "make" returned code 2 >> > > Hmm, is this with the latest version from trunk?

2 1