Mailman 3 October 2007 - lxml - The Python XML Toolkit

[lxml-dev] objectify StringElement bugs
by jholg＠gmx.de 24 Oct '07

24 Oct '07

Hi, I just found 2 StringElement bugs (both 1.3 series and trunk are concerned): Comparison with other (parsed, not set through the API) empty string elements: >>> root = objectify.fromstring("<root><s/><s2/></root>") >>> root.s == root.s2 False >>> String formatting (__mod__): >>> root.s = "%d" >>> print root.s % 5 Traceback (most recent call last): File "<stdin>", line 1, in ? File "objectify.pyx", line 691, in objectify.StringElement.__mod__ TypeError: int argument required >>> I've a fix (with tests) ready for this, gonna check that in soon. Holger -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail

2 3

[lxml-dev] custom resolver, why does system url start with XSLT:?
by Brad Clements 23 Oct '07

23 Oct '07

I have a project (XSL based TAL) that has used libxml2 and libxslt for a couple of years. I have a custom resolver that has worked "ok" with this. Now I have converted the project to use lxml. I am creating a parser and adding my resolver. when my resolver gets called, the URIs are weirdly mangled like this: resolve url 'XSLT:///xml/navigation.xml' id None ctext <lxml.etree._XSLTResolverContext object at 0xb6a5353c> resolve url '/xml/carrier_payables_navigation.xml' id None ctext <lxml.etree._XSLTResolverContext object at 0xb6a5353c> resolve url 'XSLT:///services/+payment_accounts' id None ctext <lxml.etree._XSLTResolverContext object at 0xb6a5353c> (the 2nd one is not mangled, looks ok to me) What's the story with XSLT:// being stuck on the front of the system urls? I don't see that happen when I use libxml2 directly. I tried looking through the lxml source to find this, but I couldn't find it in docloader, parser, or xslt. where is the XSLT scheme coming from, is it lxml or libxslt? Why is it being inserted? The last example url, comes from using document() in a stylesheet, (the converted form of this:) <option tal:repeat="account document('/services/+payment_accounts')/data/accounts/item" tal:attributes="value $account/id" tal:content="concat($account/name, ' ($', $account/currency, ')')" /> I expect my resolver to get a system url of '/services/+payment_accounts' The 2nd example url above (the non-mangled one), also comes from a document call, like this: <tal:block tal:omit-tag="" tal:repeat="tab document($const/@secondary_navigation)/root/tab"> Maybe the difference is due to one document() using a constant string, the other using a variable..? I am using 2.0 alpha4 and 2.0 alpha 3 (two different systems, same problem). I can't see how from lxml to tell you which version of libxml2 and libxslt I am using. (the .xsl that is the converted from of the above TAL statements has these statements) <xsl_:for-each select="document('/services/+payment_accounts')/data/accounts/item"> <xsl_:for-each select="document($const/@secondary_navigation)/root/tab"> Thanks for any suggestions! -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM: BKClements

2 8

[lxml-dev] lxml 1.3.5 released
by Stefan Behnel 23 Oct '07

23 Oct '07

Hi, lxml 1.3.5 is on PyPI. This is a minor bug-fix release for the stable 1.3 series. Stefan 1.3.5 (2007-10-22) ================== Bugs fixed ---------- * lxml.etree could crash when adding more than 10000 namespaces to a document * lxml failed to serialise namespace declarations of elements other than the root node of a tree

2 4

[lxml-dev] empty schematron test segfault, revisited
by jholg＠gmx.de 22 Oct '07

22 Oct '07

Hi, I'm taking another look at the seqfaults I see with schematron support, and: 0 lb54320@adevp02 .../XML $ PYTHONPATH=/data/pydev/hjoukl/LXML/lxml/build/lib.solaris-2.8-sun4u-2.4 python2.4 -c "from lxml import etree; tree=etree.parse('invalid_empty.xst'); schema = etree.Schematron(etree=tree)" Segmentation Fault (core dumped) 139 lb54320@adevp02 .../XML $ PYTHONPATH=/data/pydev/hjoukl/LXML/lxml/build/lib.solaris-2.8-sun4u-2.4 python2.4 -c "from lxml import etree; schema = etree.Schematron(file='invalid_empty.xst')" Traceback (most recent call last): File "<string>", line 1, in ? File "schematron.pxi", line 111, in lxml.etree.Schematron.__init__ lxml.etree.SchematronParseError: Document is not a valid Schematron schema 1 lb54320@adevp02 .../XML $ When handing in a pre-parsed tree I run into the segfault, whereas I get a correct error message when leaving file parsing to xmlSchematronParse(). As different parser context factories (xmlSchematronNewDocParserCtxt / xmlSchematronNewParserCtxt ) get used for the 2 entry points, I suspect that s.th. in libxml2 is buggy here, i.e. that xmlSchematronNewDocParserCtxt forgets to initialize something crucial that is then erroneously accessed in the error reporting. Holger -- Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten Browser-Versionen downloaden: http://www.gmx.net/de/go/browser

1 0

[lxml-dev] trunk test problems, release tags
by jholg＠gmx.de 22 Oct '07

22 Oct '07

Hi, I'm having issues with the latest trunk head. When trying to test my build it suddenly just hangs, and has then to get killed the hard way: 0 lb54320@adevp02 .../trunk $ python2.4 test.py -p -v /data/pydev/hjoukl/LXML/tmp/trunk/src/lxml/html/__init__.py:22: UserWarning: This version of libxml2 has a known XPath bug. Use it at your own risk. _rel_links_xpath = etree.XPath("descendant-or-self::a[@rel]") TESTED VERSION: 2.0.alpha4-47483 Python: (2, 4, 4, 'final', 0) lxml.etree: (2, 0, -196, 47483) libxml used: (2, 6, 27) libxml compiled: (2, 6, 27) libxslt used: (1, 1, 20) libxslt compiled: (1, 1, 20) 172/824 ( 20.9%): test_element_with_attributes_ns (lxml.tests.test_elementtree.ETreeTestCase) ^Z[1]+ Stopped python2.4 test.py -p -v 0 lb54320@adevp02 .../trunk $ kill -9 %+ I've tried this with newer libxml2/libxslt versions and it shows just the same behaviour. It is also reproducible on different (solaris) machines. I'm building with the latest Cython 0.9.6.7 on sparc Solaris. Testing an 1.3.3 checkout works just fine, on the same machine, with the same library versions. Trussing the test run doesn't really tell me much: 0 lb54320@adevp02 .../trunk $ truss python2.4 test.py -p -vv [...] test_element_with_attributes_keywords (lxml.tests.test_elementtree.ETreeTestCase)write(2, " t e s t _ e l e m e n t".., 81) = 81 ... write(2, " . . . ", 5) = 5 mkdir("/tmp/tmpV5TpiJ", 0700) = 0 open64("/tmp/tmpV5TpiJ", O_RDONLY|O_NDELAY) = 3 fcntl(3, F_SETFD, 0x00000001) = 0 fstat64(3, 0xFFBED3E8) = 0 getdents64(3, 0x007CD178, 1048) = 48 getdents64(3, 0x007CD178, 1048) = 0 close(3) = 0 rmdir("/tmp/tmpV5TpiJ") = 0 okwrite(2, " o k", 2) = 2 write(2, "\n", 1) = 1 172write(2, "\r 1 7 2", 5) = 5 /824 ( 20.9%)write(2, " / 8 2 4 ( 2 0 . 9 %".., 13) = 13 : write(2, " : ", 2) = 2 test_element_with_attributes_ns (lxml.tests.test_elementtree.ETreeTestCase)write(2, " t e s t _ e l e m e n t".., 75) = 75 ... write(2, " . . . ", 5) = 5 mkdir("/tmp/tmpvCHqNz", 0700) = 0 ^Z [1]+ Stopped truss python2.4 test.py -p -vv 0 lb54320@adevp02 .../trunk $ kill -9 %+ [1]+ Stopped truss python2.4 test.py -p -vv 0 lb54320@adevp02 .../trunk $ Any hints on what might be going wrong? I didn't see anything like this with my last trunk checkout (that's when I reported a core dump for the empty schematron test case). To try and investigate further I wanted to check with an earlier 2.0alpha version, but: 0 lb54320@adevp02 .../lxml-1.3 $ svn list https://codespeak.net/svn/lxml/tag before-dom-nodereg-removal/ before-grisel-patch/ lxml-0.5/ lxml-0.5.1/ lxml-0.6/ lxml-0.7/ lxml-0.8/ lxml-0.9/ lxml-1.0/ lxml-1.0.1/ lxml-1.0.2/ lxml-1.0.3/ lxml-1.0.4/ lxml-1.0.beta/ lxml-1.1/ lxml-1.1.1/ lxml-1.1.2/ lxml-1.1alpha/ lxml-1.1beta/ lxml-1.2/ lxml-1.2.1/ lxml-1.3/ lxml-1.3.1/ lxml-1.3.2/ lxml-1.3.3/ Not all (development) releases have been tagged, e.g. 1.3.4 and also the complete 2.0alpha line. Holger -- Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kanns mit allen: http://www.gmx.net/de/go/multimessenger

2 5

[lxml-dev] iteraprse comment parsing
by kris 18 Oct '07

18 Oct '07

Hello, I noticed a difference in parsing behavior between iterwalk and iterparse. It could simply be that I do not know how to turn off comment parsing. Any help would be appreciated. Thanks, Kris python ~/xml-test.py using parse ['start', <Element r at 2aad85d0f3c0>, 'end', <Element r at 2aad85d0f3c0>] using walk ['start', <Element r at 2aad85d0f460>, 'start', , 'end', , 'end', <Element r at 2aad85d0f460>] BAD lxml.etree: (1, 1, 2, 0) libxml used: (2, 6, 27) libxml compiled: (2, 6, 27) libxslt used: (1, 1, 20) libxslt compiled: (1, 1, 19) ==================================================================================== from lxml import etree from StringIO import StringIO x = "<r> </r>" print "using parse" parser = [] for a,e in etree.iterparse(StringIO(x), events=('start','end')): parser += (a,e) print parser print "using walk" walker = [] for a,e in etree.iterwalk(etree.XML(x), events=('start','end')): walker += (a,e) print walker if parser != walker: print "BAD" print "lxml.etree: ", etree.LXML_VERSION print "libxml used: ", etree.LIBXML_VERSION print "libxml compiled: ", etree.LIBXML_COMPILED_VERSION print "libxslt used: ", etree.LIBXSLT_VERSION print "libxslt compiled: ", etree.LIBXSLT_COMPILED_VERSION -- Kristian Kvilekval kris(a)cs.ucsb.edu http://www.cs.ucsb.edu/~kris w:805-636-1599 h:504-9756

2 1

Re: [lxml-dev] Next question (Was: Beginner question)
by Frederik Elwert 16 Oct '07

16 Oct '07

Andreas Tille schrieb: > On Tue, 16 Oct 2007, Frederik Elwert wrote: > >> elif lname(elem) == 'vaccination': >> print 'vaccination status response:',\ >> elem.find('status').get('response') >> elif lname(elem) == 'therapy': >> print 'therapy status response:',\ >> elem.find('status').get('response') >> >> I think this should help. > > It sounds promissing but does not help: > > elif event == 'end' and lname(elem) == 'vaccination': > print 'vaccination:', etree.tostring(elem, > pretty_print=True) > stat = elem.find('status') > if stat != None: > print 'vaccination status response:', > elem.find('status').get('response') > else: > print 'status not found' > > > vaccination: <vaccination> > <status response="n"/> > </vaccination> > > status not found > > > Did I missed something? Probably a namespace issue, I forgot that. In a namespace aware application, you would use something like elem.find('{http://my-namespace/}status') Now since you want to avoid namespaces, it will get a bit more tricky. Maybe something like this helps: elif event == 'end' and lname(elem) == 'vaccination': print 'vaccination:', etree.tostring(elem, pretty_print=True) try: print 'vaccination status response:', elem.xpath('./*[local-name() = "status"]')[0].get('response') except IndexError: print 'status not found' Or you simply iter the children and test for lname(): elif event == 'end' and lname(elem) == 'vaccination': print 'vaccination:', etree.tostring(elem, pretty_print=True) for child in elem: if lname(child) == 'status': print 'vaccination status response:', child.get('response') I cc'd this to the list, what I forgot to do before, since there are the experts that might have better ideas. Frederik

3 2

Re: [lxml-dev] Dealing with segfaults in lxml?
by Stefan Behnel 11 Oct '07

11 Oct '07

Mike Meyer wrote: > On Wed, 10 Oct 2007 09:03:04 +0200 Stefan Behnel <stefan_ml(a)behnel.de> wrote: >> d = dict(node.items()) >> >> or something in that line, which should even be faster as it avoids the >> intermediate attrib proxy and iterator creation steps. If you wan to be more >> selective, a generator expression will do. > > I tried the node.items() variation, and that was still causing > segfaults. Then it's still different than I thought. If all you change is this line: d = dict(node.attrib) and you get segfaults with this: d = dict(node.items()) but not with this: d = dict() for key in keys: d[key] = node.get(key, '´) I really can't extract anything meaningful from that. The complete valgrind trace would be helpful. Stefan

1 0

[lxml-dev] Beginner question
by Andreas Tille 11 Oct '07

11 Oct '07

Hi, I'm sorry to start with this beginner question. Yesterday I stumbled over lxml and I think it is a really great tool which exactly is what I ever wanted but I'm afraid I need some kick start. I try to parse some XML files that are used as transport medium between different databases. We use a self defined XSD schema. The xml file lokes like this: <?xml version="1.0" encoding="ISO-8859-1"?> <envelope xmlns="http://www3.rki.de/ns/agi/ibs/2007/T06/report" xmlns:ct="http://www3.rki.de/ns/rki/base/ct/2007/T03" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www3.rki.de/ns/agi/ibs/2007/T06/report http://www3.rki.de/ns/agi/ibs/2007/T06/RKI_IBS_SVD2RKI.xsd"> <preamble> <tracking sequence="1" timestamp="2007-10-10T02:10:11"> <source idSource="NRZ Berlin"> <ct:software name="psql2xml" version="0.1"/> </source> <target idTarget="RKI"/> </tracking> </preamble> <content> <virologics elements="9"> <virologic notifyWeek="2" period="2001/2002"> <properties elements="2"> <ct:property name="ageYear" value="21"/> <ct:property name="ageMonth" value="o"/> </properties> <sender idSender="603" site="15362" class="o"/> <patient birthYear="1980" birthMonth="o" sex="w"/> <illness from="2002-01-04"> <vaccination> <status response="n"/> </vaccination> <therapy> <status response="x"/> </therapy> <contact response="n"/> </illness> <symptoms> <acute response="y"/> <fever> <status response="y"/> </fever> <cough response="n"/> <pain response="n"/> </symptoms> <complications> <pneumonia response="n"/> <hospitalisation response="x"/> </complications> <report idLab="02-0750"> <material from="2002-01-05" receipt="2002-01-09" name="R"/> <results date="2002-01-09"> <result pathogen="InvA" value="o" interpretation="ni" method="PCR"/> <result pathogen="InvB" value="o" interpretation="ni" method="PCR"/> </results> </report> </virologic> <virologic notifyWeek="2" period="2001/2002"> ... With the code that I adopted from the tutorial for event, elem in etree.iterparse(infile, events=("start")): if event == "start": print "start:", etree.tostring(elem, pretty_print=True) print "--->", elem.tag I got something like: ... start: <source idSource="NRZ Berlin"> <ct:software name="psql2xml" version="0.1"/> </source> ---> {http://www3.rki.de/ns/agi/ibs/2007/T06/report}source start: <ct:software name="psql2xml" version="0.1"/> ---> {http://www3.rki.de/ns/rki/base/ct/2007/T03}software start <target idTarget="RKI"/> ---> {http://www3.rki.de/ns/agi/ibs/2007/T06/report}source ... the elements as a whole with children on the one hand but I have no idea how to finally access the values like 'idSource="NRZ Berlin" ' nor do I have an idea how to get rid of the default name space that is prepended before the tags. I would rather like to access the tag called "source" (without the default name space) or "ct:software" with the shortcut of the name space. I also found the very interesting objectify method at http://codespeak.net/lxml/objectify.html but I finally have no idea how to use that in the parser because the page just describes creating objects (or did I missed something?) Sorry for my ignorance in case things should be obvious from reading the docs. Kind regards Andreas. -- http://fam-tille.de

3 4

Re: [lxml-dev] Dealing with segfaults in lxml?
by Stefan Behnel 10 Oct '07

10 Oct '07

Hi, ok, that wasn't the problem then (it's still good to have it fixed, though). Mike Meyer wrote: > A master process reads in a a couple of config files, and parses and > checks them against a schema, and then possibly plugs in some default > attribute values. It then forks two processes: > > 1) Uses http to get xml documents from a remote server. These are the > ones I described; they have a meta element and then a data element > containing "row "elements, with the actual values in the attributes > to "row" elements. This process uses iterparse to pull one value > from the meta element, and then saves the entire thing to disk. That's the process that fails, right? Can you find out if it's the iterparse() or something else that fails here? Using valgrind is usually a great way to find out what's going wrong. It will make the run a lot slower, but it should print some helpful infos when it crashes. Run it like this: valgrind --tool=memcheck --leak-check=no --suppressions=valgrind-python.supp \ python yourscript.py preferably only on the process that crashes. > I.e. - the only documents that gets reused a lot is the schema, which > are built, passed to RelaxNG, and then used to validate each of those > thousands of documents. That's ok. > architecture makes things a little convoluted, but the basic path is > something like: > > data = urlopen(....) > try: > parsed = fromstring(data.read()) parse(data) should do, BTW. > if not schema.validate(parsed): > handle_broken_document(parsed=parsed) > for node in parsed.xpath('Types/Type'): > d = dict(node.attrib): > save_for_db(d) > for node in parsed.xpath('AltTypes/AltType'): > d = dict(node.attrib): > save_for_db(d) > for node in parsed.xpath('MoreTypes/MoreType'): > d = dict(node.attrib): > save_for_db(d) That's pretty straight forward code, I don't see any risk here. But I'm wondering which of the two processes actually fails now - you're presenting this one, but from your previous posts I though it was the other one that crashed. > I tried turning of the parsing - which pretty much makes everything > else do nothing but pass around the raw data - and got no failures. I > also tried turning off just the validation, so that the work is still > getting done - and got failures. Hmmmm, are those failures related to validation errors? Just in case it's the second process that fails (the XPath one), it could be worth testing if using the XPath() class instead of the xpath() method works better. That might give us a hint on where the problem comes from. It should also be faster, BTW. Stefan

2 2