Mailman 3 November 2005 - lxml - The Python XML Toolkit

[lxml-dev] lxml branch
by Godefroid Chapelle 24 Jan '06

24 Jan '06

Hi Martijn, list I have checked code in a branch that allow to use a different parser than the default globally shared. This allows to setup different options than the default. All previous tests (321 ?) do pass. I'd be pleased to get feedback about the code before working on the supplementary tests needed. I am not totally sure I went the right way even if it works for me... Cheers -- Godefroid Chapelle (aka __gotcha)- BubbleNet http://bubblenet.be

4 17

[lxml-dev] the reason for _fakeRootDoc?
by Martijn Faassen 25 Nov '05

25 Nov '05

Hi there, I'm trying to understand why in scoder2 there's the use of _fakeRootDoc in a few places. I imagine this is to work around some bugs that show up in the trunk, but I don't have enough context on what and why. Stefan, can you enlighten me? Any tests that expose this behavior in particular? Regards, Martijn

2 8

[lxml-dev] bug
by Jan Procházka 23 Nov '05

23 Nov '05

Hi, i probably found bug: after this >>> from lxml import etree >>> from StringIO import StringIO >>> etree.parse(StringIO('')) python under windows xp falls (no exception, probably access violation) Honza Prochazka

2 2

[lxml-dev] lxml 0.8 on windows fails tests
by Brad Clements 22 Nov '05

22 Nov '05

Hi, I have lxml 0.8 compiled with the latest windows binaries from http://xmlsoft.org/sources/win32/ python selftest.py passes, but python test.py fails oddly. E:\prj\src\lxml-0.8>python test.py Traceback (most recent call last): File "test.py", line 591, in ? exitcode = main(sys.argv) File "test.py", line 554, in main test_cases = get_test_cases(test_files, cfg, tracer=tracer) File "test.py", line 254, in get_test_cases module = import_module(file, cfg, tracer=tracer) File "test.py", line 197, in import_module mod = __import__(modname) File "E:\prj\src\lxml-0.8\src\lxml\tests\test_errors.py", line 9, in ? from lxml import etree ImportError: cannot import name etree But if I run by hand, it works: E:\prj\src\lxml-0.8>python Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree >>> ^Z Is it possible that test.py is doing something with the pythonpath that might contribute to this oddness? -- My site-packages/lxml directory looks like this: Directory of D:\Python24\Lib\site-packages\lxml 11/19/2005 04:37 PM <DIR> . 11/19/2005 04:37 PM <DIR> .. 11/19/2005 04:30 PM 163,840 etree.pyd 10/25/2003 01:31 PM 888,832 iconv.dll 05/03/2005 08:37 PM 102,400 libexslt.dll 08/06/2005 04:22 PM 898,560 libxml2.dll 05/03/2005 08:37 PM 282,624 libxslt.dll 11/19/2005 04:34 PM <DIR> tests 08/06/2005 04:15 PM 73,728 zlib1.dll 07/07/2005 08:04 AM 5,956 _elementpath.py 11/19/2005 04:34 PM 3,591 _elementpath.pyc 07/07/2005 08:04 AM 21 __init__.py 11/19/2005 04:34 PM 126 __init__.pyc I would suspect a .dll load dependency problem, but PATH and CWD is the same in both cases, so .. stumper. -- Also, and this could be a clueless question, could lxml build off the python libxlm binary package on windows, so that I wouldn't have two copies of libxml2.dll and friends kicking around.. ? I suppose not, since Pyrex would be "upcalling" to the python interface exposed by the libxml python bindings.. I guess libxml directly calls libxml C interface.. ? -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM or SKYPE: BKClements

5 11

[lxml-dev] SAX support in trunk
by Martijn Faassen 19 Nov '05

19 Nov '05

Hi there, In the past few days I've done some reviewing and editing of the SAX support for lxml. As a result I've now added a sax.py module to the trunk that contains the SAX support. There's a doc/sax.txt document (and doctest) that explains how to use things. Thanks Stefan for this contribution! Your contributions as well as patience is much appreciated! Now on to the rest... :) I have been doing some more reviewing and thinking of what's on the scoder2 branch so I expect more progress to occur next week. Regards, Martijn

2 2

[lxml-dev] TracOS.org hosting
by Andrey Tatarinov 19 Nov '05

19 Nov '05

Hi, As I see getting trac or something similar on Codespeak.net is a long time task, I suggest to migrate lxml development to TracOS.org, it gives complete Trac installiation (which is Wiki, Bug-tracker and has Svn integration), I think that will boost development and what is more important - documenting. Martjin?

4 4

[lxml-dev] SAX review (part 1)
by Martijn Faassen 15 Nov '05

15 Nov '05

Stefan Behnel wrote: > I'd finally like to get my "bigfat.patch" a bit smaller. The main > additions that are still pending are these four: > > * SAX I/O Okay, starting to take a look at your SAX work now, giving my feedback, thoughts and questions as I go. I understand that you implemented a ElementTree builder that consumes Python SAX events first, and then later on you created a Python-style SAX event producer on top of the ElementTree API. Very cool! I have quite a bit of SAX code lying around that may eventually benefit from this. It'd be nice to eventually have a Python SAX producer on top of libxml2's SAX2 APIs directly as well. First a few small Python style issues I noticed: * I want to stick with a character limit per line of 80 characters. Longer lines should be broken up. If they can be broken 'naturally' with a newline due to () and such that's preferable to the use of the backslash. * I see you paired for instance startElementNS and endElementNS together without an empty line between them. I'd like to stick to empty lines between all method definitions. * I prefer the use of the unicode literal u'' to a barebones unicode() call. I think it's a bit clearer to use the literal for this. Now on to more details of the code, focusing on the SaxTreeBuilder class for now (I'll look at saxify later). * I presume that due to the namespace reconciliation when adding elements repeating the ns_mapping each time you create an element is safe and won't result in a lot of namespace definitions in the serialized XML, correct? * This code is *almost* compatible with ElementTree proper, except for the namespaces bit. For future potential reusability it might be nicer indeed to put it in a separate module instead of integrating it into etree.pyx. * Hm, wouldn't characters() be a bit nicer like this? (untested) parent = self._element_stack[-1] try: # the text comes after another element, thus tail element = parent[-1] element.tail = (element.tail or u'') + data except IndexError: # the text is before any element in the parent, thus text parent.text = (parent.text or u'') + data * while this shouldn't matter normally, it might be nice to stick with the argument names for the content handler defined in the Python library, like name and qname. * I prefer tuple unpacking of the 'tag_tuple' ('name'), like this. I also don't think it's necessary to have a template for such a simple thing as "{%s}%s"; repeating it a few times is actually less text than using the template... No need to make it a unicode string either, though it doesn't matter much in this case. (It would matter more for plain ElementTree -- this stores ascii strings where possible, so *not* forcing things into unicode might help it slightly, though I think SAX delivers unicode always so it wouldn't matter after all..). Also look at the bug I think I found (see comment). ns_uri, localname = name and then: if ns_uri: el_name = '{%s}%s' % (ns_uri, localname) elif self._default_ns: # order seems reversed in your code. Is this a bug or my # misunderstanding? Would be nice to have a test which # exercises this code! el_name = '{%s}%s' % (self._default_ns, localname) else: el_name = localname Anyway, going home now and not entirely done with the review yet. Hope this was useful! Lots of this was about minor matters of taste and style so far, but it's good to have a uniform style in lxml... Regards, Martijn

2 5

[lxml-dev] xslt transformators
by Andrey Tatarinov 15 Nov '05

15 Nov '05

Hi, Just an interesting thing to think of: I think that it would be pythonic to treat xslt transformations just as other functions, for example: >>> tr1 = lxml.etree.xslt_fromfile('tr1.xsl') >>> tr2 = lxml.etree.xslt_fromstring('''<?xml ... ...  ... ''') >>> data = lxml.etree.parse('data.xml') # maybe .fromfile() ? >>> data1 = tr1(data) >>> data2 = tr2(data) >>> data12 = tr2(tr1(data)) What do you think?

3 9

[lxml-dev] helper functions for accepting elementtrees/elements/documents as input
by Stefan Behnel 15 Nov '05

15 Nov '05

Hi! I noticed that many of the API classes accept _ElementTree object, although it would make just as much sense to apply them to elements or (internally) to documents. So here are three helper functions that you can use at the top of API functions to check what actually came in and to determine how to extract the document that is refered to and the context node within the document. This simplifies clumsy expressions like XSLT(ElementTree(XML("..."))) and the like to XSLT(XML("...")) I checked them into my branch, although they may need modification later, depending on Geerts work. @Geert: I think these could be of interest to you if you extend them to check for filename strings and file-like objects. This may encourage writing a new function that returns both document and node at once to facilitate on-the-fly parsing. Stefan Index: src/lxml/etree.pyx =================================================================== --- src/lxml/etree.pyx (Revision 19828) +++ src/lxml/etree.pyx (Arbeitskopie) @@ -1093,6 +1093,36 @@ # Private helper functions +cdef _Document _documentOrRaise(object input): + cdef _Document doc + doc = _documentOf(input) + if doc is None: + raise TypeError, "Invalid input object: %s" % type(input) + else: + return doc + +cdef _Document _documentOf(object input): + # call this to get the document of a + # _Document, _ElementTree or _NodeBase object + if isinstance(input, _ElementTree): + return (<_ElementTree>input)._doc + elif isinstance(input, _NodeBase): + return (<_NodeBase>input)._doc + elif isinstance(input, _Document): + return <_Document>input + else: + return None + +cdef _NodeBase _rootNodeOf(object input): + # call this to get the root node of a + # _Document, _ElementTree or _NodeBase object + if hasattr(input, 'getroot'): # Document/ElementTree + return <_NodeBase>(input.getroot()) + elif isinstance(input, _NodeBase): + return <_NodeBase>input + else: + return None + cdef xmlDoc* _fakeRootDoc(xmlDoc* c_base_doc, xmlNode* c_node): # build a temporary document that has the given node as root node # note that copy and original must not be modified during its lifetime!!

2 3

[lxml-dev] RelocatableRelaxNG
by Stefan Behnel 14 Nov '05

14 Nov '05

Hi! Anyone annoyed by the fact that RNG grammars need a fixed starting point? That you can't just validate some subtree that you're interested in? Well, here we go. Stefan Index: src/lxml/relaxng.pxi =================================================================== --- src/lxml/relaxng.pxi (Revision 19820) +++ src/lxml/relaxng.pxi (Arbeitskopie) @@ -59,3 +59,53 @@ if ret == -1: raise RelaxNGValidateError, "Internal error in Relax NG validation" return ret == 0 + +cdef object _build_relocation_stylesheet(): + return XSLT(ElementTree(XML('''\ +<xsl:stylesheet version="1.0" + xmlns:xsl="http://www.w3.org/1999/XSL/Transform" + xmlns:rng="http://relaxng.org/ns/structure/1.0"> + <xsl:template match="rng:start/rng:ref"> + <xsl:copy> + <xsl:attribute name="name"> + <xsl:value-of select="$newref"/> + </xsl:attribute> + </xsl:copy> + </xsl:template> + <xsl:template match="text()"> + <xsl:value-of select="normalize-space()"/> + </xsl:template> + <xsl:template match="*"> + <xsl:copy><xsl:copy-of select="@*"/><xsl:apply-templates /></xsl:copy> + </xsl:template> +</xsl:stylesheet> +'''))) + +class RelocatableRelaxNG: + def __init__(self, tree, start=None): + self._tree = tree + if start is not None: + start = "'%s'" % start + self._start = start + try: + self.__class__._RELOCATION_XSLT + except AttributeError: + self.__class__._RELOCATION_XSLT = _build_relocation_stylesheet() + + def validate(self, xml_tree): + if self._start is None: + rng_tree = self._tree + else: + rng_tree = self._RELOCATION_XSLT.apply(self._tree, + newref=self._start) + rng = RelaxNG(rng_tree) + self.validate = rng.validate # replace this method by the real thing + return rng.validate(xml_tree) + + def copy(self, start=None): + return self.__class__(self._tree, start) + + def relocate(self, start): + self._start = start + try: del self.validate + except AttributeError: pass Index: src/lxml/tests/test_relaxng.py =================================================================== --- src/lxml/tests/test_relaxng.py (Revision 19831) +++ src/lxml/tests/test_relaxng.py (Arbeitskopie) @@ -111,7 +111,39 @@ self.assertEqual(self._rootstring(b_tree), 'B') self.assert_(schema.validate(b_tree)) + def test_relaxng_relocation(self): + schema = self.parse('''\ +<grammar xmlns="http://relaxng.org/ns/structure/1.0"> + <start><ref name="a"/></start> + <define name="a"> + <element name="a"> + <ref name="b"/> + </element> + </define> + <define name="b"> + <element name="b"> + <ref name="c"/> + </element> + </define> + <define name="c"> + <element name="c"> + <empty/> + </element> + </define> +</grammar> +''') + schema1 = etree.RelocatableRelaxNG(schema) + tree1 = self.parse('<a><c/></a>') + self.assert_(schema1.validate(tree1)) + + schema2 = schema1.copy(start='b') + tree2 = self.parse('<c/>') + self.assert_(schema2.validate(tree2)) + self.assert_(schema1.validate(tree1)) + self.assertFalse(schema1.validate(tree2)) + self.assertFalse(schema2.validate(tree1)) + def test_suite(): suite = unittest.TestSuite() suite.addTests([unittest.makeSuite(ETreeRelaxNGTestCase)])

2 2

lxml - The Python XML Toolkit November 2005