Mailman 3 August 2007 - lxml - The Python XML Toolkit

[lxml-dev] Keep track of file URL in Elements
by Frederik Elwert 26 Aug '07

26 Aug '07

Hello! XForms turns out to be quite challenging... Sorry to bother you again. I need to access a document given by a link in an attribute. When using XSLT, this would be a typical usecase for document(). But when not using XSLT, I need to mimic it's behaviour, since document() doesn't work in ordinary XPath expressions. So I have something like this in an XML document: <instance resource="default-scales.xml"/> Since the link is relative to the document's location, and not to the current working directory, I need to get some absolute path for this. But the information for the document's location is only stored in the ElementTree initially generated by etree.parse(). The descendants know neither about the document's location, nor about their "parent" ElementTree, as far as I could see. So would it be anyhow possible to get the document's URL information in a custom xpath function? The only thing I could think of is searching for ElementTrees using dir(), seeing if the current element is in that tree using getpath(), and then using the trees docinfo. But that's really ugly. Maybe another possibility would be to use a custom Element class and add an attribute containing this information to each element during parsing the XML file. But (a) I'm not sure if this is possible at all, and (b) all elements would have a new, arbitrary attribute that I would have to get rid of before writing the tree back. So maybe anyone of you has an idea how this could be done using lxml? Kind regards, Frederik

2 1

[lxml-dev] Possible 'freeing null pointer' problem with 'document()' function
by Sidnei da Silva 24 Aug '07

24 Aug '07

Hi there, Just a heads up in case anyone is facing similar issues. I have written a Apache output filter using lxml and it has been working fine so far, but after some changes to a xsl file it started crashing Apache. I've tracked down the issue to two variable definitions that were using the 'document()' function to include a external xml file. <xsl:variable name="geographic_vocabularies" select="document('../vocabularies/geographic_vocabularies.xml')"/> <xsl:variable name="workspace_vocabularies" select="document('../vocabularies/workspace_vocabularies.xml')"/> I wrote a script that performs the same transformation outside of Apache, and surprisingly it works just fine. The only difference being that the script resolves to a file in the filesystem, and the filter in Apache uses urlopen(). I'm testing this on Windows, and when the crash happens and I select debug the current line is in free() and the previous item in the stack is etree.pyd, but I couldn't get a debug build going to see where exactly in etree.pyd. This happens with both lxml 1.2.1 and 1.3.3, using libxml2 2.6.28 and libxslt 1.1.19. Any clues appreciated... -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

2 7

[lxml-dev] Context of custom XPath functions
by Frederik Elwert 24 Aug '07

24 Aug '07

Hello! Currently, I'm trying to implement some of the XForms functions with lxml. The most urgent one is instance(), which gets the instance data from an XForms model. But if you're not familiar with XForms, don't bother, just think of id() or something similar: With id(), it is necessary to access the context of the function call, i.e., the tree in which it is called. id() is supposed to return an element node (or node-set) matching the given argument. So id() has to know about the tree, and not only about it's given argument. So how would one implement id() or any similar function with lxml? As far as I got with custom functions, they only can handle the arguments they get passed direcly, but they don't know about the broader context. Any suggestions how I could solve this? Thanks in advance, Frederik

2 7

Re: [lxml-dev] Problem with ":" char in tag names
by Martijn Faassen 20 Aug '07

20 Aug '07

[whoops reply only sent to Stefan while I meant to include the list] Hey Stefan, I agree that this is a bugfix; sorry for the confusion. I never tried to use namespace prefixes this way and used Clarke notation consistently. My feedback is not coming from the perspective of supporting broken use, but wondering whether we cannot make lxml easier to use. On 8/17/07, Stefan Behnel <stefan_ml(a)behnel.de> wrote: [snip in the past people used to be able to construct programs that looked like they produced correct XML but actually didn't] > So raising an exception here *really* prevents a lot of pitfalls and helps > people fix their programs. Okay, it is clear that the previous behavior was at best undefined, so raising an error is a good idea. It does indicate one thing though - if we *wanted* to write a feature that explictily used namespace prefixes, we could, as it's certainly not doing anything else. :) > > I wonder whether it'd be possible to support namespace prefixes the > > proper way this way. I.e if I write: > > > > Element('foo:bar', nsmap={'foo': 'blah}) > > > > that could be equivalent to: > > > > Element('{blah}bar', nsmap={'foo': 'blah'}) > > No. There should be one way to do this. We already use prefixes in XPath, > which causes a lot of annoyance for new users. Yes, and prefixes are used in the XML serialization. The way we have both prefixes and Clarke notation *already* creates a lot of confusion for users. In addition, the Clarke notation pattern forces one to write code like this: SubElement(el, '{%s}foo' % MY_NS) > BTW, this is an extremely rare use pattern. Normally, you would either work on > an XML document that already comes with its pre-defined prefixes, or you would > define an nsmap once (as you show above) and then stick to using > SubElement(..., "{ns}tag") without redefining the prefixes. Yes, but this is a very common pattern: SubElement(el, '{%s}foo' % MY_NS) i.e. people generally don't want to spell out their entire namespace URI over and over again when constructing XML. Therefore I started to wonder whether we could create a convenience that uses namespace prefixes *and* does the right thing: SubElement(el, 'myns:foo') will work *if* myns has been defined as a namespace prefix in the context of 'el'. Of course, accessing tags through .tag would still return Clarke notation. We can have various objections against this. We can for instance say, this is a bad idea as it's it is surprising behavior. After all, if you set a tag and then get it, you'll get something else back. Then again, since XML parsing already has this behavior and thus the user will have to be familiar with it anyway, I don't think it's that surprising in the end. It might actually be a useful convenience that will make some code look cleaner. Another objection is that we should have only one way to do it. But we don't, really. In order to set namespaces for elements, we currently have a number of ways to do arrange your code. One is to use the "%s" pattern. Another is to use a custom factory specific to your codebase. And we *already* have two ways to get namespace information into the application - through namespace prefixes in the parser, and through Clarke notation in the API. > Note that lxml nicely reassigns prefixes now when inserting an element into an > existing tree, so there really is no need to assign prefixes more than once > (if at all). Assigning prefixes, sure. *Using* prefixes is what I'm talking about. > > The nice thing is that you could avoid having to write '{%s}foo' % > > my_namespace a lot. > > Feel free to assign it to a global constant or to use the E factory as in > lxml.html.builder. Yes, remember that I've used lxml before. :) I often use a global constant. It still means I scatter "{%s}foo" % MY_GLOBAL_CONSTANT throughout my code. Meanwhile, I *already* have a "global constant" that I also set somewhere, in the XML, namely my namespace map. I can of course create my own factory, which I've also frequently done. That runs the risk of obscuring otherwise clear use of the lxml API. (then again, the application's concerns may often force a factory on the developer anyway). > > Of course this has consequences for other areas, such as 'tag', so I'm > > not sure whether this is a good idea, but throwing it in. > > Right, it would let ".tag" return something other than what you passed into > the Element() function. Yes. If we make this change, we'd also need to figure out what happens if you explictily *set* tag. Should we allow: foo.tag = 'foo:bar' allowing potentially inconsistent behavior as you can set something and then get back something else in Clarke notation, or still forbid it? > > It's definitely another extension on ElementTree, which can't really do > > this kind of stuff well due to the lack of parent pointers. > > Right, so it would unnecessarily add an additional namespace definition > pattern that is not supported by ET and at the same time allow the pitfalls > that the users who reported the problem currently run into. Meaning: it would > let people write programs that would stop working the day they wanted to > switch to ET or the day they started using XPath. Great. I think there are two potential drawbacks: * allow users to write programs that will stop working as soon as they switch back to ET. This is a drawback. It's also a drawback that already exists - we have many many extensions above the ElementTree API and people's programs will stop working if they don't stick to the common subset. * allow users to write programs that stop working when they switch to XPath. I don't understand why you say this. Of course it's broken *now*. I'm not advocating the current broken behavior at all, and support disabling undefined behavior. I'm just wondering whether we shouldn't support this behavior explicitly *and do the right thing*. We can still raise an exception as soon as someone uses an undefined namespace prefix, of course. > No, this change is definitely a bug fix. I'm sorry for people who were not > aware of this bug in the past and accidentally misused it, but this has to change. Sorry for the confusion in my original reply. I didn't mean to say the bugfix should be rolled back. It's indeed a bugfix and I support it. It just led me to think we might have an opportunity there for improving our API. Regards, Martijn

2 3

[lxml-dev] lxml.html landed
by Stefan Behnel 18 Aug '07

18 Aug '07

Hi all, I finally merged the lxml.html branch into the trunk. This includes the package lxml.html, the module lxml.cssselect, and the XML/HTML doctest support. All were written by Ian Bicking, with only a couple of minor changes and comments by myself. Thanks Ian! One more thing I changed for the merge was to let get_element_by_id() raise an exception if no default is passed (like getattr() does). That way, you can be explicit about what you want: None or an exception. I took another quick look at the parser functions and I think they look OK now. The alpha cycle of lxml 2.0 will hopefully tell us if they work as people expect. I'll try to take another look at the pending objectify changes by Holger. Once I get through them, the time for the first alpha release of lxml 2.0 has come. You can expect it at the end of august. So, a call to all contributors and those who want to help out: *Please help out looking through the new APIs and the trunk documentation!* There were loads of changes and the new modules could well benefit from a couple of additional paragraphs in the docs. As usual, the HTML version is at http://codespeak.net/lxml/dev/ Thanks for the great code, Ian! Stefan

1 0

[lxml-dev] Problem with ":" char in tag names
by Dave Kuhlman 18 Aug '07

18 Aug '07

I've been using lxml and think it is great, but ... I recently installed lxml-1.3.3. Now I find that the following gives me an error: In [3]: from lxml import etree In [4]: etree.Element('abc:def') ------------------------------------------------------------ Traceback (most recent call last): File "<ipython console>", line 1, in <module> File "etree.pyx", line 1801, in etree.Element File "apihelpers.pxi", line 101, in etree._makeElement File "apihelpers.pxi", line 723, in etree._getNsTag ValueError: Invalid tag name It's because of the ":" in the tag name. That's critical for me, because I use lxml in my rst2odt project to produce OpenOffice ODF .odt files. See: http://www.rexx.com/~dkuhlman/odtwriter.html An ODF/.odt file is a zipped archive of XML files. Those XML files contain many tags that contain colons. Here are the relevant portions of the XML spec, I believe: http://www.w3.org/TR/2006/REC-xml11-20060816/#sec-starttags http://www.w3.org/TR/2006/REC-xml11-20060816/#NT-Name Aren't I correct that a colon should be allowed in a tag name? In apihelpers.pxi, it looks like the following lines were added in lxml version 1.3.3 and which I believe are raising the exception: elif cstd.strchr(c_tag, c':') is not NULL: raise ValueError, "Invalid tag name" Is there a reason for that? Hoping for enlightenment. Dave -- Dave Kuhlman http://www.rexx.com/~dkuhlman

4 8

[lxml-dev] Problem with xpath behaviour.
by Bruno Deferrari 14 Aug '07

14 Aug '07

Hi, I'm using lxml 1.3.3, libxml2 2.6.29 and python 2.5.1, and I'm getting a weird behaviour when using xpath, for example: >>> from lxml import etree >>> html = '<html><body>text1text2text3text4text5</body></html>' >>> doc = etree.HTML(html) >>> etree.tostring(doc.xpath('//p')[0]) 'text2text3' Shouldn't I be getting just 'text2' ?

2 3

[lxml-dev] Fwd: Round Trip w/ Processing Instructions (Newbie)
by Greg Willden 13 Aug '07

13 Aug '07

It looks like this message didn't go to the list. ---------- Forwarded message ---------- Hi Stefan, On 8/9/07, Stefan Behnel <stefan_ml(a)behnel.de> wrote: > > > I see that t1.docinfo.doctype contains: > > <!DOCTYPE project SYSTEM "test.dtd"> > > > > So why isn't the PI written to the file? > > Hmm, the problem here is that libxml2 lacks a good backwards compatible > way of > serialising documents including things like processing instructions, > internal > subsets (DTDs) and doctype declarations (which you have here, it's not a > PI). > > I'll look into it (once again). IIRC, we serialise PIs by now, but not > internal subsets and doctypes. We should do that when serialising root > nodes > (especially ElementTree objects). Sorry I've got my terminology mixed up. Obviously I'm not an XML expert. Just enough to be dangerous ;-) Thanks for looking into it. I searched the mailing list to see if this stuff had come up before and I had seen a lot of similar questions so I'm sorry if it's a FAQ or something. You mention serializing root nodes. That brings up another question I had along these lines. Given the same XML file: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE project SYSTEM "test.dtd"> <project><a></a></project> If I do an etree.parse('test.xml') I get an ElementTree object but If I were to have that same data as a string and call etree.XML() or etree.fromstring() then all I get is an Element. That seems really odd to me. If this question has been discussed (I'm guessing it has), I'd be happy with an answer like "take a look at this thread for the discussion". I guess it's probably more of an ElementTree API question/confusion than a problem with lxml.etree though. Thanks again for your work. Greg -- Linux. Because rebooting is for adding hardware.

2 1

[lxml-dev] schema typifier for objectify
by jholg＠gmx.de 11 Aug '07

11 Aug '07

Hi, I'm interested in having a "schema typifier" for lxml.objectify that can enrich an instance document with type information taken from an XML Schema (xsi:type and/or py:pytype). Similar to the annotate()/pyannotate() stuff, but making use of the schema type definitions. I'm wondering how one realized such thing best: Is there a possibility to make use of the lxml/libxml2 validation capabilities, i.e. to somehow hook this into the validation proceedings? Any hints welcome, Holger -- Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten Browser-Versionen downloaden: http://www.gmx.net/de/go/browser

2 1

[lxml-dev] Round Trip w/ Processing Instructions (Newbie)
by Greg Willden 09 Aug '07

09 Aug '07

Hello all, I love the etree stuff and I'm excited about the API additions that this team is adding to make it better. Thanks for all your work. That said I'm really new to ElementTree. I am using lxml.etree version 1.3.3 on both Python 2.4.3 and 2.5.1. I have a question about processing instructions and round trip file processing. Let's say I have a file 'test.xml' containing the following <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE project SYSTEM "test.dtd"> <project><a></a></project> Then I run these commands t1=etree.parse('test.xml') t1.write('testout.xml', xml_declaration=True) testout.xml looks like this: <?xml version='1.0' encoding='ASCII'?> <project><a></a></project> I see that t1.docinfo.doctype contains: <!DOCTYPE project SYSTEM "test.dtd"> So why isn't the PI written to the file? I am parsing/modifying a file for consumption by another program that refuses to load the file without the PI. Thanks Greg -- Linux. Because rebooting is for adding hardware.

2 1