Mailman 3 September 2007 - lxml - The Python XML Toolkit

[lxml-dev] lxml + mod_python: cannot unmarshal code objects in restricted execution mode
by Dmitri Fedoruk 15 Sep '07

15 Sep '07

Hello everyone, I'm developing a mod_python application that is based on XML\XSLT transforming. I used 4Suite libraries for that, but as the speed was unacceptable for me, I switched to lxml. Everything became much easier and 10 times faster, but I've encountered the subject problem. In brief - all my data and xslt are stored and transferred in UTF-8. With 4Suite everything was fine all the time. With lxml it works fine from the console, but inside mod_python it occasionaly dies, ~ one time out of three. Strange - the same code with the same data works or dies by its own means. As far as I have found, there was a similar problem with PyXML and encodings module, this is the problem with UTF, but there was no clear solution. So, my configuration is the following: Python 2.5.1 Server version: Apache/2.2.4 (FreeBSD) mod_python-3.3.1 And the relevant parts of my code are these: def extApplyXslt(xslt, data, logger ): try: strXslt = urllib2.urlopen(xslt).read() # i have to read the xslt url to the python string except urllib2.HTTPError, e: ....... except urllib2.URLError, e: ............. try: xslt_parser = etree.XMLParser() xslt_parser.resolvers.add( PrefixResolver("XSLT") ) # and now I have to use the string; a more elegant solution, anyone? f = StringIO(strXslt) xslt_doc = etree.parse(f, xslt_parser) # and here where the problem comes transform = etree.XSLT(xslt_doc) except Exception, exc: logger.log(logging.CRITICAL, exc.__str__() ) try: result_tree = transform(data) return etree.tostring(result_tree, 'utf-8') except Exception, exc: print "xslt processing error!", exc.__str__() return "" It dies with the message 'cannot unmarshal code objects in restricted execution mode'. By profiling I detected the point where problem occurs: transform = etree.XSLT(xslt_doc) So, I would be grateful for any suggestions how to get rid of this. I'd really like to use lxml. Maybe I should initialize the xslt processor in somehow other way? Thanks in advance, Dmitri

4 10

[lxml-dev] 2.0alpha too visible
by Ian Bicking 15 Sep '07

15 Sep '07

I think the 2.0alpha release might be too visible. If you do "easy_install lxml" you get that version. One way to help this would be to not upload 2.0alpha to PyPI, but instead just put a link to a tarball with #egg=lxml-twoalpha or something, so it won't be considered newer than 1.3 (but you could install it with easy_install lxml==twoalpha). -- Ian Bicking : ianb(a)colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers

3 2

Re: [lxml-dev] lxml + mod_python: cannot unmarshal code objects in restricted execution mode
by Stefan Behnel 13 Sep '07

13 Sep '07

... just forwarding to the list ... [original mail by Dmitri Fedoruk] On 9/13/07, Stefan Behnel wrote: > Looks like you forgot to mention the lxml version you are using. The most important thing lxml-1.3.4 > As I already mentioned on c.l.py, you can pass the result of urlopen() > directly into parse(). Thank you, that looks better. > Hmmm, I can't see where any "unmarshaling" should be taking place here - > definitely not in XSLT(). And I don't get why this should only happen once in > a while. The point is that it than happens again and again, but I can't see any regularity. Pretty random. Here is the real code and it's profiling output: try: xslt_parser = etree.XMLParser() xslt_parser.resolvers.add( PrefixResolver("XSLT") ) inLogger.log(logging.INFO, "parser created" ) xslt_doc = etree.parse( urllib2.urlopen(xslt) , xslt_parser) inLogger.log(logging.INFO, "%s parsed" % xslt ) transform = etree.XSLT(xslt_doc) inLogger.log(logging.INFO, "xslt transformation created" ) except Exception, exc: inLogger.log(logging.CRITICAL, exc.__str__() ) logging output: Thu, 13 Sep 2007 19:53:31 INFO parser created Thu, 13 Sep 2007 19:53:31 INFO http://***/web-out-long.xsl parsed Thu, 13 Sep 2007 19:53:31 CRITICAL cannot unmarshal code objects in restricted execution mode As there is no "xslt transformation created" line, that's why I had to assume that the error happens in etree.XSLT . > Can you figure out what is writing this message? The python interpreter or > mod_python? mod_python . The python interpreter runs fine with it, not a single error. Dmitri

1 0

[lxml-dev] Serialization with namespaces
by Anders Bruun Olsen 13 Sep '07

13 Sep '07

Hi, I need to chop up some XML based on XPath expressions and serialize the resulting chunks individually. I thought LXML would be perfect for this task but have run into some problems. Here is the sample I use, test.xtm: <?xml version="1.0" encoding="UTF-8"?> <topicMap xmlns="http://www.topicmaps.org/xtm/1.0/" xmlns:xlink="http://www.w3.org/1999/xlink" id="personnavnereg1"> <topic id="abeleHenriksdatter"> <instanceOf> <topicRef xlink:href="person-template.xtmp#kvinde"/> </instanceOf> <baseName> <baseNameString>Abele Henriksdatter i Radsted, Gotfred Bangs hustru</baseNameString> </baseName> <occurrence> <instanceOf> <topicRef xlink:href="DDref.xtmp#dato1407.01.06"/> </instanceOf> <resourceRef xlink:href="http://xxx/diplomer/07-002.html"/> </occurrence> </topic> </topicMap> First I parse the file and grab the root: >>> from lxml import etree >>> tree = etree.parse("test.xtm") >>> root = tree.getroot() >>> root.nsmap {None: 'http://www.topicmaps.org/xtm/1.0/', 'xlink': 'http://www.w3.org/1999/xlink'} Then I do a little XPath magic: >>> find_topics = etree.ETXPath("//{%s}topic" % root.nsmap[None]) >>> elem = find_topics(root)[0] >>> elem <Element {http://www.topicmaps.org/xtm/1.0/}topic at 2b8501aa37e0> Now the problem occurs when I try to serialize. When I serialize the root, everything looks fine: >>> etree.tostring(root, pretty_print=True) '<topicMap xmlns="http://www.topicmaps.org/xtm/1.0/" xmlns:xlink="http://www.w3.org/1999/xlink" id="personnavnereg1"> ... The XML Namespace is applied as it should. However on the topic-element that I found using XPath no XML Namespace is output: >>> etree.tostring(elem, pretty_print=True) '<topic id="abeleHenriksdatter">\n\t\t<instanceOf>\n\t\t\t<topicRef ... Even though the nsmap attribute is set correctly: >>> elem.nsmap {None: 'http://www.topicmaps.org/xtm/1.0/', 'xlink': 'http://www.w3.org/1999/xlink'} I realize this might be because the element is not the root of the current document. How can I make LXML output the xmlns in this case? -- Anders

2 8

Re: [lxml-dev] findall() returns an iterable instead of a sequence in ET 1.3
by Stefan Behnel 13 Sep '07

13 Sep '07

Fredrik Lundh wrote: >> I just noticed the above when I tried to copy over the new ElementPath >> implementation from the current ET 1.3 SVN. The current ET docs of 1.2 clearly >> state that findall() returns a sequence. I'm not questioning the new >> behaviour, but it's not even mentioned in your "ET 1.3 intro" text. > > That's probably because I dropped in the new (still pretty rough) path > implementation after I wrote the first episode, but before I uploaded > the code... > > The ET 1.2 documentation does indeed say that findall may return a > sequence *or* an iterator: > > http://effbot.org/zone/pythondoc-elementtree-ElementTree.htm#elementtree.El… Ok, but this page doesn't (and I find it pretty visible): http://effbot.org/elementtree/elementtree-element.htm#tag-ET.Element.findall > but as you say, chances are that people are relying on behaviour > rather than implementation. Yet, it's pretty nice to have an iterator > for things like: > > for elem in tree.findall(simple pattern): > check elem properties > if right elem: > break I think the "findALL()" makes it sound like something that returns a sequence rather than an iterator. The API shouldn't work against people's intuition. > But maybe we could provide an "iterfind", perhaps? (that may or may > not be the same thing as findall). I would definitely prefer that, and I like the name already. Then, findall() could be as simple as return list(self.iterfind()) And it /should/ do the same as findall(), as it carries "find" in its name. It meats my intuition that findall() and iterfind() return exactly the same results, just in the expected different ways. > fwiw, I've had the same concerns wrt the iter/getiterator changes; in > 1.2, getiterator returned a list, not an iterator. in 1.3a3, it's an > alias for "elem.iter()". maybe it should be an alias for > "list(elem.iter())" instead? I think that's different as people do not generally expect something called "getiterator" to return a sequence, so as long as they don't look into the documentation, they would not easily use it in a way that breaks after the change. BUT, since you already deprecated getiterator() anyway, why not make it a pure legacy function that works as it did in the early days? Stefan

1 0

[lxml-dev] findall() returns an iterable instead of a sequence in ET 1.3
by Stefan Behnel 13 Sep '07

13 Sep '07

Hi Fredrik, I just noticed the above when I tried to copy over the new ElementPath implementation from the current ET 1.3 SVN. The current ET docs of 1.2 clearly state that findall() returns a sequence. I'm not questioning the new behaviour, but it's not even mentioned in your "ET 1.3 intro" text. Don't you think that change will break a lot of code out there? It already breaks a couple of places in lxml.html, e.g. code where the author knew that there were few results to expect (and thus a list was the perfect thing to return) and where it is was convenient to test for the truth value of the returned list to check for results. Admittedly, it's easy to write list(el.findall()), but the thing is: a) someone has to do that, and b) it's not always the best solution, so the change requires people to rethink their code. And the worst is: you will not even get an exception in all cases, as "if result" will simply behave differently and your code after that may still work - just not as expected. That's a pretty heavy change IMHO. Stefan

1 0

[lxml-dev] ET 1.3
by Ian Bicking 12 Sep '07

12 Sep '07

I was just reading the ElementTree 1.3 release notes: http://effbot.org/zone/elementtree-13-intro.htm Generally I like the changes. The change from Element as a factory function to Element as a subclassable class (akin to ElementBase), is nice -- I never understood why there was a distinction. Except... because "el = Element(tag)" doesn't necessarily mean that "el.__class__ is Element"...? getiterator to iter is a simple seeming change. Since getiterator actually returns an iterable, not an iterator, it's also just a little more accurate. Looks like it also moves to an iterator, not a list. I don't have much of an opinion on the parser and serializer stuff, though I'd love it if there was a proper serializer for HTML (not the dumb XSLT-based thing I put in lxml.html). I notice that elements now give warnings when treated as booleans. I like this a lot, as I've found many bugs in my code where I did "if el" where I should have done "if el is not None". And an element with no children doesn't feel falsish at all to me. I've actually already taken to using len(el) to test for children, just because I can't get myself to commit to this weird-seeming behavior. -- Ian Bicking : ianb(a)colorstudy.com : http://blog.ianbicking.org : Write code, do good : http://topp.openplans.org/careers

2 1

[lxml-dev] ET compatible parser interfaces
by Stefan Behnel 12 Sep '07

12 Sep '07

Hi all, just wanted to send a note that I implemented ElementTree compatible interfaces on top of lxml's parsers. This means that you can now do two additional things: use "parser.feed(data)" and "parser.close()" to pass data to a parser in a step-by-step fashion, and pass a "target" keyword argument to a parser to receive SAX-like method calls on the object you pass. The interface is described here: http://effbot.org/elementtree/elementtree-xmlparser.htm This *should* work for both XML and HTML (the latter was hard enough to implement due to differences in the libxml2 API). I also added a parser section to the lxml tutorial while I was at it. The current down-side is that the trunk will require a patched version of Cython until the next Cython release. I added the patch to SVN (and to the Cython bug tracker). The reason is a syntax addition that allows grabbing the GIL during a function call, which simplifies the implementation considerably. Have fun, Stefan

1 1

[lxml-dev] buildout lxml fetch 404 error since lxml-2.0alpha1
by Jeff Kowalczyk 07 Sep '07

07 Sep '07

The 2007-08-30 addition of lxml-2.0alpha1 to PyPI is possibly causing buildouts depending on lxml to fail with the following error when running in fetch-newest mode: Getting required 'zope.testbrowser' required by z3c.etestbrowser 1.0.2-r75829. We have the best distribution that satisfies 'zope.testbrowser'. Picked: zope.testbrowser = 3.4.1 Getting required 'lxml' required by z3c.etestbrowser 1.0.2-r75829. Getting distribution for 'lxml'. While: Installing test. Getting distribution for 'lxml'. Error: Can't download http://codespeak.net/svn/lxml/branch/lxml-2.0: 404 Not Found Thanks.

2 3

[lxml-dev] Resolve RelaxNG document
by Lawrence Oluyede 06 Sep '07

06 Sep '07

I'd like to know if there's a practical way to resolve all the references in a RelaxNG document. Maybe 'expand' is the correct word. I have a document with some <define> which are mostly custom data types and some elements using those types. For example: ---- <define name="foo"> <choice> <value>a</value> <value>b</value> </choice> </define> <define name="ABC"> <element name="abc"> <ref name="foo"> </element> </define> --- What I'd like is to replace the reference "foo" with the definition of "foo" as a whole. Is there an easy way? I read something about custom resolvers... I use lxml2.0alpha1 -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

2 4