Mailman 3 July 2008 - lxml - The Python XML Toolkit

[lxml-dev] XSLT and threading
by Martijn Faassen 30 Jul '08

30 Jul '08

Hi there, We've been trying to do XSLT transformations in a multi-threaded (Zope 2) situation. On 1.3.6 this won't work, but 2.0.x is supposed to have support for this. Unfortunately we're getting memory errors from within XSLT.__call__, and we think this is the problem: transform_ctxt = xslt.xsltNewTransformContext(self._c_style, c_doc) if transform_ctxt is NULL: _destroyFakeDoc(input_doc._c_doc, c_doc) python.PyErr_NoMemory() For some reason this sometimes works, sometimes fails; it doesn't always do this, so we suspect perhaps we're already in a copied XSLT sheet (by _copyXSLT) when this happens. We're also wondering about the threading strategy of 2.1.x; the copyXSLT code is removed. Is there a new strategy? I couldn't find anything about it in CHANGES.txt. I mean, I wouldn't be unhappy with a new strategy as actually re-parsing the stylesheet each time this gets called from a different thread is rather expensive (the stylesheet isn't cached as far as I can see). What's the new strategy, if any? (Unfortunately 2.1.x also gives us an error, though a different one. I don't have this error handy here.) We're trying to reduce this to a simpler test case that demonstrates the problem but we're having a hard time so far. Any hints would be welcome. Regards, Martijn

4 13

[lxml-dev] Segfault on XSLT/XPath undefined variable error.
by John Krukoff 29 Jul '08

29 Jul '08

So, I've got a tricky to repeat segfault that occurs on an edge case, and I'm wondering if it's even an lxml bug, or a libxslt bug? What appears to be happening, is that if I run this stylesheet which has an undefined variable reference in an xpath expression, sometimes I can get a segfault. Not always, and it seems to be based on what other xml has been processed beforehand. If I start with a clean slate, it works fine and raises an exception as expected. If I run a bunch of other stuff first (yeah, I know how vague this is, and will work on a real test case), it reliably segfaults for me. A coworker noticed this problem, his symptoms were different, instead of segfaulting, the XSLT parser would unexpectedly exit (with no exception!) at the point where the invalid xpath was run, returning whatever result document had been generated up to that point. <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:gizmo="http://www.lt-systems.com/gizmo.xsd"> <xsl:output method="xml" indent="no"/> <xsl:template match="/"> <root> <first/>  <xsl:value-of select="$var"/> <second/> </root> </xsl:template> </xsl:stylesheet> As I don't have debug symbols, the backtrace on the core dump is fairly useless. Does look like it's deep in libxslt where it breaks though. #0 0xb76f127d in ?? () from /usr/lib/python2.5/site-packages/lxml/etree.so (gdb) bt #0 0xb76f127d in ?? () from /usr/lib/python2.5/site-packages/lxml/etree.so #1 0xb7dcaff4 in ?? () from /lib/libc.so.6 #2 0xb7747dd3 in ?? () from /usr/lib/python2.5/site-packages/lxml/etree.so #3 0x083b7459 in ?? () #4 0xbff57388 in ?? () #5 0xb7ce3a6e in sprintf () from /lib/libc.so.6 #6 0xb76f1673 in ?? () from /usr/lib/python2.5/site-packages/lxml/etree.so #7 0x088cd9dc in ?? () #8 0xbff573c0 in ?? () #9 0xb7688430 in ?? () from /usr/lib/libxslt.so.1 #10 0x083b7450 in ?? () #11 0xb75f9af5 in ?? () from /usr/lib/libxml2.so.2 #12 0xb7dcc120 in ?? () from /lib/libc.so.6 #13 0x00000030 in ?? () #14 0x083b7450 in ?? () #15 0xb7688430 in ?? () from /usr/lib/libxslt.so.1 #16 0xb7dcc120 in ?? () from /lib/libc.so.6 #17 0x083f333c in ?? () #18 0xbff57438 in ?? () #19 0x00000016 in ?? () #20 0x00000000 in ?? () The lxml tests run fine for me: TESTED VERSION: 2.0.5 Python: (2, 5, 2, 'final', 0) lxml.etree: (2, 0, 5, 0) libxml used: (2, 6, 31) libxml compiled: (2, 6, 31) libxslt used: (1, 1, 24) libxslt compiled: (1, 1, 24) 899/899 (100.0%): Doctest: xpathxslt.txt ---------------------------------------------------------------------- Ran 899 tests in 20.205s OK PYTHONPATH=src python selftest.py 173 tests ok. PYTHONPATH=src python selftest2.py 94 tests ok. I'll run it through valgrind tomorrow when I have more time, and see if that provides anything interesting. I don't imagine there'll be much useful to be done until I can come up with a real test case, just hoping there's something obvious. -- John Krukoff <jkrukoff(a)ltgc.com> Land Title Guarantee Company

2 1

[lxml-dev] HTML character code interpretation
by Spencer Crissman 29 Jul '08

29 Jul '08

I am using lxml to process some xhtml files. The files have html character codes embedded in them. For instance: & #39;s rather than a '. When I parse the files, edit them, and then write them back out, I want my edits to be the only changes in the output files, but lxml is replacing the character codes with the actual characters they are supposed to represent as well. So if I have: It& #39;s an example. <-- Space inserted to help readability. It is writing out: It's an example. I've tried setting resolve_entities to false, ala: tree = etree.parse(input, etree.XMLParser(resolve_entities=False)) But this seems to have no effect. There a way to tell lxml to ignore these/leave them as is? Thanks. -s

2 3

[lxml-dev] Transform parameter variables
by Niels Bjerre 27 Jul '08

27 Jul '08

I can get transform to accept literal parameter. eg. >>> transform = etree.XSLT(xsl_tree) >>> result = transform(xml_tree, area="'xxx'") *But I can't get transform to accept variables!!?* I have tried: - param = "area=\"\'xxx\'\"" - result = transform(xml_tree, param) - result = apply(transform(xml_tree,param)) I have altso tried to send parameter as dictionary (as I want to send more parameters than one). What am I doing wrong? -- Regards. Niels Bjerre

2 1

[lxml-dev] Ubuntu Installation Woes
by Richard Alex Hofer 25 Jul '08

25 Jul '08

Hi, I've been trying to install lxml 2.1 for days and it seems nothing I've tried works. gcc keeps giving me a enormous amount of errors and I've installed libxml2, libxml2-dev, libxslt1.1, and libxslt1-dev (all form the Ubuntu repository) and it seems nothing works. http://ubuntuforums.org/showthread.php?p=5452334 has more info about all I've done. Any help would be greatly appreciated, I'm working on a project which heavily relies on lxml.

3 8

[lxml-dev] Hello, is this a bug ?
by mmdumi 25 Jul '08

25 Jul '08

Hello, First of all, good job with lxml. Actually, i have a problem, and i think it might be a bug: Let's say i have to parse this xml: <title>casual carpool, serendipity and radovan karadzic</title> <link rel="alternate" type="text/html" href="http://sippey.typepad.com/filtered/2008/07/casual-carpool-serendipity-and-r…" /> <link rel="replies" type="text/html" href="http://sippey.typepad.com/filtered/2008/07/casual-carpool-serendipity-and-r…" thr:count="3" thr:updated="2008-07-23T12:06:32-07:00" /> <id>tag:typepad.com,2003:post-53080848</id> <published>2008-07-22T11:54:03-07:00</published> <updated>2008-07-23T12:06:33-07:00</updated> <summary type="html">"[Casual carpool](http://www.ridenow.org/carpool/)" is one of those (sub)urban fabric things that makes life in the Bay Area go 'round...and occasionally makes it just a bit more interesting. Every morning commuters gather at one of a couple dozen sites in the East...</summary> <author> <name>Michael Sippey</name> </author> <category scheme="http://www.sixapart.com/ns/types#category" term="Berkeley" /> -- Dumitrache Mihai - Marian http://www.takemyview.com/blog

2 2

[lxml-dev] lxml 2.1.1 and 2.0.8 released
by Stefan Behnel 24 Jul '08

24 Jul '08

Hi all, I just released new bug-fix versions of the 2.1 and 2.0 series to PyPI. They mainly fix a crash bug in the XSLT code. Changelog follows as usual. Have fun, Stefan 2.1.1 (2008-07-24) Bugs fixed * Crash when parsing XSLT stylesheets in a thread and using them in another. * Encoding problem when including text with ElementInclude under Python 3. 2.0.8 (2008-07-24) Features added * lxml.html.rewrite_links() strips links to work around documents with whitespace in URL attributes. (backport from 2.1) Bugs fixed * Crash when parsing XSLT stylesheets in a thread and using them in another. * CSS selector parser dropped remaining expression after a function with parameters.

1 0

[lxml-dev] etree.Element from RNG schema?
by Frank Cusack 23 Jul '08

23 Jul '08

Can lxml give me back an Element object from a parsed RNG schema? That is, given parser = etree.RelaxNG(etree.parse(file("foo.rng"))) I'd like to do something like parser.getroot() in order to get back an Element/ElementTree which is like the original schema except with grammar/define's expanded. What I want is to go from <grammar> <start> <element name="foo"> <ref name="bar" /> </element> </start> <define name="bar"> <text /> </define> </grammar> to <element name="foo"> <text /> </element> I realize this can't be done in all cases, e.g. there's no way to express recursive references outside of a <grammar>. But in that case lxml could return an error, or simply not recurse (so the expanded element tree would contain the <ref> itself). But even without handling recursion, wow it would be valuable for me not to have to chase down includes and externalRef's, and more so not to have to [re]implement combine rules. -frank

2 4

[lxml-dev] Parsing i18n UTF-8 files
by Matt Grove 23 Jul '08

23 Jul '08

Hi, I'm attempting to parse several UTF-8 encoded xml files in several different languages - ranging from English to Japanese - but I've run into some trouble. I first want to parse the xml files, gather certain elements into a dictionary or some other data structure, and then write them out to other files. I know how to parse the files when they're in English, but I don't know how to read the Japanese text (for example) without encountering encoding exceptions. I've tried understanding the process through the tutorials but I'm still confused. Would someone like to try steering me on the correct path?

2 1

[lxml-dev] Does lxml/libxslt support passing in nodes as xsl:param parameters?
by Paul Everitt 20 Jul '08

20 Jul '08

I believe the answer is no. I can't tell if this is part of the spec, but docs on other processors imply that you can use a node or node list as the value of a parameter passed to a transformation. In my use case, I'm trying to get information from the current web request. I don't want to modify the input document (my normal way to do this.) Thus, an XPath extension function or a document resolver will have to serve the purpose. This will be tricky, because I'll have to do a thread local or something to know I'm grabbing a *particular* request. Perhaps I need to ponder this. I must be looking at it the wrong way. I want to make sure I don't reparse the input document (it might be pretty big) nor the XSLT processor. --Paul

2 3