Mailman 3 February 2009 - lxml - The Python XML Toolkit

Re: [lxml-dev] resolve_entities=False seems to have no effect
by Stefan Behnel 07 Feb '09

07 Feb '09

Hi, I forwarded your question to the lxml mailing list, which is a much better place to discuss this as there are more people listening who might have an idea. http://comments.gmane.org/gmane.comp.python.lxml.devel/4359 usernamenumber wrote: >> Well, what you get is well-formed XML. May I ask why you need the entity >> references in the output? > > I am calculating checksums based on the combined contents of several > specific tags within a given document. The tool I am writing is designed > to replace a pre-existing tool, which did the same thing and stored > those checksums for comparison. The old tool does not convert entities, > so in order for it to not generate a slew of false-negative checksum > mismatches when we switch over, mine can't either. It's rarely easy to replace a tool if you are required to mimic the original quirks. The right way to do it is to calculate the checksums on the parsed in-memory tree rather than the serialised XML stream. The second best solution is to serialise to canonical XML (C14N) and to work on that. But having checksums depend on a byte stream as serialised by a specific tool is definitely not future proof. To emulate the old behaviour, you could maybe build the checksum from the in-memory tree and just replace all occurrences of »'« and »"« by their escaped equivalent before using a text value. If your XML source documents consistently use the entity references everywhere, this should yield the same checksums. Does that help? Stefan

1 0

Re: [lxml-dev] resolve_entities=False seems to have no effect
by Stefan Behnel 07 Feb '09

07 Feb '09

> s = cStringIO.StringIO("""<foo>"She's the MAN!"</foo>""") > e = etree.parse(s,etree.XMLParser(resolve_entities=False)) Note that there's also etree.fromstring(). > etree.tostring(e) > '<foo>"She\'s the MAN!"</foo>' > > I would have expected resolve_entities=False to have prevented the > translation of, eg, " to ". The "resolve_entities" option is meant for entities defined in a DTD of which you want to keep the reference instead of the resolved value. The entities you mention are part of the XML spec, not of a DTD. > is there another way to prevent this behavior (or, if nothing else, > reverse it after the fact)? Well, what you get is well-formed XML. May I ask why you need the entity references in the output? Stefan

1 0

Re: [lxml-dev] Unable to solve a crash on Windows with LXML
by Stefan Behnel 06 Feb '09

06 Feb '09

Hi, I'm CC-ing the list, I hope you don't mind. I think your description is abstract enough not to reveal anything about your application. Robert Liebeskind wrote: > The trace you received was from v2.2 of lxml but we continue to > experience > the same issue with v2.5. We use XPath extensively. We do not use > XSLT. I guess you meant 2.2beta1 and 2.1.5? > 1. An etree is loaded from an xml file and the data displayed for the > user. > 2. The etree is modified as the result of user edits using a GUI I assume that this happens inside one thread. > 3. The etree is the copied using copy.deepcopy() to etree2 > 4. etree2 is passed via a queue to a thread in which it is further > processed. Try copying the tree inside the target thread, (preferably) instead of copying it inside another thread and passing it over. Trees inherit state from the thread that built them. Also, using a tree inside a thread that did not build it will result in some additional adaptation overhead. > 5. etree2 is modfied as a result of processing in its own thread. > during this processing > additional trees/elems are fetched from disk and used to modify/ > augment etree2. > 6. etree2 is copied to etree3 > 7. etree3 is sent for a additional processing in its own thread. > 8. etree2 is copied to etree4 > 9. etree 4 is sent for additional processing in its own thread. Same thing for 6/7 and 8/9. Copying the tree from inside the target thread will make things more stable. Even if multiple copying is not really memory friendly, it's very fast in lxml, so as long as we are not talking about documents with several megabytes, and as long as this thing really runs on a multi processor machine, you should be fine even with a work-around that copies the tree redundantly in both threads. > at this point the initial thread is complete and tears down. > the two additional spawned threads finish quickly and tear down as well. > These processes will succeed quite often. They fail intermittently > and result in a Windows Unhandled Exception. lxml.etree uses a per-thread dictionary that holds names of tags and attributes. That's one of the reasons why it's so fast and memory friendly. In the stack trace you showed me, it seems that a tree is freed in a different thread than the one that built it, but (for whatever reason) some of it content is still linked to a dictionary of the original thread. In this case, the tree cleanup cannot detect that the name is stored in a dictionary and will free it manually. When the originating thread goes down, either before or after the thread that freed the tree, it will destroy the dictionary that stores the name, which results in a double free. Does that help for now? Stefan

2 4

[lxml-dev] Templating
by Audun Wilhelmsen 06 Feb '09

06 Feb '09

Hey I'd like to use lxml for creating a simple HTML templating engine, by implementing some custom tags and attributes. A simple example: <p gc:text="foobar"></p> Would replace the text content of the tag with the variable foobar. But I'm not sure how to implement it using lxml. I've been using Genshi succesfully, by simply processing the tag stream before serializing. But I'd like to have the DOM-like capabilities and speed of lxml. What would be the best approach? I've been trying to use XPath to find all the elements and attributes with a gc: prefix, but with little success so far (maybe mixing namespaces with non-xml html doesn't work that well?). Audun Wilhelmsen Student, NTNU, Norway

3 2

[lxml-dev] String parameters to xslt transformation
by Alexander Shigin 06 Feb '09

06 Feb '09

lxml lacks ways to apply an external parameter containing both single and double quotes. The patch adds `transform` method to XSLT object with `params` and `strparams` argument. `params` works like `**kw` of `__call__` method (i.e. you still need to surround string literals with quotes). `strparams` are treated literally, so you do not need to use any escaping or quotes.

2 5

[lxml-dev] how knowing the types return by .xpath
by Sergio Monteiro Basto 05 Feb '09

05 Feb '09

Example: from lxml import etree f = open(options.file).read() hparser = etree.HTMLParser(encoding='utf-8', remove_comments=True) etree_document = etree.HTML(f, parser=hparser) elems=etree_document.xpath(strxpath) for frags in elems: print type (frags) (...) if strxpath is equal //h1/following-sibling::text() prints <type 'lxml.etree._ElementUnicodeResult'> if strxpath is equal //div[@class="news"] prints <type 'lxml.etree._Element'> How I do a "if" to detected the types ? thanks -- Sérgio M. B.

3 7