Mailman 3 June 2007 - lxml - The Python XML Toolkit

[lxml-dev] Encoding problems with lxml
by Bruno Barberi Gnecco June 29, 2007

June 29, 2007

Hello, I'm having some encoding problems with lxml that I can't solve. My application is a small web mining spider. Pages downloaded can be in any encoding, but I'm expecting mostly utf8 and iso-8859-1. I need to get the parsed data in iso-8859-1. I'm having two problems: a) when reading pages in iso-8859-1, accented characters are converted to HTML sequences, such as à for ` + a. I don't want this to happen, how to avoid it? b) I can't convert pages originally in UTF to ISO, even using etree.tostring(entry, 'iso-8859-1') or string.encode("iso-8859-1"). Have I missed something in the docs? I want to have a homogeneous behavior for all encodings--even if it means to convert first to UTF and later to ISO. Thanks a lot for any help, Bruno

3 5

[lxml-dev] [PATCH] 1.3 Fatal Python error: deallocating None
by David M. Grimes June 27, 2007

June 27, 2007

All - I have identified a bug with the new attrib.pop() method in lxml-1.3 There is a piece of code in etree.pyx (line 1483) related to using the provided default if the key lookup fails. The issue is that the returned value (from PyTuple_GET_ITEM) is a borrowed reference. I was using it as such: v = elt.attrib.get(k, None) And eventually (after several iterations of my parsing loop while processing various documents) I would get this error: Fatal Python error: deallocating None I was able to create a trivial test case to reproduce: import lxml.etree as etree xml = '''\ <?xml version="1.0"?> <xml/>''' for i in range(10000): et = etree.fromstring(xml) et.attrib.pop('x', None) And the patched code replaces the pop() method in etree.pyx with: def pop(self, key, *default): if python.PyTuple_GET_SIZE(default) > 1: raise TypeError, "pop expected at most 2 arguments, got %d" % \ (python.PyTuple_GET_SIZE(default)+1) result = _getAttributeValue(self._element, key, None) if result is None: if python.PyTuple_GET_SIZE(default) == 0: raise KeyError, key else: result = python.PyTuple_GET_ITEM(default, 0) python.Py_INCREF(result) else: _delAttribute(self._element, key) return result This solves the issue for me, and should not introduce any memory leaks, as the returned object in the defaulted path should be INCREF'd. Thanks, --Dave

2 1

[lxml-dev] merge annotate() and xsiannotate() into single function
by Stefan Behnel June 27, 2007

June 27, 2007

Hi Holger, in the current trunk, objectify has those two big functions "annotate()" and "xsiannotate()" that do very similar things in similar ways. When I was changing the annotation behaviour for empty elements, I noticed that it would be hard to keep the two in sync, so I decided to merge them. Here is a patch that matches the current test cases on the trunk (note that I did not yet merge your tests for 1.3 back). Holger, could you look through it to check if this is what you would expect? I know that it does not guarantee exactly the same behaviour as before, so there will likely be cases where some annotations come out different than now. Actually, xsiannotate() was not yet in a release, but I even think that objectify shouldn't guarantee any particular annotations anyway. I rather prefer something like: "it always yields types that make sense for the current data". Another point here: the current XSI type order prefers "integer" over "long" and "float" over "double". Shouldn't those be swapped to always annotate with the widest possible type? Stefan

2 14

[lxml-dev] lxml 1.3 is on cheeseshop
by Stefan Behnel June 25, 2007

June 25, 2007

Hi all, I'm proud to announce the release of lxml 1.3. This is a major new release with various new features and lots of fixes compared to the 1.2 series. The complete changelog follows below. Major objectives of this release were: - API consolidation: make everything work with everything - improved namespace handling: avoid redundant namespaces wherever possible - simplicity and accessibility: improved, restructured documentation and simpler XML/HTML generation Future versions of lxml will continue this trend to make lxml the leading tool for XML and HTML in the Python world. Have fun, Stefan Behnel 1.3 (2007-06-24) ================ Features added -------------- * Module ``lxml.pyclasslookup`` implemens an Element class lookup scheme that can access the entire tree to determine a suitable Element class * Parsers take a ``remove_comments`` keyword argument that skips over comments * ``parse()`` function in ``objectify``, corresponding to ``XML()`` etc. * ``Element.addnext(el)`` and ``Element.addprevious(el)`` methods to support adding processing instructions and comments around the root node * Extended type annotation in objectify: cleaner annotation namespace setup plus new ``deannotate()`` function * Support for custom Element class instantiation in lxml.sax: passing a ``makeelement()`` function to the ElementTreeContentHandler will reuse the lookup context of that function * '.' represents empty ObjectPath (identity) Bugs fixed ---------- * Removing Elements from a tree could make them loose their namespace declarations * ``ElementInclude`` didn't honour base URL of original document * Replacing the children slice of an Element would cut off the tails of the original children * ``Element.getiterator(tag)`` did not accept ``Comment`` and ``ProcessingInstruction`` as tags * API functions now check incoming strings for XML conformity. Zero bytes or low ASCII characters are no longer accepted. * XSLT parsing failed to pass resolver context on to imported documents * More ET compatible behaviour when writing out XML declarations or not * ``Element.attrib`` was missing ``clear()`` and ``pop()`` methods * More robust error handling in ``iterparse()`` * Documents lost their top-level PIs and comments on serialisation * lxml.sax failed on comments and PIs. Comments are now properly ignored and PIs are copied. * Raise AssertionError when passing strings containing '\0' bytes 1.3beta (2007-02-27) ==================== Features added -------------- * ``DTD`` validator class (like ``RelaxNG`` and ``XMLSchema``) * HTML generator helpers by Fredrik Lundh in ``lxml.htmlbuilder`` * ``ElementMaker`` XML generator by Fredrik Lundh in ``lxml.builder.E`` * Support for pickeling ``objectify.ObjectifiedElement`` objects to XML * ``update()`` method on Element.attrib * Optimised replacement for libxml2's _xmlReconsiliateNs(). This allows lxml a better handling of namespaces when moving elements between documents. Bugs fixed ---------- * Possible memory leaks in namespace handling when moving elements between documents Other changes ------------- * major restructuring in the documentation

3 8

[lxml-dev] Final check for lxml 1.3 - anyone?
by Stefan Behnel June 22, 2007

June 22, 2007

Hi all, since there were a few more additions since the last time I asked, could a few people please test the current 1.3 branch on their machines? Preferably Windows, Mac-OS (with current libraries), etc. Please run the tests, maybe also run your own code against it. That would make it less likely that I will have to release a 1.3.1 right next week. :) If all works fine, I'll release 1.3 either this week-end or early next week. Have fun, Stefan @Holger: sorry, but xsiannotate() won't make it. I wouldn't mind adding the functionality to "annotate()" later on in the 1.3 series, though.

1 0

Re: [lxml-dev] documentation wiki
by Stefan Behnel June 22, 2007

June 22, 2007

Hi Roger, Roger Patterson wrote: > I don't know what the ML's etiquette is for tangential > discussions, so I didn't include the mailing list on this email. this kind of discussion definitely belongs on the ML. We only have one, so we use it to discuss basically everything from infrastructure to development to helping users. :) It's rather low traffic, so it's usually not a big problem if there is one mail more coming through. > Thanks for that link to the mailing-list search tool. Very handy. > Also - this led me to actually find http://codespeak.net/lxml/dev for > documentation, which seems far more detailed than what I was looking at > before: http://codespeak.net/lxml/api.html It's the new web page that will replace the current one with the release of lxml 1.3 (hopefully soon). > I'm not an uber-expert with wikis. I've just done very basic > administration for my own company's documentation, using dokuwiki > (http://wiki.splitbrain.org/wiki:dokuwiki) - which uses PHP to serve > file-system-based wiki files. Not speaking for anyone else here, but I guess any PHP based solution is rather out - unless we find nothing else :) > I did see some discussion, in searching the ML, on moving lxml to Trac a > while ago - is this still under consideration? codespeak has its own tracker by now. Not wiki based, just a simple tracker. So I guess the decision on the codespeak side was rather against Trac. > I don't have oodles of time, but I'd be willing to help administer a > wiki to help out. > I might be able to put a wiki on my company's website if codespeak isn't > up for it. Thanks for the offer. But let's wait for a word by the codespeak people first. Stefan

1 0

[lxml-dev] documentation wiki
by Roger Patterson June 22, 2007

June 22, 2007

Hello all, I moved to lxml/elementtree from 4Suite and am loving it. However, it would be good to improve the documentation for us muddling our way through it - but of course the issue of who-will-do-it/who-has-the-time always comes up. To solve this - would it be possible to start a documentation wiki on the codespeak site (I use dokuwiki - which is excellent, and I hear moinmoin is good and python-powered)? This would disperse the documentation effort among those who use it (maybe only allow members of the mailing list, to prevent spam). I would be more than happy to add documentation as I figure things out. Not sure if this has already been discussed on the mailing list, as I don't seem to be able to perform searches on the mailing list archives. Cheers, -Roger Patterson

2 1

[lxml-dev] lxml 1.3 annotate() behaviour for empty string data elements
by jholg＠gmx.de June 21, 2007

June 21, 2007

Hi, I just noticed that annotate() does not add type information to empty string elements when parsed: >>> root = etree.fromstring(""" ... <root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ... xmlns:py="http://codespeak.net/lxml/objectify/pytype" ... xmlns:xsd="http://www.w3.org/2001/XMLSchema"> ... <s1>foobar</s1> ... <s2></s2> ... </root> ... """) >>> objectify.annotate(root) >>> >>> print etree.tostring(root, pretty_print=True)oot) <root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:py="http://codespeak.net/lxml/objectify/pytype" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <s1 py:pytype="str">foobar</s1> <s2/> </root> >>> Whereas type annotation happens when setting attributes manually: >>> root = objectify.Element("root") >>> root.s1 = "foobar" >>> root.s2 = "" >>> objectify.annotate(root) >>> print etree.tostring(root, pretty_print=True) <root xmlns:py="http://codespeak.net/lxml/objectify/pytype" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <s1 py:pytype="str">foobar</s1> <s2 py:pytype="str"></s2> </root> >>> I know this happens due to the .text of the node being None in the 1st case instead of '' in the second case (which is lxml/ElementTree/libxml2 behaviour that bites me once and again). Still, I'd prefer to have annotate() provide all data elements with type information; after all, the element in question is treated as a StringElement (the default emtpy_data_class) anyway. Objections? Holger -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail

2 1

Re: [lxml-dev] ElementInclude doesn't honour base URLs
by Stefan Behnel June 21, 2007

June 21, 2007

Mike Meyer wrote: > In <4679A865.3010706(a)behnel.de>, Stefan Behnel <stefan_ml(a)behnel.de> typed: >> Mike Meyer wrote: >>> The problem is that using ElementInclude.include doesn't resolve href >>> attributes relative to their containing document. >> Fixed on the trunk and for 1.3. > > Cool. Thank you. > > For the archives, my workaround for 1.2.x is: > > from os.path import join, split > from lxml.etree import parse > from lxml.ElementInclude import include, default_loader > > class Loader(object): > """Custom loader for XInclude processing to fix relative addressing for files.""" > > def __init__(self, name): > self.base = split(name)[0] > > def __call__(self, href, typ, encoding=None): > return default_loader(join(self.base, href), typ, encoding) This only works for local files. You can use "urlparse.urljoin" to make it work for more or less any file path or URL (remember that lxml supports ftp and http). You don't even need to do the split before hand, BTW. Ah, and I don't think this works for included files including files themselves from different directories. But it's still a good work around for most use cases. Stefan

1 0

Re: [lxml-dev] ElementInclude doesn't honour base URLs
by Stefan Behnel June 20, 2007

June 20, 2007

Hi, Mike Meyer wrote: > The problem is that using ElementInclude.include doesn't resolve href > attributes relative to their containing document. Fixed on the trunk and for 1.3. Stefan

1 0