Mailman 3 July 2007 - lxml - The Python XML Toolkit

[lxml-dev] new Namespace() registry behaviour for lxml 2.0
by Stefan Behnel July 21, 2007

July 21, 2007

Hi all, this has been in the queue for quite a while: the global Namespace() class lookup will go away in lxml 2.0, although it will be possible to write backwards compatible code. In lxml 1.x, the namespace class lookup was configured using the global Namespace() factory. This was easy, but it was also an application wide setup: >>> ns = etree.Namespace("http://my/namespace") >>> ns["mytag"] = MyElementClass The lookup itself was then configured through the ElementNamespaceClassLookup class, which (starting with lxml 1.2) was a parser local setup: >>> lookup = etree.ElementNamespaceClassLookup() >>> parser = etree.XMLParser() >>> parser.setElementClassLookup(lookup) In lxml 2.0, the namespace registry will become local to the lookup instance. Consequently, the new setup will look like this: >>> lookup = etree.ElementNamespaceClassLookup() >>> parser = etree.XMLParser() >>> parser.setElementClassLookup(lookup) >>> ns = lookup.get_namespace("http://my/namespace") >>> ns["mytag"] = MyElementClass Not a big difference, but this still has the huge benefit that the namespace registry can now be different for different parsers, i.e. different modules using lxml can now have their own registry without having to fear any interference with other modules or parts of an application. This means that all Element class lookup schemes are now parser local. As the usage itself does not change, it is easy to adapt existing code so that it runs with lxml 2.0 while still supporting lxml 1.2/3 (at least, it was easy enough to adapt the unit tests in lxml...). All you have to do is set the same lookup instance for all parsers you use (note that the initial setup is already required today) and replace the occurrences of calls to etree.Namespace by the "get_namespace" method of that lookup instance. I believe that this simple change is easy enough to work around for existing code, so it will not have a big impact here. However, the advantage of removing the global setup and thus supporting independent configurations for different parts of an application (and third party libs) outweighs this overhead considerably. Have fun, Stefan

1 0

[lxml-dev] new ElementSoup module in lxml.html
by Stefan Behnel July 21, 2007

July 21, 2007

Hi, I rewrote Fredrik's ElementSoup.py module for lxml.html so that you can now have lxml read in tag soup with BeautifulSoup and convert it into an lxml.html tree of Elements. While libxml2 can also parse broken HTML, it is not made to parse sick soup of tags, so if you need to work with web pages that sort of look like they might have been HTML once, the lxml.html.ElementSoup module can help you get there. http://codespeak.net/svn/lxml/branch/html/doc/elementsoup.txt http://codespeak.net/svn/lxml/branch/html/src/lxml/html/ElementSoup.py Have fun, Stefan

2 2

[lxml-dev] Segfault in lxml 1.3.2
by Itamar Shtull-Trauring July 19, 2007

July 19, 2007

Hi, One of my co-workers is getting an occasional segfault in some code using lxml; it uses no other non-stdlib libraries, and it's single threaded. The segfault is in: __pyx_tp_dealloc_5etree__Element (src/lxml/etree.c:11282). Any info I can try to get to help debug this? Would a core dump be helpful? He's so far been unable to create a minimal reproducing example. Thanks, Itamar Shtull-Trauring

2 1

[lxml-dev] invalid tag names get serialized
by jholg＠gmx.de July 19, 2007

July 19, 2007

Hi, I noticed that lxml (both objectify and etree) happily accepts broken tag names (numbers, containing whitespace, ...) throughout the API and also serializes such document; only when trying to re-parse it this fails: >>> root = etree.Element("root") >>> etree.SubElement(root, " __foo bar ") '' >>> print etree.tostring(root) <root>< __foo bar /></root> >>> print etree.fromstring(etree.tostring(root)) Traceback (most recent call last): File "<stdin>", line 1, in ? File "etree.pyx", line 1970, in etree.fromstring File "parser.pxi", line 980, in etree._parseMemoryDocument File "parser.pxi", line 876, in etree._parseDoc File "parser.pxi", line 533, in etree._BaseParser._parseDoc File "parser.pxi", line 660, in etree._handleParseResult File "parser.pxi", line 608, in etree._raiseParseError etree.XMLSyntaxError: StartTag: invalid element name, line 1, column 8 >>> I gather this is basically libxml2 behaviour. It is not nice, though, since you can produce serialized data without knowing your evil doings, and only detect it when you try to parse it back in (in vain). Would it be a problem to have the tag name checked before it is set for an element? Holger -- Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kanns mit allen: http://www.gmx.net/de/go/multimessenger

2 9

[lxml-dev] Encoding bug in lxml.etree.HTML
by Alexander Kozlovsky July 18, 2007

July 18, 2007

Hello! I've probably discovered a bug in lxml.etree.HTML: >>> from lxml import etree >>> a = u'<html><body><p>\u044b</p></body></html>' >>> b = etree.HTML(a) >>> b[0][0].text u'\xd1\x8b' Expected: u'\u044b' It seems that etree.HTML function works with non-ascii symbols incorrectly. I can reproduce it on Windows. This bug is relatively new: it happens with lxml with statically linked libxml2 version 2.6.28 and libxslt2 version 1.1.19 (current version of lxml-1.2.1 from Cheese Shop and the newer releases). Older versions of lxml (lxml-1.2.1 with libxml2 version 2.6.26 and libxslt2 version 1.1.17, which are no longer available from Cheese Shop, or older releases such as lxml-1.2) work fine. -- Best regards, Alexander mailto:alexander.kozlovsky@gmail.com

2 1

[lxml-dev] lxml.html and forms
by Ian Bicking July 17, 2007

July 17, 2007

I feel a little bad adding a bunch of stuff to lxml.html when it's supposed to get all stable. But I was getting ready for a presentation on lxml.html, and this seemed like it made it a lot more fun. So with my last commit you can do things like: from lxml.html import parse, open_in_browser url = 'http://tripsweb.rtachicago.com/' page = parse(url) page.make_links_absolute(url) form = page.forms[0] form.inputs['Orig'].value = '1535 W Leland' form.inputs['Dest'].value = '847 W Bertrand' res = form.submit() res_page = parse(res) res_page.make_links_absolute(res_page.geturl()) open_in_browser(res_page) It's kind of like Mechanize, only of course better. There's some things I still haven't figured out. Some data structures are convenient, but maybe have some non-obvious aspects. Like form.inputs, which doesn't always return elements (for things like checkboxes it can return something that is more like a logical element). Also, I'd like to merge in most of the functionality of lxml.html.formfill (except for error-filling), so where form.form_values() currently returns a list of all the values as they'd be if submitted, I'd like to make it settable. And maybe even have form.form_values return something that would modify inputs in-place, like form.form_values['Orig'] = '1535 W Leland' mean the equivalent of form.inputs['Orig'].value = '1535 W Leland'. Another option question is actual form submission. Right now it uses urllib. But I like httplib2, for instance, and I'd like it to be possible to use that. Also, I'm wondering about how to keep track of the URL when a page is parsed. Stefan mentioned if you use parse(url) it would keep track of that... where? I'd like it to be possible to keep the URL around for any kind of parsing, e.g., with document_fromstring(html, url=X). Ian

2 9

[lxml-dev] Compact RelaxNG Validation
by Leonard J. Reder July 16, 2007

July 16, 2007

Hello, Does the lxml validation support the compact form of RelaxNG Schema language? Thanks, Len -- ____________________________________________________ Leonard J. Reder Jet Propulsion Laboratory Mar Science Laboratory Project Flight Software Applications & Data Product Management, Section 316D Email: reder(a)jpl.nasa.gov Phone (Voice): 818-354-3639 Mail Address: Mail Stop: 171-113 4800 Oak Grove Dr. Pasadena, CA. 91109 ---------------------------------------------------

3 5

[lxml-dev] Test Failures in lxml 1.3.2
by Sidnei da Silva July 16, 2007

July 16, 2007

I get one test failure with lxml 1.3.2, doesn't look too bad. Maybe it has something to do with the libxml2 version? ====================================================================== FAIL: test_module_HTML_unicode (lxml.tests.test_htmlparser.HtmlParserTestCaseBas e) ---------------------------------------------------------------------- Traceback (most recent call last): File "c:\Python24\lib\unittest.py", line 260, in run testMethod() File "C:\src\lxml-build\lxml-1.3.2\src\lxml\tests\test_htmlparser.py", line 33 , in test_module_HTML_unicode unicode(self.uhtml_str.encode('UTF8'), 'UTF8')) File "c:\Python24\lib\unittest.py", line 333, in failUnlessEqual raise self.failureException, \ AssertionError: u'<html><head><title>test \xc3\x83\xc2\xa1\xef\xa3\x92</title></ head><body><h1>page \xc3\x83\xc2\xa1\xef\xa3\x92 title</h1></body></html>' != u' <html><head><title>test \xc3\xa1\uf8d2</title></head><body><h1>page \xc3\xa1\uf8 d2 title</h1></body></html>' ---------------------------------------------------------------------- -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 18

Re: [lxml-dev] DOM2 range() support?
by Stefan Behnel July 16, 2007

July 16, 2007

Gloria W wrote: > Stefan Behnel wrote: >> This is actually the first time I come across the concept of DOM2 ranges. >> isn't that simply a tuple of two positions, where each position contains an >> Element and optionally one of the following: >> >> - an attribute "{ns}name" and a string position in the attribute value >> - a string position in the text >> - a string position in the tail > > Yes, I think so. > The time consuming part of the implementation is that a range can access > any position in any component of the DOM structure, without restriction. > So this means accessing characters anywhere in the tag elements, values, > etc. > For example, the DOM range can begin at the 'r' and end at the 'x' in > '<a href="x">' Hmm, but honestly, that use pattern isn't very likely, right? I mean, I would consider putting the index /on/ an element (start and/or end) or somewhere into textual content to be the only useful patterns anyway. Wouldn't such an implementation be much simpler and still largely sufficient? You can always tell users to stick to that... > The DOM structure has to be able to be treated like a node structure and > a string index simultaneously. It has to be able to interpret changes to > the string and translate those into node changes, which is the very > tedious, time-consuming part of implementing this feature. Wait, does this mean adding 20 to the range index could let you end up on a different node? Say, switch from an element to an attribute, or maybe from an element to its closing parent? Who designs these things??? And: is a string index defined based on the original encoding or is it the unicode content of the infoset? (I *hope* the latter) >> Admittedly, the DOM2 interface on top of that is a little more complex and >> (should I say it?) DOM-ishly obfuscated, but there shouldn't be more to it >> than that, right? I mean, the respective W3C spec is only some 13 sections >> long, there *can't* be more than that. :) >> > 8) I would be so happy with just a fully functional range() > implementation. I'm sure there's more to it, but developers can > hopefully contribute as they need these features on the server side. As I said, go for the common cases first. >> Hmm, now that we have cool HTML support and CSS selection, I wouldn't mind >> having a Range class hanging around in some (lxml.range?) module. Doesn't even >> sound like you'd have to implement it in Pyrex, Python code should be enough here. >> > I think so as well. regex should be enough. Well, what you describe above sounds more like I'd try to traverse the tree to get along, not the completely serialised representation, and then serialise each single element, attribute, etc. in order to see where we end up. > I would have liked to do it, but I don't have the spare time right now. Well, that's usually a show-stopper in the OS world. It's either time or money (but at least it's not *always* money). Stefan

1 0

Re: [lxml-dev] Element children not right?
by Stefan Behnel July 15, 2007

July 15, 2007

Mike Meyer wrote: > On Sun, 15 Jul 2007 19:46:31 +0200 Stefan Behnel <stefan_ml(a)behnel.de> wrote: >>> Right. But I don't need the extra filter in ElementTree - it returns >>> just the element children and that's what it's documented as doing. I >>> quite literally found this when I took working ElementTree code and >>> tried to move it to lxml's implementation of ElementTree (and then >>> moved back to ElementTree rather than use test on the tag attributes >>> type). >> Hmm, I see your point. Maybe we should provide an ElemenTree compatible parser >> that strips comments and PIs from the document. That would at least help in >> porting existing code. > > I think that would certainly solve the problem I ran into. The current trunk has an "ETCompatXMLParser" class that is a subclass of XMLParser with a different default setup. Currently that only means that remove_comments and remove_pis are active by default. Please try setting it as a default parser on your side and report back if you find any further incompatibilities. BTW: ETCompatXMLParser will not become the default parser in lxml.etree, it will only be available to simplify code portability. Stefan

1 0