Mailman 3 October 2007 - lxml - The Python XML Toolkit

[lxml-dev] lxml 2.0alpha4 released
by Stefan Behnel Oct. 6, 2007

Oct. 6, 2007

Hi all, I just released a 4th alpha version of lxml 2.0 to PyPI. It hopefully sets an end to the tag name validation problems by distinguishing between HTML tags and XML tags based on the associated parser, i.e. either the one that parsed it or the one that created the element through its "makeelement" method. Note that the Element factory of lxml.etree uses the XMLParser by default, while the factory in lxml.html uses the HTMLParser, and thus allows HTML tag names. Everyone who bumped into and/or reported problems with this, please verify that this provides a viable solution to you. Have fun, Stefan 2.0alpha4 (2007-10-07) Features added Bugs fixed * AttributeError in feed parser on parse errors Other changes * Tag name validation in lxml.etree (and lxml.html) now distinguishes between HTML tags and XML tags based on the parser that was used to parse or create them. HTML tags no longer reject any non-ASCII characters in tag names but only spaces and the special characters <>&/'"

1 0

[lxml-dev] Namespace serialization patch
by Lawrence Oluyede Oct. 6, 2007

Oct. 6, 2007

I had the same problem Anders Bruun Olsen had in this thread: http://comments.gmane.org/gmane.comp.python.lxml.devel/2924 What I'd like to know if I have to wait for 2.0 completion (using the alpha is not an option AFAIK) to use it or you plan to release an interim 1.3.x version with that patch applied. Thanks -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 3

[lxml-dev] prefix mappings
by FnH Oct. 6, 2007

Oct. 6, 2007

Hi, I would like to generate the following serialization: <a xmls="foo"> <b xmls="bar"/> </a> Currently, the closest I can come is: <a xmls="foo" xmlns:x="bar"> <x:b/> </a> using the following code: a = Element("{foo}a", nsmap={None:"foo", "x":"bar"}) a.append(Element("{bar}b")) Note that even though these serializations are equivalent, sometimes remapping namespaces is neccessary to work around faulty implementations in other products. The feed validator for example warns against using namespace prefixes on elements for good reason. If you don't, many readers aren't able to parse your feed (http://www.intertwingly.net/wiki/pie/XmlNamespaceConformanceTests). If you're using xhtml as content this means remapping the default namespace. In order to solve this I think it would be a good idea to allow (or take into account) prefix mappings on non root nodes as well. The output I'd like could then be achieved by the following code snippet: a = Element("{foo}a", nsmap={None:"foo"}) a.append(Element("{bar}b", nsmap={None:"bar"})) What do you think? Regards, Nick

2 1

[lxml-dev] Tag name validation and HTML
by James Graham Oct. 6, 2007

Oct. 6, 2007

The development branch of lxml 2 appears to restrict the characters that may appear in a tag name. Whilst this may be appropriate for XML, it does not match the behavior of all common HTML UAs and, as such, does not match the current draft of the HTML 5 spec [1]. This is an issue for html5lib [2] as we are keen to keep support for building lxml trees from HTML input, something which is currently possible with lxml 1.3. In an only tangentially related question, is there a recommended way of creating a custom tag type, preferably using the same code for ElementTree and lxml.etree? In particular html5lib needs to create a notional document root element whilst parsing. So far, we have been using an ordinary Element with a .tag that cannot be produced by parsing any input e.g. root.tag="<DOCUMENT_ROOT>" but this doesn't feel very elegant. [1] http://www.whatwg.org/specs/web-apps/current-work/#tag-name0 [2] http://code.google.com/p/html5lib/ -- "Eternity's a terrible thought. I mean, where's it all going to end?" -- Tom Stoppard, Rosencrantz and Guildenstern are Dead

2 5

[lxml-dev] lxml 2.0alpha1 released
by Stefan Behnel Oct. 6, 2007

Oct. 6, 2007

Hi all, I'm proudly announcing the first alpha release of lxml 2.0. It features a major cleanup both behind the scenes and at the surface, that improves the XML tool integration and makes the API clearer and more consistent in many places. The major new addition, however, is the lxml.html package, a new toolkit for HTML handling. The web site for the pre-2.0 series is online at http://codespeak.net/lxml/dev/ The "what's new" page has a description of the major changes: http://codespeak.net/lxml/dev/lxml2.html and the ChangeLog has a more detailed list, see below. This being an alpha release means that not everything is stable, both in terms of crashes and the API. There will be a small number of alpha releases to make the advancements publicly available, before the beta releases focus on improving the stability. I warmly invite everyone to contribute to the final release by discussing the API changes and the new features on the mailing list. There is always space for improvements! There is currently a known problem with Microsoft's compilers, so Windows builds may not become available for 2.0alpha1. The next alpha will hopefully come with prebuilt binaries for that platform. Building with the more standards compliant MinGW compilers should work. Note that working on the code now requires Cython (version 0.9.6.5), an enhanced fork of Pyrex. lxml therefore no longer ships with a copy of Pyrex or Cython, but as usual, building from the distribution sources does not require Cython. It can be installed with "easy_install Cython" or from here: http://www.cython.org/ I hope that lxml 2.0 will become a straight continuation of the success story that lxml 1.x was already. Have fun, Stefan 2.0alpha1 (2007-09-02) Features added * Reimplemented objectify.E for better performance and improved integration with objectify. Provides extended type support based on registered PyTypes. * XSLT objects now support deep copying * New makeSubElement() C-API function that allows creating a new subelement straight with text, tail and attributes. * XPath extension functions can now access the current context node (context.context_node) and use a context dictionary (context.eval_context) from the context provided in their first parameter * HTML tag soup parser based on BeautifulSoup in lxml.html.ElementSoup * New module lxml.doctestcompare by Ian Bicking for writing simplified doctests based on XML/HTML output. Use by importing lxml.usedoctest or lxml.html.usedoctest from within a doctest. * New module lxml.cssselect by Ian Bicking for selecting Elements with CSS selectors. * New package lxml.html written by Ian Bicking for advanced HTML treatment. * Namespace class setup is now local to the ElementNamespaceClassLookup instance and no longer global. * Schematron validation (incomplete in libxml2) * Additional stringify argument to objectify.PyType() takes a conversion function to strings to support setting text values from arbitrary types. * Entity support through an Entity factory and element classes. XML parsers now have a resolve_entities keyword argument that can be set to False to keep entities in the document. * column field on error log entries to accompany the line field * Error specific messages in XPath parsing and evaluation NOTE: for evaluation errors, you will now get an XPathEvalError instead of an XPathSyntaxError. To catch both, you can except on XPathError. * The regular expression functions in XPath now support passing a node-set instead of a string * Extended type annotation in objectify: new xsiannotate() function * EXSLT RegExp support in standard XPath (not only XSLT) Bugs fixed * lxml.etree did not check tag/attribute names * The XML parser did not report undefined entities as error * The text in exceptions raised by XML parsers, validators and XPath evaluators now reports the first error that occurred instead of the last * Passing '' as XPath namespace prefix did not raise an error * Thread safety in XPath evaluators Other changes * objectify.PyType for None is now called "NoneType" * el.getiterator() renamed to el.iter(), following ElementTree 1.3 - original name is still available as alias * In the public C-API, findOrBuildNodeNs() was replaced by the more generic findOrBuildNodeNsPrefix * Major refactoring in XPath/XSLT extension function code * Network access in parsers disabled by default

3 20

[lxml-dev] Re: Tag name validation and HTML
by jholg＠gmx.de Oct. 5, 2007

Oct. 5, 2007

Hi, > The : thing is difficult because HTML UAs are expected to deal with : in > the tag name and there is content in the wild that depends on this being > accepted; MS Office produces "HTML" containing tags like <o:p>, for > example. Since I, and I guess others too, want to use lxml to process > random content that may have colons in the tag names, hard failure for > this case is a problem. To make matters worse it is possible that the > HTML spec will change in the future to introduce some sort of > namespacing feature which may or may not use colons. You'd get errors when parsing such stuff with the XML parser: >>> etree.fromstring("""<o:p>foo</o:p>""") Traceback (most recent call last): File "<stdin>", line 1, in ? File "etree.pyx", line 2137, in etree.fromstring File "parser.pxi", line 1301, in etree._parseMemoryDocument File "parser.pxi", line 1207, in etree._parseDoc File "parser.pxi", line 782, in etree._BaseParser._parseDoc File "parser.pxi", line 444, in etree._ParserContext._handleParseResultDoc File "parser.pxi", line 523, in etree._handleParseResult File "parser.pxi", line 471, in etree._raiseParseError etree.XMLSyntaxError: Namespace prefix o on p is not defined, line 1, column 5 but not with the HTML parser: >>> etree.HTML <built-in function HTML> >>> etree.HTML("""<o:p>foo</o:p>""") <Element html at 2c8030> >>> So here's a distinction between HTML and XML, but not API-wise, e.g when creating elements. For my usecase, I must *rely* on producing valid XML through the API, so making things more liberal potentially breaks my system. That's because I need to pickle (i.e. serialize) tree content and reparse somewhere else. Now if I allow for producing invalid XML, some data receiver will choke on my data. > Given all of this I would prefer it if it were possible to have an > HTML-specific mode with much more liberal rules than the XML mode. This > could then be adapted to support any namespacing features HTML grows in > the future. For example, if one could do something like > > import lxml.html > lxml.html.Element("o:p") > > where lxml.html.Element would be just like lxml.etree.Element but > without XML-specific validity checks. I guess there might be serious > practical difficulties with that exact solution, but I think the general > idea of being able to flag an element as following HTML rules or XML > rules would be more user-friendly than having a set of rules that > neither matches the XML nor the HTML model correctly. Sounds better to me than introducing some mixed set of rules. And I don't even think that it's difficult to implement, though it might mean introducing another public factory or some sort of switch on Element(). Holger -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer

1 0