Mailman 3 April 2006 - lxml - The Python XML Toolkit

[lxml-dev] Resolver branch merged, trunk status for 1.0
by Stefan Behnel 27 Apr '06

27 Apr '06

Hi all, the resolver branch is now in the trunk. This means that lxml 1.0 will have full-fledged support for * custom document loaders (see doc/resolvers.txt) * EXSLT * Python regexps in XSLT (can be switched off via 'regexp' keyword) * the XSLT node-set() function * xml:id and DTD IDs * HTML parsing I personally consider the trunk now feature-complete for 1.0. Anyone who is interested in getting the hands on a release rather sooner than later should now take the opportunity to start testing the trunk and report any remaining bugs. http://codespeak.net/svn/lxml/trunk Please install Pyrex 0.9.4.1 to compile it. Have fun, Stefan

1 0

[lxml-dev] exslt:regexp implementation based on 're'
by Stefan Behnel 26 Apr '06

26 Apr '06

Hi, I noticed that exslt:regexp was not supported by libexslt, so I wrote three extension functions that use Python's re module (which is not really JavaScript compatible as requested by the spec, but who cares...). Here's an example: ---------------------------------------- >>> xslt = etree.XSLT(etree.XML("""\ <xsl:stylesheet version="1.0" xmlns:regexp="http://exslt.org/regular-expressions" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="*"> <test><xsl:copy-of select="*[regexp:test(string(.), '8.')]"/></test> </xsl:template> </xsl:stylesheet> """)) >>> result = xslt(etree.XML('<a>123098987</a>')) >>> print str(result) <test>987</test> ---------------------------------------- Since the test cases worked out perfectly, it's already in the trunk. So, when the regular exslt support gets merged, lxml will have more complete exslt support than libxslt itself. :) Stefan

3 7

Re: [lxml-dev] Re: HTMLParser status and issues
by Stefan Behnel 23 Apr '06

23 Apr '06

Stefan Behnel wrote: > Paul Everitt wrote: >> I got this: >> >> # 175 "/usr/include/libxml2/libxml/HTMLparser.h" >> typedef enum { >> HTML_PARSE_NOERROR = 1<<5, >> HTML_PARSE_NOWARNING= 1<<6, >> HTML_PARSE_PEDANTIC = 1<<7, >> HTML_PARSE_NOBLANKS = 1<<8, >> HTML_PARSE_NONET = 1<<11 >> } htmlParserOption; > > > That's not libxml2 2.6.22 then. I think your C compiler uses the Mac-OS system > libraries instead of the libraries installed by what your xmllint uses. Back to this issue, the missing option HTML_PARSE_RECOVER came up in libxml2 2.6.21, while Mac-OS Tiger ships with 2.6.16. However, it looks like the HTML_PARSE_* options follow the numeric values of the XML_PARSE_* enum exactly. So, as a work-around, we could use XML_PARSE_RECOVER to make it compile and simply state that libxml2 2.6.21+ is required for parsing broken HTML. That way, it would keep working with the system libraries on Mac-OS X. Paul, I applied the above change to the branch for now. I'd be glad if you could check that it now compiles with the Mac-OS system libraries. Please run the test suite. If everything works as expected, only the test case(s) for parsing broken HTML should fail. If there are no objections, I'll then start merging the HTML parser into the trunk. Stefan

2 1

[lxml-dev] Linking against libexslt
by Stefan Behnel 23 Apr '06

23 Apr '06

Hi, I'd like to include EXSLT support in lxml as it greatly enhances the capabilities of XSLT. It isn't currently enabled, although supported since libxslt 1.0.19. However, this requires linking against libexslt and xslt-config does (intentionally) not help here: http://mail.gnome.org/archives/xslt/2001-October/msg00133.html I don't know if simply adding "-lexslt" works on all systems (that may be the GCC way of doing it), so I added the following to my setup.py: --------------------------- xslt_libs = flags('xslt-config --libs') for i, libname in enumerate(xslt_libs): if 'exslt' in libname: break if 'xslt' in libname: xslt_libs.insert(i, libname.replace('xslt', 'exslt')) break [...] # near the end in setup() call: extra_link_args = xslt_libs --------------------------- This basically replaces "-lxslt" by "-lexslt -lxslt" on my machine (Linux). I hope this is sufficiently platform-independent to make it work on other systems, but I don't have Windows or Mac-OS available to check that this actually works. Since I was working on XSLT in the "resolver-new" branch anyway, I committed it there. I also attached the complete patch that can be applied to the trunk. It includes a new test case for exslt. Could anyone on the Win/Mac platforms please tell me if this is necessary, or if the following (which also works for me) is sufficient: extra_link_args = flags('xslt-config --libs') + ['-lexslt'] Also, could you please verify that the resolver-new branch compiles and that the test cases pass? Please remember that you may want to install Pyrex 0.9.4.1 before compiling from SVN. Thanks, Stefan

1 0

[lxml-dev] document('') fixed
by Stefan Behnel 23 Apr '06

23 Apr '06

Hi, I played with the XSLT document loaders and found that the default loader can apparently handle "document('')" on XSL documents read from strings as long as they have a non-empty URL. This only makes sense when you know that libxslt keeps a list of known documents during the transformation, so it apparently searches that list for the URL of the requested document. I changed the code on the trunk to create a fake URL for the case that the document URL is empty. So, document('') should now work from any stylesheet (if anyone wants to verify...) Stefan

2 8

[lxml-dev] Custom resolvers
by Stefan Behnel 22 Apr '06

22 Apr '06

Hi, since Paul kept bugging me, I created a new branch (resolver-new) and implemented an API for the custom resolvers stuff. It should be pretty simple to use, just create a parser and register the resolver: parser = XMLParser() parser.resolvers.add(my_resolver) "my_resolver" must be of type etree.Resolver and provide a method resolve(system_url, public_id, context) that returns either None (== "can't resolve, ask someone else") or a _ParserInput object. These can be built from files or strings using the Resolver methods 'resolve_string' and 'resolve_filename'. So, to create a custom resolver, you basically do this --------------- class MyResolver(lxml.etree.Resolver): entity = "This was an entity" def resolve(self, url, id, context): if url == 'my.dtd': # I can handle this return self.resolve_string( u'<!ENTITY myentity "%s">' % self.entity, context) elif url.startswith('http://'): # the default resolver can handle this return super(MyResolver, self).resolve(url, id, context) else: # don't know what to do, let someone else try return None my_resolver = MyResolver() --------------- I'll see how to integrate that in other places of the API, especially XSLT and schemas. Anyway, this works so far. Feel free to comment on it. Stefan

2 7

[lxml-dev] Pyrex 0.9.4.1 is out
by Stefan Behnel 21 Apr '06

21 Apr '06

Hi, Pyrex 0.9.4.1 was released today. It finally compiles lxml nicely and out-of-the-box with Python 2.4 and gcc 4.x. And it can be installed with easy_install Pyrex I updated the INSTALL.txt on the trunk and the 0.9.x branch to make this version a requirement for those who want to twiddle with the source. Stefan

1 0

[lxml-dev] HTML parser is in the trunk! Ready for 1.0 ?
by Stefan Behnel 20 Apr '06

20 Apr '06

Hi all, I just merged the HTML parser branch into the trunk. Paul reported that the latest branch version compiled cleanly on Mac-OS X Tiger (libxml 2.6.16) - and it even passed all tests there, including those on broken HTML. Newer versions of both libxml2 and libxslt are recommended, though. Another recent update on the trunk is the support for xml:id, which is currently available through an XMLDTDID function (XMLID was already in use by ET and is compatible in lxml). The new functionality is now directly based on the libxml2 ID hash table provided by the parser. This means that lxml now supports dictionary-like access to elements having an "xml:id" attribute or DTD-REF attributes. I think it is now the time to fix features for lxml 1.0. Expect it to be released next month (hopefully after Pyrex 0.9.4.1). If you think that lxml still misses something that should be in 1.0 or if you know about any remaining (or new) bugs, report back to the list. Please start a separate thread in that case instead of replying to this mail. Martijn and I are happy about any comment that helps us get lxml better. Have fun, Stefan

1 0

[lxml-dev] HTMLParser status and issues
by Paul Everitt 17 Apr '06

17 Apr '06

Howdy. I was giving the htmlparser branch a try. In trying to compile it, I got: python setup.py build_ext -i running build_ext building 'lxml.etree' extension gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c src/lxml/etree.c -o build/temp.darwin-8.6.0-Power_Macintosh-2.4/src/lxml/etree.o -w -I/usr/include/libxml2 src/lxml/etree.c: In function '__pyx_f_5etree_10HTMLParser___init__': src/lxml/etree.c:17245: error: 'HTML_PARSE_RECOVER' undeclared (first use in this function) src/lxml/etree.c:17245: error: (Each undeclared identifier is reported only once src/lxml/etree.c:17245: error: for each function it appears in.) src/lxml/etree.c:17256: error: 'HTML_PARSE_COMPACT' undeclared (first use in this function) src/lxml/etree.c: In function 'initetree': src/lxml/etree.c:31135: error: 'HTML_PARSE_RECOVER' undeclared (first use in this function) src/lxml/etree.c:31135: error: 'HTML_PARSE_COMPACT' undeclared (first use in this function) error: command 'gcc' failed with exit status 1 --Paul

2 5

Re: [lxml-dev] bug - lxml crash after broken schema instantiation
by Tom Lynn 13 Apr '06

13 Apr '06

On Thu, 13 Apr 2006, Stefan Behnel wrote: > Hi Tom, > > thanks for reporting this, I can reproduce it, even with a test case as simple as > > >>> from lxml.etree import XMLSchema, XML, ElementTree > >>> et = ElementTree(XML("<test/>")) > >>> XMLSchema(et) > > So it is not related to RelaxNG or your schema. This problem does not appear > if the parsed document is within the XML-Schema namespace (we have a test case > for that), but only if the document is not XML-Schema at all. > > This fact (and the Valgrind trace that shows xmlSchemaParse) makes me believe > that this is a bug in libxml2, not lxml. Note that XML-Schema support is > marked as "incomplete" even in the latest libxml2 versions. I see you've submitted a libxml2 bug report too (#338303), which has already been marked as fixed. Remarkable stuff, this open source. :-) > I applied a work-around that checks if the root node of the document passed > into XMLSchema is within the XMLSchema namespace and otherwise rejects the > document without calling libxml2 at all. That's great, thanks. Tom -- Brain: Pinky, are you pondering what I'm pondering? Pinky: I think so, Brain, but if we give peas a chance, won't the lima beans feel left out? -- Pinky and the Brain, 'All You Need is Narf'

1 0