Mailman 3 June 2008 - lxml - The Python XML Toolkit

Re: [lxml-dev] schema validation - what would be valid at this point?
by John Lovell 01 Jul '08

01 Jul '08

So with schema snippets that look like this, what would it give you as valid input options? <xs:attribute name="RefId" use="required" type="sif:RefIdType" /> <xs:simpleType name="RefIdType"> <xs:restriction base="sif:GUIDType" /> </xs:simpleType> <xs:simpleType name="GUIDType"> <xs:restriction> <xs:simpleType> <xs:restriction base="xs:token"> <xs:pattern value="[0-9A-F]{32}" xmlns:xs="http://www.w3.org/2001/XMLSchema" /> </xs:restriction> </xs:simpleType> </xs:restriction> </xs:simpleType> John W. Lovell Web Applications Engineer Northwest Educational Service District 1601 R Avenue Anacortes, WA 98221 (360) 299-4086 jlovell(a)nwesd.org www.nwesd.org Together We Can ... -----Original Message----- From: Terry Brown [mailto:tbrown@nrri.umn.edu] Sent: Monday, June 30, 2008 12:23 PM To: John Lovell Subject: Re: [lxml-dev] schema validation - what would be valid at this point? On Mon, 30 Jun 2008 11:49:50 -0700 "John Lovell" <jlovell(a)esd189.org> wrote: > Can you give an example of what that might look like? What I'd really like to have is the functionality of Emac's nxml-mode via a Python API. That mode parses the text up to the insert point, and can then tell you what elements would be valid there, according to any RELAX-NX schema you specify. So really I guess it makes more sense to be able to get the answer at parse time, rather than when the tree's complete. If, so far, the parser has seen "<html><head>" the answer to the question "what's valid at this point?" would be "base, isindex, link, meta, script, style, title". I don't know if the libxml2 schema API will answer that question, if not then it's more a libxml2 feature request (initally :) rather than an lxml thing. Just using HTML as an example, you could answer in terms of any user supplied schema. Sorry my initial description was confused because I was thinking about already complete and valid trees to which you might add something, it would be nice to handle that case too, but it seems to make more sense to approach it from the incomplete, half parsed :-) tree point of view first. Cheers -Terry > John W. Lovell > Web Applications Engineer > Northwest Educational Service District > 1601 R Avenue > Anacortes, WA 98221 > (360) 299-4086 > jlovell(a)nwesd.org > > www.nwesd.org > Together We Can ... > > > -----Original Message----- > From: lxml-dev-bounces(a)codespeak.net > [mailto:lxml-dev-bounces@codespeak.net] On Behalf Of Terry Brown > Sent: Monday, June 30, 2008 11:20 AM > To: lxml-dev(a)codespeak.net > Subject: [lxml-dev] schema validation - what would be valid at this > point? > > Hi all, > > First thanks for lxml - it's great. > > If this is the wrong place for feature requests please point me > elsewhere... > > A feature that would be very useful for a number of applications, > particularly editors: the ability to ask "what attributes / elements > would be valid at this point?" where I guess a point would be > identified by an XPath or perhaps an Element object. > > I don't know if any of the libxml2 schema APIs expose this, if they > don't, that's probably where the change needs to be made first, but if > they do it would be great if lxml passed it on to python. > > Thanks, > > Terry > _______________________________________________ > lxml-dev mailing list > lxml-dev(a)codespeak.net > http://codespeak.net/mailman/listinfo/lxml-dev > ----------------------------------------------------------------------- Terry Brown Natural Resources Research Institute Research Associate 5013 Miller Trunk Highway tbrown(a)nrri.umn.edu University of Minnesota, Duluth Ph. 218 720 4345 Duluth, Minnesota 55811 Fax 218 720 4328 http://beaver.nrri.umn.edu/~tbrown/

2 1

[lxml-dev] schema validation - what would be valid at this point?
by Terry Brown 01 Jul '08

01 Jul '08

Hi all, First thanks for lxml - it's great. If this is the wrong place for feature requests please point me elsewhere... A feature that would be very useful for a number of applications, particularly editors: the ability to ask "what attributes / elements would be valid at this point?" where I guess a point would be identified by an XPath or perhaps an Element object. I don't know if any of the libxml2 schema APIs expose this, if they don't, that's probably where the change needs to be made first, but if they do it would be great if lxml passed it on to python. Thanks, Terry

2 1

[lxml-dev] Possible bug in DOM tree iteration?
by Viksit Gaur 30 Jun '08

30 Jun '08

Hi all, I'm running some tests on a page's DOM tree by assigning each element a unique identifier and then doing some analysis using this. I use code similar to, root = bs.fromstring(txtcontent) self.pagetree = etree.iterwalk(root, events=("start",)) for event, element in self.pagetree: element.attrib['uid'] = str(cnt) cnt = cnt + 1 etc. However, I notice that when iterating through the DOM, on text such as the following: -- This is something here which has some more text here and repeats here again for this statement and some more text here that doesn't have any tags at all. -- The uid is assigned only to the P and the first B, but everything after is left untouched. So, the word "statement" is never assigned an Id. Moreover, I'm not sure how to access the rest of the the text under the P tag. When iterating through the tree, shouldn't the other tags be included too, as well as the text for the P element should contain ALL the text in there, including the b tags? If this is intended behavior, could someone point me to how I could achieve accessing all the other text under the P tag, as well as the B tags in there? Cheers Viksit

2 1

[lxml-dev] pickling ElementStringResult
by Martijn Faassen 28 Jun '08

28 Jun '08

Hi there, I just ran into a problem when upgrading an application to use lxml 2.0.x instead of 1.3.6. xpath now returns a special lxml.etree._ElementStringResult object, a smart string. A bit too smart for my situation... Previously, I'd get a string back and I could stuff that into the ZODB just fine - it was picklable. Now I get this and it can't be pickled, so the same code fails. Is it only xpath that is so affected or are there other operations where these things can be returned? It's quite frustrating to have to worry whether a string came from an lxml xpath (and possibly operations?) and therefore is broken in a subtle way when you want to store them somewhere. You always worry whether your test coverage is complete enough to detect all cases. It's also relatively hard to track this kind of bug down even if you have good test coverage, as the error only occurs later on when the ZODB tries to pickle the object with this kind of special string somewhere in it. Another potential problem is that a smart string when held in an object database might keep the underlying XML document alive long past its intended lifetime. I was originally thinking about coming up with a special way to pickle these smart strings, but this actually leads me to prefer another solution: a flag that can be passed to xpath() that turns off the returning of smart strings at all. That would fit my use case quite well, though I'd have to remember to use the flag all the time. It might be nice if the flag could instead be passed to the parser, but I'm not sure whether that is implementable. Regards, Martijn

2 3

[lxml-dev] LXML and persistence with the ZODB?
by Kenneth Miller 28 Jun '08

28 Jun '08

Anyone done anything with this? I know it's simple enough to simply store an XML string, but has anyone bothered with storying lxml objects? Regards, Ken

5 8

[lxml-dev] Mac OS X Status?
by Kenneth Miller 27 Jun '08

27 Jun '08

Has support for OS X improved at all? I still find "easy_install lxml" failing. Regards, Kenneth Miller

3 3

[lxml-dev] easyinstall 2.0.7 does not work
by eric casteleijn 27 Jun '08

27 Jun '08

Hi, when I do: easy_install-2.4 lxml==2.0.7 I get: Searching for lxml==2.0.7 Reading http://pypi.python.org/simple/lxml/ Reading http://codespeak.net/lxml Best match: lxml 2.0.7 Downloading http://codespeak.net/lxml/lxml-2.0.7.tgz error: Can't download http://codespeak.net/lxml/lxml-2.0.7.tgz: 404 Not Found I see that there's no download link for 2.0.7 on the codespeak.net/lxml news page, and somehow setuptools ends up looking there. easy_install-2.4 lxml==2.0.6 works just fine (but also takes the download link on the codespeak page.) -- - eric casteleijn http://infrae.com

2 1

[lxml-dev] schema validation and resolvers
by Michael Ballbach 26 Jun '08

26 Jun '08

I've been trying to use etree.XMLSchema() to load a schema that has external imports - but I'd like to utilize an etree.Resolver object. In my application schemas aren't always on disk. This does not work, because the libxml2 xmlSchemaAddSchemaDoc() function, when called to prepare the import, creates a new parser context and the _local_resolver() function is unable to map this unknown context to a Resolvers object. I've made a simple patch but I'll preface this by saying that I don't know libxml2 or lxml internals very well, so I may well have misunderstood something about this or there may be a way to do this (other than the base_url trick, which works, but as I said my schemas aren't always in files, and when they are they aren't always in the same directories). The naive way to fix this is: Index: parser.pxi =================================================================== --- parser.pxi (revision 56012) +++ parser.pxi (working copy) @@ -393,13 +393,22 @@ context._storage.add(data) return c_input +cdef xmlparser.xmlParserCtxt* _findDefaultParserContext() with gil: + return __GLOBAL_PARSER_CONTEXT.getDefaultParser()._getParserContext()._c_ctxt + cdef xmlparser.xmlParserInput* _local_resolver(char* c_url, char* c_pubid, xmlparser.xmlParserCtxt* c_context) nogil: # no Python objects here, may be called without thread context ! # when we declare a Python object, Pyrex will INCREF(None) ! cdef xmlparser.xmlParserInput* c_input cdef int error + + # check the default parser to support contexts generated within libxml2 if c_context._private is NULL: + if _findDefaultParserContext() is not NULL: + c_context = _findDefaultParserContext() + + if c_context._private is NULL: if __DEFAULT_ENTITY_LOADER is NULL: return NULL return __DEFAULT_ENTITY_LOADER(c_url, c_pubid, c_context) This will default to using the thread's default parser's _ParserDictionaryContext object when none was found inside the passed context. This 'makes sense' to me in the sense that when I was first debugging this problem one of the first things I tried was using the default parser, which I figured might come into play for any parsing that happens behind the scenes. This works fine and I'm able to use a Resolver now - as long as I add it to the default parser's resolvers list. However, it's not 'correct' in the sense that in reality it should probably use the resolvers associated with the schema's original parser. (I think it could be argued that the patch is correct in the abstract in that it catches resolution requests that would otherwise be missed, but that it is incorrect in the sense that the more specific resolver associated with the document's original parser is what should be found in this particular case.) From my perspective this could be fixed in one of two basic ways: 1) Modify libxml2 to somehow get lxml's _private stuff in there. This could be by passing a user specified context for use in new parses or something like that. I'd imagine this is much less likely to work out than option 2. 2) Use the _ParserDictionaryContext system there to store state about when the schema code is entered so that a proper XML context can be inferred and the original document's parser's resolvers can be called. One must be careful here to make sure that any use of lxml within the resolver callbacks does not mess up this state. I'd love feedback about this issue, and I'd be happy to implement one of these changes or something else, whatever makes sense to folks, as my long term interest is in having this work 'out-of-the-box'. In closing, thanks to all you fellows working on lxml, it's really great! -- Michael Ballbach, N0ZTQ ballbach(a)rten.net -- PGP KeyID: 0xA05D5555 http://www.rten.net/

2 4

[lxml-dev] Problem handling  
by Brad Smith 24 Jun '08

24 Jun '08

Hello, I am trying to handle some html data (the content of which I don't have control over) with lxml. The problem is that whenever   is encountered lxml.etree.fromstring throws "XMLSyntaxError: Entity 'nbsp' not defined" and parsing fails. I have to admit I'm at a loss for how to deal with this. I've looked up the DTDs for html and xhtml and the entity isn't defined there, so where would it be or, since I just want to store certain bits of the content, not render it, can I make lxml less picky? The lxml.html.soupparser can handle entities, but actually mis-interprets the html because it is "too" well-formed. For example, the author does: <a name="foo"/> foostuff Which soupparser interprets as <a name="foo"> <b<foostuff </a> ...wrongly "correcting" the original markup. Any help here would be greatly appreciated. --Brad

2 1

[lxml-dev] lxml 2.1 beta3 released
by Stefan Behnel 24 Jun '08

24 Jun '08

Hi all, I'm proud to release lxml 2.1beta3 to PyPI. This is the first lxml release that builds and works on Python 2.3, 2.4, 2.5, 2.6 (beta) and 3.0 (beta). Unusual for a beta release, the third beta contains more new features than bug fixes, which is largely (but not only) due to adaptations with respect to Python 3. The changelog follows below. I expect this to be the last beta release before 2.1 final. Feedback is very much appreciated, especially on the "experimental" features like the namespace cleanup function and on Python 2.6/3.0 support. Your feedback will help in making the final release the best lxml ever. Have fun, Stefan 2.1beta3 (2008-06-19) Features added * Major overhaul of tools/xpathgrep.py script. * Pickling ElementTree objects in lxml.objectify. * Support for parsing from file-like objects that return unicode strings. * New function etree.cleanup_namespaces(el) that removes unused namespace declarations from a (sub)tree (experimental). * XSLT results support the buffer protocol in Python 3. * Polymorphic functions in lxml.html that accept either a tree or a parsable string will return either a UTF-8 encoded byte string, a unicode string or a tree, based on the type of the input. Previously, the result was always a byte string or a tree. * Support for Python 2.6 and 3.0 beta. * File name handling now uses a heuristic to convert between byte strings (usually filenames) and unicode strings (usually URLs). * Parsing from a plain file object frees the GIL under Python 2.x. * Running iterparse() on a plain file (or filename) frees the GIL on reading under Python 2.x. * Conversion functions html_to_xhtml() and xhtml_to_html() in lxml.html (experimental). * Most features in lxml.html work for XHTML namespaced tag names (experimental). Bugs fixed * ElementTree.parse() didn't handle target parser result. * Crash in Element class lookup classes when the __init__() method of the super class is not called from Python subclasses. * A number of problems related to unicode/byte string conversion of filenames and error messages were fixed. * Building on MacOS-X now passes the "flat_namespace" option to the C compiler, which reportedly prevents build quirks and crashes on this platform. * Windows build was broken. * Rare crash when serialising to a file object with certain encodings. Other changes * Non-ASCII characters in attribute values are no longer escaped on serialisation. * Passing non-ASCII byte strings or invalid unicode strings as .tag, namespaces, etc. will result in a ValueError instead of an AssertionError (just like the tag well-formedness check). * Up to several times faster attribute access (i.e. tree traversal) in lxml.objectify.

2 1