Mailman 3 July 2010 - lxml - The Python XML Toolkit

[lxml-dev] building lxml from source using Mingw32
by gialloporpora July 12, 2010

July 12, 2010

Dear all, I have a problem building lxml from source using Mingw compiler. I have already built lxml from source creating the installer using the Visual C++, but now I have updated Python to 2.7 and I would like to create the installer using Mingw (I have not MS Visual C++ installed on this machine). I have followed all instructions on the lxml documentation page, I have downloaded the packages from: ftp://ftp.zlatkovic.com/pub/libxml/ but if I try to build the installer I receive many errors messages, I have used this commands: python setup.py build --compiler mingw32 --static bdist_wininst (if you want, I could report all the messages of the output). I have tried to solve this myself using Google but I am not very expert and I have not solved. Reading this document: http://docs.python.org/install/ It seem that I must convert all DLL's in the iconv, libxml, libxslt and zlib package using the pexports tool suggested in the page. Is it true? If the reply to previous question is affermative, there exist a package of DLL's already compatible with Mingw? If not, no problem I could install the Visual C++ and building the installer using it. Sandro

3 3

[lxml-dev] (unofficial) lxml 2.2.6 Windows Binaries for Python 2.6
by Sidnei da Silva July 8, 2010

July 8, 2010

HI all, I finally managed to get the builds going again. I have some unofficial builds for ready for testing, and will upload them to pypi there's confirmation that they work just fine: http://bit.ly/bNGEol Worth of note is the fact that the x64 binaries have been stripped of iconv support because I couldn't figure out (yet!) how to build that one properly. According to Stefan, that means that the x64 build won't be able to parse even ISO8859-* encoded files. YMMV. If you're interested in some background about why it took so long, there's more information on my blog: http://bit.ly/9GCgvn Thanks for your patience, and get on testing! -- Sidnei

1 0

[lxml-dev] Python3 support (easy, not messy)
by Rasmus Schøler Sørensen July 7, 2010

July 7, 2010

Hi, Is there any easy (i.e. no source) way to get lxml to work with python3? I'm on an ubuntu (9.04) system with python 3.0.1 installed using "apt-get install python3". If I try to use "from lxml import etree" in a python script, I get "ImportError: No module named lxml". So I go to http://codespeak.net/lxml/installation.html and see that I need to "easy_install lxml". Great, I go to http://peak.telecommunity.com/DevCenter/EasyInstall to see how to get easy_install. Under http://peak.telecommunity.com/DevCenter/EasyInstall#installing-easy-install it tells me to go to http://pypi.python.org/pypi/setuptools. Ok, I go there and find instructions for Windows, RPM and "Cygwin, Mac, Linux, Other" - it tells me to find my appropriate egg in the file list. I look in the file list and see versions 2.3 to 2.6. So, I ask: No 3.0 ?? So, the easy_install'er -- does not work (out-of-the-box) with 3.0... does this also mean that there is no easy way to install lxml on python 3.0? If not, I really think you should add a note about that on http://codespeak.net/lxml/installation.html -- it's rather much trouble to go through to figure out. Cheers,

4 6

[lxml-dev] lxml eggs for Mac OS X 10.4 (maybe 10.5?)
by "Pascal Oberndörfer" July 7, 2010

July 7, 2010

All, Don't know if these eggs are of help to anybody else on "older" Macs (i.e. 10.4 on ppc and i386): <http://www.urheberrecht.org/sw/python/lxml/10.4/lxml22/> I needed lxml on such machines, so here they are at your own risk! The 2.2.6 eggs should statically link to: -libxml2-2.7.7 -libxslt-1.1.26 -libiconv-1.13.1 -zlib-1.2.5 Only tested with 'selftest.py' and 'selftest2.py'. All tests pass. But further testing is certainly needed! Anybody with PyPI rights, feel free to upload them, as I cannot guarantee the above link will remain forever ;-) Thanks. Pascal

3 3

[lxml-dev] schema validation issue
by Michael Bayer July 7, 2010

July 7, 2010

I was playing with the libxml versions that underlie my lxml installation in order to help someone with the "static deps" flag. I noticed that when I build against the latest libxml (2.7.7), as opposed to the older one I have lying around on my mac (2.2.6) , some validations change behavior. The script below illustrates three XML docs that validate completely with 2.2.6 but the second two fail on 2.7.7. My question is, does <xs:all minOccurs="0"> elem1, elem2, .. </xs:all> mean that *any* of elem1, elem2, etc. can be present , or that every subelement, or none, must be present ? In 2.7.7 it seems to be interpreting it as, "all of elem1, elem2, elem3, ... must be present, or none". This doesn't match what I can find in the xml-schema spec (http://www.w3.org/TR/xmlschema-0/#ref18, "All the elements in the group may appear once or not at all, and they may appear in any order.") and various online tutorials (http://www.w3schools.com/Schema/el_all.asp, "The example above indicates that the "firstname" and the "lastname" elements can appear in any order and each element CAN appear zero or one time!"). So is libxml wrong or am I misinterpreting ? How would I lay out a tag like <parent> with each child tag being independently optional ? The other one is how to make an "xs:int" that is optional, but that one I haven't researched yet. Previously, I could say <int-attr/> without issue, now it says its an invalid integer. from lxml import etree from StringIO import StringIO schema = """ <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:element type="parent-type" name="parent"/> <xs:complexType name="parent-type"> <xs:all maxOccurs="1" minOccurs="0"> <xs:element type="xs:int" name="int-attr"/> <xs:element type="xs:string" name="str-attr"/> </xs:all> </xs:complexType> </xs:schema> """ xmlschema = etree.XMLSchema(etree.parse(StringIO(schema))) # passes doc1 = """ <parent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.example.com/xml/schemas"> <str-attr>some value</str-attr> <int-attr>12</int-attr> </parent> """ # fails. it wants both "int-attr" and "str-attr" to be present. # didn't think this was how "xs:all" worked ? doc2 = """ <parent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.example.com/xml/schemas"> <int-attr>12</int-attr> </parent> """ # fails. doesn't allow blank for "type='xs:int'". doc3 = """ <parent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.example.com/xml/schemas"> <str-attr>some value</str-attr> <int-attr></int-attr> </parent> """ for i, doc in enumerate((doc1, doc2, doc3)): doc = etree.parse(StringIO(doc)) try: xmlschema.assertValid(doc) print "document %d is valid." % i except Exception, e: print "document %d is not valid." % i print e output: document 0 is valid. document 1 is not valid. Element 'parent': Missing child element(s). Expected is ( str-attr )., line 2 document 2 is not valid. Element 'int-attr': '' is not a valid value of the atomic type 'xs:int'., line 4

2 3

[lxml-dev] validating one document with multiple xml schemas
by Burak Arslan July 7, 2010

July 7, 2010

hi, i'm sorry if i'm asking the obvious here, but i could not deduce the proper way of doing this from the api documentation. my schemas are defined using multiple <schema targetNamespace="..."> tags that define data structures for multiple namespaces, which <import /> from each other when necessary. (fyi, this is for validating the input to a soap server, available here: github.com/arskom/soaplib) how do i instantiate an etree.XMLSchema object with multiple schema tags? best regards, burak

3 11

[lxml-dev] Fwd: iterparse() doesn't do dtd_validation
by Adam Bielański July 6, 2010

July 6, 2010

Hi, Can anyone help me with validating XML file against internal DTD with iterparse()? I just can't make iterparse() use dtd_validation flag. I ended up with two execution paths, one that uses etree.XMLParser and finds all errors in validated file and another one which uses iterparse and just prints all elements from XML file. Below is the code I used, and attached are sample XML and DTD files I used. Just to make it clear: I'm using lxml in version 2.2.4 on Python 2.6.5. The XmlWithDTDStream class (mentioned in code below) behaves like a stream, returning XML declaration (first line of well-formed XML file), DTD string and then - rest of XML file. It works correctly, since you can see that XMLParser returns correct errors and iterparse returns actual elements from input.xml. <pre><code> from lxml import etree if __name__ == "__main__": print "XMLParser" with open("internalSchema.dtd") as dtdFile: dtd = dtdFile.read() stream = XmlWithDTDStream(dtd, 'input.xml') parser = etree.XMLParser(dtd_validation=True, load_dtd=True) try: root = etree.XML(stream.read(32768), parser) except Exception, e: print "An exception:: ", e print "Error log:: ", parser.error_log print "iterparse: " with open("internalSchema.dtd") as dtdFile: dtd = dtdFile.read() stream = XmlWithDTDStream(dtd, 'input.xml') for aTuple in etree.iterparse(stream, dtd_validation=True, load_dtd=True): print aTuple </code></pre> The result is then as follows: <pre> XMLParser An exception:: No declaration for attribute badAttribute of element pName, line 23, column 26 Error log::<string>:23:26:ERROR:VALID:DTD_UNKNOWN_ATTRIBUTE: No declaration for attribute badAttribute of element pName <string>:25:7:ERROR:VALID:DTD_UNKNOWN_ELEM: No declaration for element property--badName <string>:26:5:ERROR:VALID:DTD_CONTENT_MODEL: Element propertyGroup content does not follow the DTD, expecting (property)+, got (property--badName ) <string>:29:29:ERROR:VALID:DTD_UNKNOWN_ELEM: No declaration for element name iterparse: (u'end',<Element pName at 13e6450>) (u'end',<Element pValue at 13e64e0>) (u'end',<Element property--badName at 13e6510>) (u'end',<Element propertyGroup at 13e6540>) (u'end',<Element name at 13e6570>) (u'end',<Element attributeInstance at 13e65a0>) (u'end',<Element attributeGroup at 13e65d0>) (u'end',<Element rootElement at 13e63f0>) </pre> Hope someone could assist me, or state that this is a bug in lxml. My issues comes from the fact that I need to validate files that are too large to be read into memory using etree.XML(). If there is a way I could do it without iterparse - I'd gladly learn it. Best regards, Adam.

2 5

[lxml-dev] Handling charset properly in lxml.html.parse() and friends
by Dan Egnor July 4, 2010

July 4, 2010

I'm not sure if this is a bug, a missing feature, maybe something that should be discussed in the documentation/FAQ more clearly, or maybe I'm just a moron and missed something obvious, but: Handling charsets in HTML is always challenging. Charset information may come from an HTTP Content-Type header (if the HTML comes via HTTP, which often it does), from an http-equiv meta tag in the file itself, from some other external assertion, or it may need to be guessed from the content. The lxml documentation gives this advice at one point: Similarly, you will get errors when you try the same with HTML data in a unicode string that specifies a charset in a meta tag of the header. You should generally avoid converting XML/HTML data to unicode before passing it into the parsers. It is both slower and error prone. However, if you take that advice, then I don't know any way to supply external charset information to lxml.html.parse() (or lxml.html.document_fromstring() and so on). These functions also don't seem to do much in the way of charset guessing. The *html5lib* parser has a "guess_charset" flag, but the ordinary HTML parser apparently does not. I have resorted to calling chardet myself and converting the input bytes to Unicode using whatever charset I get from chardet, and passing Unicode to lxml.html.document_fromstring(), in explicit contravention of the advice quoted above. This ignores HTTP headers (which I do have access to in my application, which is a content transforming proxy) as well as meta tags in the document, which is a bummer. Ideally, the HTML parsing functions would accept a byte string along with any charset assertions you might have access to, and then do the right thing (using your assertions, looking for meta tags, and applying chardet, in some order). _Really_ ideally it would even parse Content-type header values (since it will need to do so to understand the meta tag anyway). Maybe there's a way to get it to do something like this, but if so I couldn't figure it out. Are there currently known best practices? Searching around the forums, I find other puzzled users, but no crisp answers. I'm happy to help however I can. -- egnor

2 1

[lxml-dev] html.fromstring returning encoded string from a non unicoded string source
by matt barto July 2, 2010

July 2, 2010

Hello, I am trying to obtain a title from a website which has a Unicode tm and register mark, but the Unicode behavior is not what I expect. tree = html.fromstring("<html><body><h1>Apple® - iPad with Wi-Fi - 16GB</h1></body></html>" print tree.text_content() --------> print(tree.text_content()) Apple® - iPad™ with Wi-Fi - 16GB I would expect the output from the print is "Apple® - iPad with Wi-Fi - 16GB", but it seems some encoding occurred during the tree creation. The browser is able to handle these html characters correctly using "iso-8856-1" char set (even though the document says "utf-8"). Can you provide some insight how these two html tags are handled and what I can do to have the expected behavior? My lxml version is 2.2.2. Thanks in advanced. Best, Matt

2 1

[lxml-dev] iterparse() doesn't do dtd_validation
by Adam Bielański July 2, 2010

July 2, 2010

Hi, Can anyone help me with validating XML file against internal DTD with iterparse()? I just can't make iterparse() use dtd_validation flag. I ended up with two execution paths, one that uses etree.XMLParser and finds all errors in validated file and another one which uses iterparse and just prints all elements from XML file. Below is the code I used, and attached are sample XML and DTD files I used. Just to make it clear: I'm using lxml in version 2.2.4 on Python 2.6.5. The XmlWithDTDStream class (mentioned in code below) behaves like a stream, returning XML declaration (first line of well-formed XML file), DTD string and then - rest of XML file. It works correctly, since you can see that XMLParser returns correct errors and iterparse returns actual elements from input.xml. <pre><code> from lxml import etree if __name__ == "__main__": print "XMLParser" with open("internalSchema.dtd") as dtdFile: dtd = dtdFile.read() stream = XmlWithDTDStream(dtd, 'input.xml') parser = etree.XMLParser(dtd_validation=True, load_dtd=True) try: root = etree.XML(stream.read(32768), parser) except Exception, e: print "An exception:: ", e print "Error log:: ", parser.error_log print "iterparse: " with open("internalSchema.dtd") as dtdFile: dtd = dtdFile.read() stream = XmlWithDTDStream(dtd, 'input.xml') for aTuple in etree.iterparse(stream, dtd_validation=True, load_dtd=True): print aTuple </code></pre> The result is as follows: <pre> XMLParser An exception:: No declaration for attribute badAttribute of element pName, line 23, column 26 Error log:: <string>:23:26:ERROR:VALID:DTD_UNKNOWN_ATTRIBUTE: No declaration for attribute badAttribute of element pName <string>:25:7:ERROR:VALID:DTD_UNKNOWN_ELEM: No declaration for element property--badName <string>:26:5:ERROR:VALID:DTD_CONTENT_MODEL: Element propertyGroup content does not follow the DTD, expecting (property)+, got (property--badName ) <string>:29:29:ERROR:VALID:DTD_UNKNOWN_ELEM: No declaration for element name iterparse: (u'end', <Element pName at 13e6450>) (u'end', <Element pValue at 13e64e0>) (u'end', <Element property--badName at 13e6510>) (u'end', <Element propertyGroup at 13e6540>) (u'end', <Element name at 13e6570>) (u'end', <Element attributeInstance at 13e65a0>) (u'end', <Element attributeGroup at 13e65d0>) (u'end', <Element rootElement at 13e63f0>) </pre> Hope someone could assist me, or state that this is a bug in lxml. My issues comes from the fact that I need to validate files that are too large to be read into memory using etree.XML(). If there is a way I could do it without iterparse - I'd gladly learn it. Best regards, Adam.

1 0