Mailman 3 August 2009 - lxml - The Python XML Toolkit

[lxml-dev] lxml2.2 doctype missing
by Mary Lei 10 Aug '09

10 Aug '09

I noticed that the xhtml converted from the parse tree has doctype missing. I am using lxml 2.2. Is this bug still not fixed in lxml 2.2 ? -- Mary Lei Software Testing IPAC-NExScl Rm: KS-233 MS: 220-6 Phone: 395-1998

2 3

Re: [lxml-dev] About the position of html parsing by HTML Target parser
by Stefan Behnel 09 Aug '09

09 Aug '09

[bringing this back to the list] Nicholas Dudfield wrote: >>> even then it's only a *byte* at a time, not a *character* at a time). > > I have been feeding it a unicode character at a time and tested it > with some Chinese characters. Ah, sure, that works. >> Searching for the >> above regexp is safe as "<" cannot occur anywhere in the XML data stream >> except for a tag start/end or comment/PI > > or in embedded <script></script> with CDATA eg > $("someSelector").html('<p> ...') Right, I keep forgetting that CDATA is evil. But you can check for CDATA as well. > For clarification, you are saying skip the target parser and feeding > process and just iterate over the elements and regex for the tags? > something like "<tagname\s" but accounting for tags such as <br/> etc I'd try that, yes. Should be a lot faster as you only handle the elements in your code, not the text content, for example. And you always know exactly what the next opening/closing tag must be. You can special case self-closing tags by checking for contained text and children. If there isn't anything in them, be prepared to find the next opening element before finding a closing tag. Having the parsed tree available gives you a lot of information that you can exploit. You may also consider using something like ahocorasick to search for all opening tags at once (note that lxml makes the namespace prefix available for each Element), plus "<![CDATA". Then double-check the result by traversing the tree and fixing up any false positives 'somehow'. http://hkn.eecs.berkeley.edu/~dyoo/python/ahocorasick/ >> I will also check if there is a way to provide the position at the (target) >> parser level, but that needs to fit the current interface. And I currently >> do not have much time to dig into this. > > That would be awesomely ideal :) ... although not that helpful, as I just noted in another post. So I don't think this is a viable solution for now. Stefan

2 11

[lxml-dev] xml vulnerability
by Bryan 08 Aug '09

08 Aug '09

we use this library at our work, and i was asked to find out if xml parsers in lxml are affected for the following xml vulnerability? http://voices.washingtonpost.com/securityfix/2009/08/researchers_xml_securi… thanks, bryan

2 2

[lxml-dev] Bug with whitespace in namespaces
by Christian Zagrodnick 08 Aug '09

08 Aug '09

Hoi, it is possible to create invalid XML with lxml: >>> import lxml.etree >>> import lxml.objectify >>> xml = lxml.objectify.XML('<a/>') >>> xml.set('{a b}c', 'foo') # This should fail! >>> lxml.etree.tostring(xml) '<a xmlns:ns0="a b" ns0:c="foo"/>' >>> lxml.objectify.fromstring(lxml.etree.tostring(xml)) Traceback (most recent call last): ... File "parser.pxi", line 625, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:64741) File "parser.pxi", line 565, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:64084) lxml.etree.XMLSyntaxError: xmlns:ns0: 'a b' is not a valid URI, line 1, column 13 >>> Regards, -- Christian Zagrodnick · cz(a)gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 4 · fax +49 345 1229889 1 Zope and Plone consulting and development

2 2

[lxml-dev] iterating xpath?
by bl8cki 06 Aug '09

06 Aug '09

I was searching the api and found things like iterfind, but it seems that this work with ElementPath I would like to do something like iterxpath. Is there any way to achieve this? Thanks a lot!

2 1

[lxml-dev] Splitting an xml file.
by Chris Wj 06 Aug '09

06 Aug '09

I'm looking for the best way to split an xml file with many children into multiple files with the same parent tags but individual children. Example: Turn this... file1.xml <parent> a whole bunch of stuff... <child> child 1 stuff... </child> <child> child 2 stuff... </child> </parent> Into 2 files... file1a.xml <parent> a whole bunch of stuff... <child> child 1 stuff... </child> </parent> file1b.xml <parent> a whole bunch of stuff... <child> child 2 stuff... </child> </parent> Should I use lxml.etree to find the line numbers and then just use file operations? You guys think that is most efficient?

3 4

[lxml-dev] automatic attribute unicode decode?
by Hervé Cauwelier 06 Aug '09

06 Aug '09

Hi, I'm quite puzzled by the following excerpt: >>> from lxml import etree >>> r = etree.fromstring('<root toto="français" titi="ascii" tata="1"/>' >>> r.attrib {'titi': 'ascii', 'toto': u'fran\xe7ais', 'tata': '1'} In a bare document with no encoding declaration, lxml has decoded itself a string that did not match the ascii table (what heuristic did it use?). Now I have three attributes of two different types. I wonder why the integer was not decoded. ;-) I actually found this in a real-world document with encoding and namespaces (An ODF xml part). Is this a bug to report and how to circumvent it? Thanks, Hervé

3 2

[lxml-dev] Whitespace foiling pretty_print - any fix?
by B Wooster 06 Aug '09

06 Aug '09

f an XML element has whitespace along with sub elements, pretty_print does not work well. This is probably a XML whitespace issue, but is there any way to work around this issue? Here's example code and the output it prints, and what I would like to get: from lxml import etree as ET root = ET.XML("""<a> <b>test</b> </a>""") ET.SubElement(root, "c") ET.SubElement(root, "d") print ET.tostring(root, pretty_print=True) # This prints: """<a> <b>test</b> <c/><d/></a> """ # Would like: """<a> <b>test</b> <c/> <d/> </a> """

2 1

[lxml-dev] Strange segmentation fault if class inherited from objectify.ObjectifiedElement
by Daniel Albeseder 06 Aug '09

06 Aug '09

I just tried this and got an segmentation fault :-( Python 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import objectify >>> from lxml import etree >>> >>> print etree.LXML_VERSION, etree.LIBXML_VERSION (2, 2, 2, 0) (2, 6, 32) >>> >>> class test (objectify.ObjectifiedElement) : ... pass ... >>> good = objectify.Element ("abc") >>> print type (good), repr (good) <type 'lxml.objectify.ObjectifiedElement'> <Element abc at 7f0797587fc8> >>> >>> bad = test ("abc") Segmentation fault I read that you have to instantiate a parser with objectify.makeparser and use the method makeelement from there. However I would expect that at least not Segmentation Fault happens, if someone is using the normal inheritance as used to be. regards Daniel

2 1

[lxml-dev] id function of xpath and parseid
by Daniel Albeseder 06 Aug '09

06 Aug '09

Hi! I wonder why there is no parseid function inside the objectify module? Is there a reason, why this is only in etree? Additionally I wonder, how the `id` function of XPath does work with lxml. I created a schema-aware parser, which reads an XML-file, where some attributes are declared as xs:ID inside the schema. However the `xpath` method always returns an empty list of nodes, even if the IDs given are inside the XML. Example: ---- from lxml import etree xsd = \ """ <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="root" type="rootType"/> <xsd:complexType name="rootType"> <xsd:sequence> <xsd:element name="foo" type="fooType" maxOccurs="unbounded"/> </xsd:sequence> <xsd:attribute name="id" type="xsd:ID" use="required"/> </xsd:complexType> <xsd:complexType name="fooType"> <xsd:attribute name="id" type="xsd:ID" use="optional"/> </xsd:complexType> </xsd:schema> """ xml = """<root id="a"><foo id="b"/><foo id="bar"/><foo/></root>""" schema = etree.XMLSchema (etree.XML (xsd)) parser = etree.XMLParser (schema = schema) root = etree.XML (xml, parser = parser) for i in ["a", "b", "bar"] : print root.xpath ("id('%s')" % i) print "---" print root.get ("id") for el in root : print el.get ("id") --- Returns just: [] [] [] --- a b bar None where the first three lists should not be empty IMHO. Can anyone help me? Best regards Daniel

2 3