Mailman 3 February 2008 - lxml - The Python XML Toolkit

Re: [lxml-dev] Validating cElementTrees with lxml
by Stefan Behnel 17 Feb '08

17 Feb '08

Hi, Ryan K wrote: > If I have a cElementTree.ElementTree (or the one from the Standard > Library), can I use lxml's validation features on it since it > implements the same ElementTree API? Not directly. lxml and cElementTree use different tree models internally, so you can't just apply C-implemented features of one to the other. Any reason you can't just use lxml /instead/ of cElementTree? In case you really want to combine both: to validate the tree, you only need to create an lxml tree from your ElementTree in a one-way operation, which is easy, as this can be done through the (identical) API. Just create a new lxml Element and then traverse the ElementTree recursively, create a new SubElement for each child you find, and set its text, tail and attrib properties. Something like this might work (though untested): class TreeMigrator(object): def __init__(ET_impl_from, ET_impl_to): self.Element = ET_impl_to.Element self.SubElement = ET_impl_to.SubElement def copyChildren(self, from_parent, to_parent): for from_child in from_parent: tag = from_child.tag if not isinstance(tag, basestring): # skip Comments etc. continue to_child = self.SubElement( to_parent, tag, **from_child.attrib) to_child.text = child.text to_child.tail = child.tail self.copyChildren(from_child, to_child) def __call__(self, from_root): tag = from_root.tag to_root = self.Element(tag, **from_root.attrib) to_root.text = from_root.text to_root.tail = from_root.tail if isinstance(tag, basestring): # skip Comments etc. self.copyChildren(from_root, to_root) return to_root Feel free to finish it up and send it back to the list. :) Stefan

1 0

[lxml-dev] Setting default namespace
by Fabien 16 Feb '08

16 Feb '08

Hello, I've a document with a namespace of the root element of my document and when I display an element .tag, I get it with the namespace. Also, when I'm using xpath(), I need to use the namespaces={..}) parameters. Is there a way to define a default namespace in order to manipulate the document like if it hasn't namespace ? >>> xml = """ ... <root xmlns="http://www.example.com"> ... <a /> ... <b /> ... </root> ... """ >>> tree = etree.fromstring(xml) >>> print tree <Element {http://www.example.com}root at b73e702c> >>> for element in tree: ... print element.tag ... {http://www.example.com}a {http://www.example.com}b Thanks in advance. -- Fabien SCHWOB _____________________________________________________________ Derrière chaque bogue, il y a un développeur, un homme qui s'est trompé. (Bon, OK, parfois ils s'y mettent à plusieurs).

2 1

[lxml-dev] Source code "meta" documentation
by Stefan Behnel 15 Feb '08

15 Feb '08

Hi, Stefan Behnel wrote (elsewhere): > The .pyx and .pxi files are what you want to look at first (maybe I should > write up some "how to read the source" docs...) I started an initial document here: http://codespeak.net/svn/lxml/trunk/doc/lxml-source-howto.txt I'd be happy if others could contribute sections about the parts they know, such as lxml.objectify and lxml.html. Questions that help in focussing the document are also appreciated: what would you like to know for starters? Stefan

1 0

[lxml-dev] One-time memory leak?
by Marius Gedminas 14 Feb '08

14 Feb '08

Hi! I've been using libxml2 (before lxml was even created) and I've built some infrastructure for catching libxml2 memory leaks in my unit tests. Recently I've started using lxml on a completely different project and noticed that my old leak watcher was hooked up -- because it reported a leak. This is most likely a false positive (the "leak" happens only once during the program's lifetime), but I'd like to understand what exactly happens. I'm attaching a short test program that produces this output on my machine: $ bin/python lxml-memleak.py test_libxml2_html: leaked 0 bytes test_libxml2_xml: leaked 0 bytes test_lxml_html: leaked 9423 bytes test_lxml_xml: leaked 9479 bytes This is in a virtualenv sandbox with lxml 2.0.1 from cheeseshop and system-wide libxml2 2.0.30 (plus a security patch or two) from Ubuntu Gutsy. Each of those tests was run in a separate Python process to avoid contamination. Note that if I run the same test more than once, I see no new leaks: $ bin/python lxml-memleak.py test_lxml_html 3 test_lxml_html: leaked 9423 bytes test_lxml_html: leaked 0 bytes test_lxml_html: leaked 0 bytes which leads me to think this "leak" is in fact harmless on-demand initialization of some sort. I would like to improve my leak detector to avoid false positives (it already does a funny dance with initParser/cleanupParser to do so). I've tried looking at the lxml source code but gave up in about 30 seconds. I don't know Cython. I can't tell which is generated code and which is the source for that. I cannot find the entry point that would let me trace how lxml.etree.HTML() is implemented ("HTML" is a pretty ungreppable string). ltrace'ing a Python process failed to notice any dynamic library calls to libxml2's functions. How can I translate the short lxml code snippet from lxml.etree import HTML doc = HTML(sample_document) del doc to low-level libxml2 library function calls and see where it allocates the extra memory? I could, of course, declare lxml to be leak-free and just disable my leak finder, but I cannot resist the opportunity to make sure of it (and for that I need a leak detector without false positives). Regards, Marius Gedminas -- Professionalism has no place in art, and hacking is art. Software Engineering might be science; but that's not what I do. I'm a hacker, not an engineer. -- jwz

2 1

[lxml-dev] c14n, pretty printing and diffing
by Olivier Collioud 13 Feb '08

13 Feb '08

Hello, I would like to use my favourite text diffing tool to compare XML files. Is their a way to produce a pretty printed canonical version of my XML files using lxml ? Thanks, Olivier. ------ World Intellectual Property Organization Disclaimer: This electronic message may contain privileged, confidential and copyright protected information. If you have received this e-mail by mistake, please immediately notify the sender and delete this e-mail and all its attachments. Please ensure all e-mail attachments are scanned for viruses prior to opening or using.

3 2

[lxml-dev] Bug in lxml.html with nameless form submit?
by Paul Winkler 13 Feb '08

13 Feb '08

This is a pretty common idiom in html - to have a submit button that has no name: <input type="submit" /> But lxml.html barfs if you use the form's fields.items() or fields.values() method: >>> import lxml.html >>> tree = lxml.html.fromstring(''' ... <html><body> ... <form> ... <input name="foo" value="bar"/> ... <input type="submit" /> ... </form> ... </body></html> ... ''') >>> tree <Element html at 2b58b8edea78> >>> tree.forms tree.forms >>> tree.forms[0] <Element form at 2b58b8edec80> >>> tree.forms[0].fields <FieldsDict for form 0> >>> tree.forms[0].fields.keys() [None, 'foo'] >>> tree.forms[0].fields.items() Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/UserDict.py", line 112, in items return list(self.iteritems()) File "/usr/lib/python2.4/UserDict.py", line 101, in iteritems yield (k, self[k]) File "/usr/lib64/python2.4/site-packages/lxml-2.0-py2.4-linux-x86_64.egg/lxml/html/__init__.py", line 749, in __getitem__ return self.inputs[item].value File "/usr/lib64/python2.4/site-packages/lxml-2.0-py2.4-linux-x86_64.egg/lxml/html/__init__.py", line 811, in __getitem__ raise KeyError( KeyError: 'No input element with the name None' >>> tree.forms[0].fields.values() Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/UserDict.py", line 110, in values return [v for _, v in self.iteritems()] File "/usr/lib/python2.4/UserDict.py", line 101, in iteritems yield (k, self[k]) File "/usr/lib64/python2.4/site-packages/lxml-2.0-py2.4-linux-x86_64.egg/lxml/html/__init__.py", line 749, in __getitem__ return self.inputs[item].value File "/usr/lib64/python2.4/site-packages/lxml-2.0-py2.4-linux-x86_64.egg/lxml/html/__init__.py", line 811, in __getitem__ raise KeyError( KeyError: 'No input element with the name None' If you give the submit element a name attribute, this doesn't happen. -- Paul Winkler http://www.slinkp.com

3 2

[lxml-dev] lxml 2.0 on Windows
by Evan Grim 12 Feb '08

12 Feb '08

Greetings, I'm a recent lxml convert and am developing a cross-platform application that needs lxml on both windows and linux platforms. I notice that 2.0 was just released and am wondering if whoever it is that so kindly makes the windows installers plans on releasing a 2.0 binary soon. If not, I'll just stick with the latest 1.x series release so that I can be consistent on both platforms. Does anyone know the plan? Cheers, Evan

2 3

Re: [lxml-dev] Fwd: News flash: Python possibly guilty in excessive DTD traffic
by Stefan Behnel 12 Feb '08

12 Feb '08

Hi Sidnei, Sidnei da Silva wrote: > http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic > Does any of that apply to lxml? I don't think so, the article relates to DTD loading through urllib. lxml leaves that to libxml2's parser. > I suppose lxml supports dtd catalogs? Yes, libxml2 has catalog support (although you can compile that out), so it will normally see network access as a last resort to resolve external entities. > Does it cache dtds in any way? There is no internal document caching (except for repeated access to the same document during a single operation, e.g. in XSLT). If you do not provide catalogs on your system, that's your own 'decision'. You can still write your own caching resolver in that case, but I would consider catalogs the best solution to this problem. Stefan > ---------- Forwarded message ---------- > From: Guido van Rossum <guido(a)python.org> > Date: 2008/2/9 > Subject: [Web-SIG] Fwd: [Baypiggies] News flash: Python possibly > guilty in excessive DTD traffic > To: Web SIG <web-sig(a)python.org> > > > ---------- Forwarded message ---------- > From: Keith Dart ♂ <keith(a)dartworks.biz> > Date: Feb 8, 2008 8:03 PM > Subject: [Baypiggies] News flash: Python possibly guilty in excessive > DTD traffic > To: baypiggies(a)python.org > > > http://www.w3.org/blog/systeam/2008/02/08/w3c_s_excessive_dtd_traffic > > This is interesting. I've noticed that when you use Python's XML > package in validating mode it does try to fetch the DTD. Be careful > when you use that.

2 3

Re: [lxml-dev] c14n, pretty printing and diffing
by Olivier Collioud 12 Feb '08

12 Feb '08

For those interested by the iterparse method, the following is much better: sourceTree = ElementTree.iterparse(open(inputDir+'/'+file, 'r'), events=("start", "end")) for event, elem in sourceTree: if event == "start": i = "\n" + depth*" " depth += 1 outputFile.write('%s<%s' % (i,elem.tag)) if len(elem.items()): attrs = elem.items() attrs.sort() outputFile.write(' ') outputFile.write(' '.join(['%s="%s"' % (a[0],a[1]) for a in attrs if a[0] != 'size'])) outputFile.write('>') if elem.text and elem.text.strip(): outputFile.write(elem.text.strip('\n').encode('utf-8')) if event == "end": outputFile.write('%s</%s>' % (i,elem.tag)) if elem.tail and elem.tail.strip(): outputFile.write(elem.tail.strip('\n').encode('utf-8')) depth -= 1 elem.clear() because when event == 'start' then len(elem) is always 0, and I don't how to guess if the element will have some content in order to produce en empty tag (or not). Therefore,the above code always produce an element end tag even when there is no content. >>> "Olivier Collioud" <Olivier.Collioud(a)wipo.int> 12/02/08 7:26 am >>> Thanks Stephan. I prefer visual diffing : the ones provided by Eclipse, TkDiff or WinMerge. I did not fin any doc or usage example of lxml.usedoctest, could you please give some pointer ? Let me share my simple (because I do not use any namespace, PI, comment...) solution based on iterparse: depth = 0 sourceTree = ElementTree.iterparse(open(inputFile, 'r'), events=("start", "end")) for event, elem in sourceTree: if event == "start": i = "\n" + depth*" " depth += 1 outputFile.write('%s<%s' % (i,elem.tag)) if len(elem.items()): attrs = elem.items() attrs.sort() outputFile.write(' ') outputFile.write(' '.join(['%s="%s"' % (a[0],a[1]) for a in attrs if a[0] != 'size'])) if elem.text and elem.text.strip(): outputFile.write('>%s' % elem.text.strip('\n').encode('utf-8')) elif len(elem): outputFile.write('>') if event == "end": if (elem.text and elem.text.strip()) or len(elem): outputFile.write('%s</%s>' % (i,elem.tag)) else: outputFile.write('/>') if elem.tail and elem.tail.strip(): outputFile.write(elem.tail.strip('\n').encode('utf-8')) depth -= 1 elem.clear() Olivier. >>> Stefan Behnel <stefan_ml(a)behnel.de> 11/02/08 7:56 pm >>> Hi, Olivier Collioud wrote: > I would like to use my favourite text diffing tool to compare XML > files. Which is not lxml.html.diff, I assume? (I'm not sure how HTML specific that is, BTW). Also, for doctests, there is lxml.usedoctest that you can import (the lxml web pages use it for doctests). > Is their a way to produce a pretty printed canonical version of my XML > files using lxml ? Not using the c14n interface (libxml2 doesn't support it). Serialising by hand is not too hard, though. You can look at ElementTree._write() for an example: http://svn.effbot.org/public/elementtree/elementtree/ElementTree.py Stefan _______________________________________________ lxml-dev mailing list lxml-dev(a)codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev ------ World Intellectual Property Organization Disclaimer: This electronic message may contain privileged, confidential and copyright protected information. If you have received this e-mail by mistake, please immediately notify the sender and delete this e-mail and all its attachments. Please ensure all e-mail attachments are scanned for viruses prior to opening or using. _______________________________________________ lxml-dev mailing list lxml-dev(a)codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev ------ World Intellectual Property Organization Disclaimer: This electronic message may contain privileged, confidential and copyright protected information. If you have received this e-mail by mistake, please immediately notify the sender and delete this e-mail and all its attachments. Please ensure all e-mail attachments are scanned for viruses prior to opening or using.

2 1

Re: [lxml-dev] c14n, pretty printing and diffing
by Olivier Collioud 12 Feb '08

12 Feb '08

Thanks Stephan. I prefer visual diffing : the ones provided by Eclipse, TkDiff or WinMerge. I did not fin any doc or usage example of lxml.usedoctest, could you please give some pointer ? Let me share my simple (because I do not use any namespace, PI, comment...) solution based on iterparse: depth = 0 sourceTree = ElementTree.iterparse(open(inputFile, 'r'), events=("start", "end")) for event, elem in sourceTree: if event == "start": i = "\n" + depth*" " depth += 1 outputFile.write('%s<%s' % (i,elem.tag)) if len(elem.items()): attrs = elem.items() attrs.sort() outputFile.write(' ') outputFile.write(' '.join(['%s="%s"' % (a[0],a[1]) for a in attrs if a[0] != 'size'])) if elem.text and elem.text.strip(): outputFile.write('>%s' % elem.text.strip('\n').encode('utf-8')) elif len(elem): outputFile.write('>') if event == "end": if (elem.text and elem.text.strip()) or len(elem): outputFile.write('%s</%s>' % (i,elem.tag)) else: outputFile.write('/>') if elem.tail and elem.tail.strip(): outputFile.write(elem.tail.strip('\n').encode('utf-8')) depth -= 1 elem.clear() Olivier. >>> Stefan Behnel <stefan_ml(a)behnel.de> 11/02/08 7:56 pm >>> Hi, Olivier Collioud wrote: > I would like to use my favourite text diffing tool to compare XML > files. Which is not lxml.html.diff, I assume? (I'm not sure how HTML specific that is, BTW). Also, for doctests, there is lxml.usedoctest that you can import (the lxml web pages use it for doctests). > Is their a way to produce a pretty printed canonical version of my XML > files using lxml ? Not using the c14n interface (libxml2 doesn't support it). Serialising by hand is not too hard, though. You can look at ElementTree._write() for an example: http://svn.effbot.org/public/elementtree/elementtree/ElementTree.py Stefan _______________________________________________ lxml-dev mailing list lxml-dev(a)codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev ------ World Intellectual Property Organization Disclaimer: This electronic message may contain privileged, confidential and copyright protected information. If you have received this e-mail by mistake, please immediately notify the sender and delete this e-mail and all its attachments. Please ensure all e-mail attachments are scanned for viruses prior to opening or using.

1 0