Mailman 3 September 2012 - lxml - The Python XML Toolkit

lxml eats all memory
by Kevin Leo 25 Sep '12

25 Sep '12

Hello, I’m testing the performance of lxml. My test method is: Given a lot of HTML files (About 50000). For each HTML file content, calculate the time spending in lxml.html.fromstring. Finally, calculate the total time. At the beginning, the test process used about 60M physical memory. But after processing thousands of HTML files, the test process will consume all memory quickly when processing a HTML file. “strace -p” shows it do nothing but brk(). First, I thought it’s caused by a strange HTML file. But the result of fromstring that file only is OK. Here is lxml info: 1. sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0) 2. lxml.etree : (2, 3, 5, 0) 3. libxml used : (2, 8, 0) 4. libxslt used : (1, 1, 26) 5. OS: Linux dev-test 2.6.32-gentoo-r29 #1 SMP Fri Jun 1 11:21:29 CST 2012 x86_64 Intel(R) Pentium(R) Dual CPU E2180 @ 2.00GHz GenuineIntel GNU/Linux

2 1

Memory debugging
by Christian Heimes 20 Sep '12

20 Sep '12

Hello, I'd like to suggest a trival feature for lxml. libxml2 and libxslt support extended memory debugging when both libraries are compiled with the the '--with-mem-debug' flag [1]. This is useful to debug how many memory is actually used by the XML and XSTL code. In order to show the amount of used memory and memory blocks, the two functions xmlMemBlocks() [2] and xmlMemUsed() [3] needs to be wrapped. Both functions are available in non-debug builds and return 0 in that case. It might be worth to wrap the other debug helpers like xmlMemDisplay(FILE *) [4], too. Regards, Christian Example patch: diff -ru lxml-2.3.4.orig/src/lxml/lxml.etree.pyx lxml-2.3.4/src/lxml/lxml.etree.pyx --- lxml-2.3.4.orig/src/lxml/lxml.etree.pyx 2012-03-26 12:28:58.000000000 +0200 +++ lxml-2.3.4/src/lxml/lxml.etree.pyx 2012-05-03 18:33:00.702130662 +0200 @@ -2956,6 +2956,11 @@ except _TargetParserResult, result_container: return result_container.result +def memoryUsed(): + return tree.xmlMemUsed() + +def memoryBlocks(): + return tree.xmlMemBlocks() ################################################################################ # Include submodules diff -ru lxml-2.3.4.orig/src/lxml/tree.pxd lxml-2.3.4/src/lxml/tree.pxd --- lxml-2.3.4.orig/src/lxml/tree.pxd 2011-09-25 18:58:05.000000000 +0200 +++ lxml-2.3.4/src/lxml/tree.pxd 2012-05-03 17:44:50.323798041 +0200 @@ -322,6 +322,7 @@ cdef extern from "libxml/xmlmemory.h": cdef void* xmlMalloc(size_t size) nogil cdef int xmlMemBlocks() nogil + cdef int xmlMemUsed() nogil cdef extern from "etree_defs.h": cdef bint _isElement(xmlNode* node) nogil [1] http://xmlsoft.org/html/libxml-xmlmemory.html#DEBUG_MEMORY [2] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemBlocks [3] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemUsed [4] http://xmlsoft.org/html/libxml-xmlmemory.html#xmlMemDisplay

2 2

Re: [lxml] "xmlns: URI is not absolute" mystery/bug?
by Andrew McNamara 17 Sep '12

17 Sep '12

>> I don't actually need or want the validation - what I do want is the >> type hints it provides to objectify. Is there a way to pass the schema >> type information to objectify without also performing validation? > >I think you're mistaken: There is no information taken from the schema >for the objectify type annotations. Indeed you are correct - I am mistaken. I did several experiments and thought I saw it casting to the schema-specified types, but I just repeated them and it must just have been too much wishful thinking. >This comes up from time to time but so far no one has made an effort to >implement a generalized solution for lxml. > >See also >https://mailman-mail5.webfaction.com/pipermail/lxml/2008-January/003245.html Paraphrasing (I think): libxml2 doesn't make the schema information readily accessible, which makes it difficult to implement this efficiently. Thanks for your help! -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

1 0

"xmlns: URI is not absolute" mystery/bug?
by Andrew McNamara 17 Sep '12

17 Sep '12

I'm trying to parse some XML using a parser with schema. The XML contains a data error, and should raise: lxml.etree.XMLSyntaxError: Element '{xmlapi_1.0}routerId': [facet 'minInclusive'] The value '0' is less than the minimum value allowed ('1'). Instead, I'm getting lxml.etree.XMLSyntaxError: xmlns: URI xmlapi_1.0 is not absolute, line 2, column 17 I suspect what is happening is the "URI xmlapi_1.0 is not absolute" is a warning and is being ignored, but when the later data error is encountered, the "URI is not absolute" is the first message found and it is being erroneously returned instead. As an aside, is it possible to ignore the above data error somehow and continue loading the XML (I don't control the source of data, nor the schema - both are massive and machine generated)? The code below demonstrates the problem: from lxml import objectify, etree schema = etree.XMLSchema(etree.fromstring('''\ <?xml version="1.0" encoding="UTF-8"?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="xmlapi_1.0" xmlns="xmlapi_1.0" elementFormDefault="qualified"> <xsd:element name="container"> <xsd:complexType> <xsd:all> <xsd:element name="routerId" minOccurs="0"> <xsd:simpleType> <xsd:restriction base="xsd:int"> <xsd:minInclusive value="1"/> <xsd:maxInclusive value="10240"/> </xsd:restriction> </xsd:simpleType> </xsd:element> </xsd:all> </xsd:complexType> </xsd:element> </xsd:schema> ''')) parser = objectify.makeparser(schema=schema) fails = '''\ <?xml version="1.0"?> <container xmlns="xmlapi_1.0"> <routerId>0</routerId> </container> ''' result = objectify.fromstring(fails, parser) -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/

2 2

lxml and Unicode
by Martin Mueller 16 Sep '12

16 Sep '12

Jens Quade wrote: etree.parse expects a filename or a file-like object. try doc = etree.XML(text) - it expects a string containing XML. That works but raises a documentation problem: I didn't see anything to that effect in the lxml documentation, at least not in the tutorial. I don't quite understand the following: doc = etree.parse(text) works when it is part of a separate script. It doesn't work in the combined script, where all of a sudden the 'parse' instruction is unnecessary. But I'm grateful for help that makes things work. M I'm new to lxml and a little wobbly on Unicode in Python 2.7. Here is my problem: 1. I take an SGML text with character entities and convert them to UTF-8 characters 2. I extract note elements from the text with the converted characters. This works if I use two separate Python scripts. The first goes like this: # This Python file uses the following encoding: utf-8 import entityrules filein = open('/users/martin/dropbox/eebo-SGMLfiles/200504/A10675.sgm', 'r') fileout = open('/users/martin/dropbox/lxml/A10675.xml', 'w') text = filein.read() text = entityrules.replaceCharacterEntities(text) print >> fileout, text It takes the SGML file as an input. applies the "replaceCharacterEntities" and writes that output to a new file. The second script goes like this: #encoding = 'utf-8' from lxml import etree filein ='/users/martin/dropbox/lxml/a10675.xml' fileout = open('/users/martin/dropbox/lxml/notesA10675.xml', 'w') doc = etree.parse(filein) for note in doc.getiterator('note'): print >>fileout, etree.tostring(note, with_tail = False, encoding = 'utf-8') Its input is the file generated by the first script. It converts the input file into an element tree and then iterates through the tree, extracting a particular element . Now it should be possible to combine those two scripts in a single script as follows: # This Python file uses the following encoding: utf-8 import entityrules from lxml import etree #converting the SGML character entities to UTF-8 characcters filein = open('/users/martin/dropbox/eebo-SGMLfiles/200504/A10675.sgm', 'r') fileout = open('/users/martin/dropbox/lxml/notesA10675.xml', 'w') text = filein.read() text = entityrules.replaceCharacterEntities(text) print "I finished the first part of this script" #creating an element tree and extracting instances of a particular element doc = etree.parse(text) for note in doc.getiterator('note'): print >>fileout, etree.tostring(note, with_tail = False, encoding = 'utf-8') This script produces the following output: Martin-Muellers-Mac-Pro:lxml martin$ python getNotes.py I finished the first part of this script Traceback (most recent call last): File "getNotes.py", line 14, in <module> doc = etree.parse(text) File "lxml.etree.pyx", line 2954, in lxml.etree.parse (src/lxml/lxml.etree.c:56220) File "parser.pxi", line 1533, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:82303) File "parser.pxi", line 1562, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:82596) File "parser.pxi", line 1462, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:81635) File "parser.pxi", line 1002, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:78544) File "parser.pxi", line 569, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:74488) File "parser.pxi", line 650, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:75379) File "parser.pxi", line 584, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:74650) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1126: ordinal not in range(128) The script does the first part of the job correctly, as is apparent from the first line. And the "text" variable that is passed to the etree function should in principle be identical with the content of the file that is passed to my script from a file. But it isn't, and somewhere along the road Python seems to have forgotten that we 're dealing with Unicode texts. What am I doing wrong? Or is there a bug? I am using the latest Enthought version of Python, which comes bundled with lxml Martin Mueller Professor of English and Classics Northwestern University

2 1

lxml and Unicode
by Martin Mueller 14 Sep '12

14 Sep '12

I ma new to lxml and a little wobbly on Unicode in Python 2.7. Here is my problem: 1. I take an SGML text with character entities and convert them to UTF-8 characters 2. I extract note elements from the text with the converted characters. This works if I use two separate Python scripts. The first goes like this: # This Python file uses the following encoding: utf-8 import entityrules filein = open('/users/martin/dropbox/eebo-SGMLfiles/200504/A10675.sgm', 'r') fileout = open('/users/martin/dropbox/lxml/A10675.xml', 'w') text = filein.read() text = entityrules.replaceCharacterEntities(text) print >> fileout, text It takes the SGML file as an input. applies the "replaceCharacterEntities" and writes that output to a new file. The second script goes like this: #encoding = 'utf-8' from lxml import etree filein ='/users/martin/dropbox/lxml/a10675.xml' fileout = open('/users/martin/dropbox/lxml/notesA10675.xml', 'w') doc = etree.parse(filein) for note in doc.getiterator('note'): print >>fileout, etree.tostring(note, with_tail = False, encoding = 'utf-8') Its input is the file generated by the first script. It converts the input file into an element tree and then iterates through the tree, extracting a particular element . Now it should be possible to combine those two scripts in a single script as follows: # This Python file uses the following encoding: utf-8 import entityrules from lxml import etree #converting the SGML character entities to UTF-8 characcters filein = open('/users/martin/dropbox/eebo-SGMLfiles/200504/A10675.sgm', 'r') fileout = open('/users/martin/dropbox/lxml/notesA10675.xml', 'w') text = filein.read() text = entityrules.replaceCharacterEntities(text) print "I finished the first part of this script" #creating an element tree and extracting instances of a particular element doc = etree.parse(text) for note in doc.getiterator('note'): print >>fileout, etree.tostring(note, with_tail = False, encoding = 'utf-8') This script produces the following output: Martin-Muellers-Mac-Pro:lxml martin$ python getNotes.py I finished the first part of this script Traceback (most recent call last): File "getNotes.py", line 14, in <module> doc = etree.parse(text) File "lxml.etree.pyx", line 2954, in lxml.etree.parse (src/lxml/lxml.etree.c:56220) File "parser.pxi", line 1533, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:82303) File "parser.pxi", line 1562, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:82596) File "parser.pxi", line 1462, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:81635) File "parser.pxi", line 1002, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:78544) File "parser.pxi", line 569, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:74488) File "parser.pxi", line 650, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:75379) File "parser.pxi", line 584, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:74650) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1126: ordinal not in range(128) The script does the first part of the job correctly, as is apparent from the first line. And the "text" variable that is passed to the etree function should in principle be identical with the content of the file that is passed to my script from a file. But it isn't, and somewhere along the road Python seems to have forgotten that we 're dealing with Unicode texts. What am I doing wrong? Or is there a bug? I am using the latest Enthought version of Python, which comes bundled with lxml Martin Mueller Professor of English and Classics Northwestern University

2 1

Re: [lxml] Building lxml with mac os x mountain lion fails.
by Mark Grandi 13 Sep '12

13 Sep '12

My mistake, I should of clarified that this was for python 3. Easy_install works great but it doesn't run under python3... From: Brett Sent: 9/12/2012 12:11 AM To: lxml(a)lxml.de Subject: Re: [lxml] Building lxml with mac os x mountain lion fails. mark grandi <markgrandi <at> gmail.com> writes: > > Well, I seem to of figured it out. It seems that libxml2 by default tries to > include lzma, but since I recently installed the xz libraries, its probably > having some conflict with the static vs not static version of liblzma, so i > edited the buildlibxml.py script to pass in a '--without-lzma' option, and > that worked. My entire compile process is listed below. The setup script > REALLY needs to be updated to work with recent versions of mac os x... > > Corvidae:lxml-2.3.5 markgrandi$ echo $CFLAGS > -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform > /Developer/SDKs/MacOSX10.8.sdk/ > > Corvidae:lxml-2.3.5 markgrandi$ echo $LDFLAGS > -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform > /Developer/SDKs/MacOSX10.8.sdk/usr/lib > ...snip... > add '--without-lzma' to that list, so it becomes > > # build libxml2 > libxml2_configure_cmd = configure_cmd + [ > '--without-python', > '--with-iconv=%s' % prefix, > '--without-lzma'] > ...snip... I tried the following and it seems to work fine: lilliput:lxml-2.3.5 brett$ STATIC_DEPS=true sudo easy_install "lxml==2.3.5" lilliput:lxml-2.3.5 brett$ sudo python setup.py install lilliput:lxml-2.3.5 brett$ python >>> from lxml import etree >>> x = etree.fromstring('''<note> <to> Tove </to> <from> Jani </from> <heading> Reminder </heading> <body> Don't forget me this weekend! </body> </note>'''.encode("utf-8")) >>> x <Element note at 0x103b0cfa0> >>> etree.tostring(x) "<note>\n <to>\n Tove\n </to>\n <from>\n Jani\n </from>\n <heading>\n Reminder\n </heading>\n <body>\n Don't forget me this weekend!\n </body>\n </note>" _________________________________________________________________ Mailing list for the lxml Python XML toolkit - http://lxml.de/ lxml(a)lxml.de https://mailman-mail5.webfaction.com/listinfo/lxml

3 2

Building lxml with mac os x mountain lion fails.
by Mark Grandi 12 Sep '12

12 Sep '12

Hello, I'm trying to build lxml on mac os x 10.8 and the build fails. I've also tried to change the compiler from gcc to clang, and it still fails. Output: http://paste.ubuntu.com/1172867/ (this is the output when CC=clang) any ideas? =/ ~mark

4 5

Strange memory leakage problem on Ubuntu
by zzjjzzgggg 12 Sep '12

12 Sep '12

Hello, I am new to lxml. I installed lxml on Ubuntu 12.04 via easy_install as suggested easy_install --allow-hosts=lxml.de,*.python.org lxml The version of libxml2 is Version: 2.7.8.dfsg-5.1ubuntu4.1 I use python 3.2. When I use lxml to process some HTML fragments, I find serious memory leakage issues. The test code and test data are attached. When I remove this line ps=html.xpath("//p[@node-type='feed_list_content']") , There isn't memory leakage problem. What's the problem please? Many thanks. I can provide more info if you need. Best regards zzjjzzgggg

1 0

Strange memory leakage problem on Ubuntu
by zzjjzzgggg 12 Sep '12

12 Sep '12

Hello, I am new to lxml. I installed lxml on Ubuntu 12.04 via easy_install as suggested easy_install --allow-hosts=lxml.de,*.python.org lxml The version of libxml2 is Version: 2.7.8.dfsg-5.1ubuntu4.1 I use python 3.2. When I use lxml to process some HTML fragments, I find serious memory leakage issues. The test code and test data are attached. When I remove this line ps=html.xpath("//p[@node-type='feed_list_content']") , There isn't memory leakage problem. What's the problem please? Many thanks. I can provide more info if you need. Best regards zzjjzzgggg

1 0

lxml - The Python XML Toolkit September 2012