Mailman 3 November 2010 - lxml - The Python XML Toolkit

[lxml-dev] Compile failure
by felix 23 Nov '10

23 Nov '10

According to this: http://codespeak.net/lxml/build.html we should avoid installing Cython but using easy_install to build fails saying the cython generated file is missing I tried to build from source (svn checkout) and had the same issue Last Changed Author: scoder Last Changed Rev: 78186 Last Changed Date: 2010-10-21 13:48:54 -0400 (Thu, 21 Oct 2010) so I install cython anyway and try to compile the source, this also fails : Error converting Pyrex file to C: ------------------------------------------------------------ ... c_child = _findChildForwards(c_node, 0) while c_child is not NULL: if c_child.type == tree.XML_ELEMENT_NODE: for i in range(c_tag_count): if _tagMatchesExactly(c_child, c_ns_tags[2*i], c_ns_tags[2*i+1]): c_next = _findChildForwards(c_child, 0) or _nextElement(c_child) ^ ------------------------------------------------------------ /home/crucial/tmp/lxml/src/lxml/cleanup.pxi:246:64: Cannot assign type 'int' to 'xmlNode *' building 'lxml.etree' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include -I/usr/local/include/libxml2 -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w src/lxml/lxml.etree.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation. *but then I succeeded with the old sudo easy_install lxml* because now I have Cython just FYI. so the advice on the build page says not to install Cython but that was the only way I got it to work. building from source in that checkout failed

3 3

[lxml-dev] HTMLParser and \r converted in  html entity
by Matteo Parrucci 20 Nov '10

20 Nov '10

Hi, I'm new here; I subscribed because I encountered a strange behavior in lxml: Is it normal that "\r" followed by "\n" in html code get converted in "" entity using HTMLParser? the strange behavior is reproduced in the example that follows. import lxml.etree g='<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"<http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd>>\r\n<html xmlns="http://www.w3.org/1999/xhtml"<http://www.w3.org/1999/xhtml>>\r\n<head><title>\r\ntitle\r\n</title></head></html>' lxml.etree.tostring(lxml.etree.fromstring(g, parser=lxml.etree.HTMLParser())) OUTPUT: '<html xmlns="http://www.w3.org/1999/xhtml"<http://www.w3.org/1999/xhtml> xmlns="http://www.w3.org/1999/xhtml"<http://www.w3.org/1999/xhtml>><head><meta http-equiv="Content-Type" content="text/html; charset=ASCII" /><title>\ntitle\n</title></head></html>' Thanks in advance Matteo Parrucci

3 2

[lxml-dev] Schema validation
by Krzysztof Jakubczyk 18 Nov '10

18 Nov '10

Hi, I'm trying to validate a document using XmlSchema. It works but the exception received (etree.XMLSyntaxError) has no information about location - exc.position is (0,0). Is this correct behaviour?

1 0

[lxml-dev] (no subject)
by jholg＠gmx.de 17 Nov '10

17 Nov '10

Hi, just stumbled upon this: http://stackoverflow.com/questions/3103661 In short: Should we consider this a bug: >>> root = etree.fromstring("""<ns2:foobarResponse xmlns:ns2="http://api.example.com"> ... <duration>206</duration> ... <artist> ... <tracks>...</tracks> ... </artist> ... </ns2:foobarResponse> ... """) >>> root['{}duration'] Traceback (most recent call last): File "<stdin>", line 1, in ? File "lxml.objectify.pyx", line 284, in lxml.objectify.ObjectifiedElement.__getitem__ (src/lxml/lxml.objectify.c:3345) File "lxml.objectify.pyx", line 484, in lxml.objectify._lookupChildOrRaise (src/lxml/lxml.objectify.c:5347) AttributeError: no such child: {http://api.example.com}duration >>> ? Looks like there is no way to get at a no-namespace child element apart from working around this e.g. using xpath. Holger -- GMX DSL Doppel-Flat ab 19,99 €/mtl.! Jetzt auch mit gratis Notebook-Flat! http://portal.gmx.net/de/go/dsl

3 3

[lxml-dev] Asking for advice - python lxml (fwd)
by John W. Shipman 16 Nov '10

16 Nov '10

Sorry, I don't do sysadmin. Forwarding to the mailing list. John Shipman (john(a)nmt.edu), Applications Specialist, NM Tech Computer Center, Speare 146, Socorro, NM 87801, (575) 835-5950, http://www.nmt.edu/~john ``Let's go outside and commiserate with nature.'' --Dave Farber ---------- Forwarded message ---------- Date: Sat, 13 Nov 2010 13:37:47 +0000 From: Peter Lom <peterl(a)openet.com> To: "tcc-doc(a)nmt.edu" <tcc-doc(a)nmt.edu> Subject: Asking for advice - python lxml Hi John and others in lxml world, I wonder if you can advise about the problem in installing lxml on Solaris? This is my desperate move to save the first attempt in an international company located in Ireland to use python and also lxml for a Canadian client. I developed an application around lxml on the dev box (out sysadmin was able to build python 2.6.2 with lxml on Solaris 10 with some obstacles from sources but all is OK) . BTW it did performs about 15x faster than xml processing using Ruby The client production system does not allow access to repositories and as the consequence our sysadmin cannot prepare a full build for them! The man claimed 10 hours lost in trying to do so and failed. What can be done? Many thanks Peter Lom, Melbourne This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient, please note that any review, dissemination, disclosure, alteration, printing, circulation, retention or transmission of this e-mail and/or any file or attachment transmitted with it, is prohibited and may be unlawful. If you have received this e-mail or any file or attachment transmitted with it in error please notify postmaster(a)openet.com. Although Openet has taken reasonable precautions to ensure no viruses are present in this email, we cannot accept responsibility for any loss or damage arising from the use of this email or attachments.

2 1

[lxml-dev] Correcting 'simple' broken HTML
by Wray, Paul 10 Nov '10

10 Nov '10

Background: I need a paragraph-insertion algorithm for use with Internet Explorer's webbrowser component, as the use of execCommand('InsertParagraph') or pasteHTML gives unpredictable results. I have developed a simple algorithm (~50 lines of Python) that gives acceptable results for simplest cases, but I have no confidence that it covers all cases. I thought that I could use lxml to correct the html to when a paragraph is inserted in the middle of text (pipe character represents caret position): Original: xxx|yyy Insert para: xxxyyy Fixed: xxxyyy This simplest case works OK, but I was surprised to find that this fails, when breaking a line within an inline element: Original: wwwxxx|yyyzzz Insert para: wwwxxxyyyzzz Expected output: wwwxxxyyyzzz Actual output from lxml.etree and lxml.html: wwwxxxyyyzzz So it seems that both lxml.etree and lxml.html are tolerant of a paragraph as the child of an inline element. When I use recover=False for lxml.etree parser, there is no exception raised. My questions: * Am I expecting too much, or missing something? I think that the above is a simple case of broken HTML. * Can anyone point me to a tried and true line-breaking algorithm for lxml? Code and output follows. ------------------------------------------------------------- Test Code from lxml import etree, html from StringIO import StringIO print 'lxml version', etree.LXML_VERSION print 'libxml version', etree.LIBXML_VERSION badhtml = 'wwwxxxyyyzzz' print 'With lxml.etree:' parser = etree.HTMLParser(recover=False) tree = etree.parse(StringIO(badhtml), parser) result = etree.tostring(tree.getroot(), pretty_print=True, method='html') print result print 'With lxml.html:' parsed = html.fragment_fromstring(badhtml) print html.tostring(parsed) --------------------------------------------------------- Output lxml version (2, 2, 2, 0) libxml version (2, 7, 2) With lxml.etree: <html><body>wwwxxxyyyzzz</body></html> With lxml.html: wwwxxxyyyzzz Paul ********************************************************************** This message is intended for the addressee named and may contain privileged information or confidential information or both. If you are not the intended recipient please delete it and notify the sender. **********************************************************************

2 2

[lxml-dev] etree.tostring() cannot handle Unicode
by Ravi 06 Nov '10

06 Nov '10

With reference to the bug report https://bugs.launchpad.net/lxml/+bug/671885I found that etree.tostring() cannot handle Unicode. It is giving me the UnicodeEncodeError. Thanks.

2 1

Re: [lxml-dev] Need feedback on Memory Errors
by Graff, Marc 05 Nov '10

05 Nov '10

The runtime vs. compile time lib difference went unrealized (missed the 500 lb. gorilla) in until my ride home last night even though it was right in front of me. The long ride home is often when things that allude me often come together. I was concerned I opened my self up to justifiably harsh scrutiny. Thanks for kindly confirming. Also thanks for the helpful insights. I generally run top to see what my code is using but will include prstat to my monitoring. I will recompile with the correct env vars and retest. I was considering the possible memory footprint of the current implementation but wanted to finish version 1. I will try altering with cElementTree and compare to the current code. I am also going to investigate an event driven parse_and_append approach since lmxl provide such a mechanism and I believe that could reduce memory usage drastically. Thanks for the very useful feedback and have a good weekend. Marc -----Original Message----- From: lxml-dev-bounces(a)codespeak.net [mailto:lxml-dev-bounces@codespeak.net] On Behalf Of Stefan Behnel Sent: Thursday, November 04, 2010 6:01 PM To: Graff, Marc Cc: lxml-dev(a)codespeak.net Subject: Re: [lxml-dev] Need feedback on Memory Errors Hi, Graff, Marc, 04.11.2010 19:41: > I just finished an app that parses a large xml file "FeedA" and appends > another smaller file fragmentB to the tree from FeedA for an xpath > specified parent node. All seems fine when processing a file less than > 500MB but anything large results in one of two errors. You may not be aware of it, but this is huge. If that's just the size of the serialised XML, this means that the in-memory tree representation is several times that size, easily 10x or more. Depending on the text-to-tag ratio in the content, it may well reach the size of your available memory. Check the size of the Python process while it's building the tree, prstat is your friend. > All libs were built from src in my home dir and LD_LIBRARY_PATH reflects > the home dir lib. Not sure if that will distort the following lib > details > > lxml.etree: (2, 2, 8, 0) > > libxml used: (2, 7, 7) > > libxml compiled: (2, 6, 23) > > libxslt used: (1, 1, 26) > > libxslt compiled: (1, 1, 15) Try to build against the libraries that you use at runtime. lxml has several bug work-arounds and compile time adaptations for the various library versions. A major discrepancy between the version used at compile time and runtime, such as in your case, may have unexpected side effects. You can pass the path to the configuration scripts (xml2-config and xslt-config in the bin directories of the install destinations) using the XML2_CONFIG and XSLT_CONFIG environment variables. > There should be ample memory. This is running on a Solaris M5000 with > 96GB of memory and unlimit is unlimited. The FeedA test file contains > valid xml and is the same test file for 512MB, 768MB, 1.5GB and 3GB file > tests. > > Just over 500MB and the app returns a MemoryError on the serializer.pxi. The serialiser needs to reallocate additional memory step by step while it's doing its work. Normally, the OS handles this by enlarging the allocated area and without copying. However, if the available memory runs low, memory fragmentation may trigger the allocation of a completely new memory area of very large size to copy the previously allocated memory into, which may easily fail since memory is low already. So even if there is some memory left in the system, it may not be enough to satisfy the memory allocation scheme at hand. Remember that your output alone is 500-3000 MB in one single piece of memory, and libxml2 can't know in advance that it will need that much. So, please monitor the memory consumption of the process. If you are really running out of memory, one thing you can try is to switch to cElementTree (xml.etree.cElementTree). It has a somewhat lighter memory footprint which may just be enough to make a difference here (although likely not for 3GB of XML). It also has less features than lxml.etree (a bit fewer less in Py2.7/ET1.3), but currently, your only real problem seems to be the memory requirement. Stefan _______________________________________________ lxml-dev mailing list lxml-dev(a)codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev

1 0

[lxml-dev] Need feedback on Memory Errors
by Graff, Marc 04 Nov '10

04 Nov '10

I just finished an app that parses a large xml file "FeedA" and appends another smaller file fragmentB to the tree from FeedA for an xpath specified parent node. All seems fine when processing a file less than 500MB but anything large results in one of two errors. All libs were built from src in my home dir and LD_LIBRARY_PATH reflects the home dir lib. Not sure if that will distort the following lib details lxml.etree: (2, 2, 8, 0) libxml used: (2, 7, 7) libxml compiled: (2, 6, 23) libxslt used: (1, 1, 26) libxslt compiled: (1, 1, 15) There should be ample memory. This is running on a Solaris M5000 with 96GB of memory and unlimit is unlimited. The FeedA test file contains valid xml and is the same test file for 512MB, 768MB, 1.5GB and 3GB file tests. Just over 500MB and the app returns a MemoryError on the serializer.pxi. Attached is the full error trace from the captured exception. The parse obj has (huge_tree=True). I didn't expect this to make a difference since the error in on the serialization of the output but tried anyway. File "serializer.pxi", line 133, in lxml.etree._tostring (src/lxml/lxml.etree.c:79345) MemoryError Anything over 1.5GB and it core dumps (first error in the stack trapped in libxml2). Attached is the stack and mappings from mdb. Including incase related: libc.so.1`strlen+0x50(39e730, 3ceef0, 1e3718, 755, 1e3724, 756) etree.so`__pyx_f_4lxml_5etree_13_BaseErrorLog__receive+0xcc(397490, 3ceef0, feff5d24, 74e, 2, 74e) etree.so`__pyx_f_4lxml_5etree__forwardError+0x6c(fef081e0, 3ceef0, d9d64, fed303a0, ff1303bc, 1) libxml2.so.2.7.7`__xmlRaiseError+0x2c4(fef07980, fee9c4c8, 397490, 3ced70, ffffffff, 1) libxml2.so.2.7.7`xmlErrMemory+0xa4(3ced70, fedc1e70, d9d64, ff0566a8, ff1303bc, ff13a558) libxml2.so.2.7.7`xmlSAX2TextNode+0x2e0(0, 3f5616, 7, 1, 6, feddb774) I am new to python so any help/suggestions would be greatly appreciated. Thanks Marc

2 1

[lxml-dev] XSLT - xsltMaxDepth setting
by Jérôme Carretero 04 Nov '10

04 Nov '10

Hi, libxslt uses a xsltMaxDepth variable (global...) to limit recursion, and I used to increase it when processing some big files (for instance, DocBook containing tables spanning over dozens of pages). xsltproc --maxdepth 10000 .... At the moment, lxml does not touch this value. Maybe providing a lxml.etree.XSLT.maxdepth property would not be too complicated ? Regards, -- cJ

2 3