Mailman 3 January 2008 - lxml - The Python XML Toolkit

[lxml-dev] Possible bug in xpath?
by Bruno Barberi Gnecco 24 Jan '08

24 Jan '08

Hi, I think I may have run into a bug. I'm attaching a sample code to reproduce it. Instead of getting back just the '<strong>..</strong>', I get the entire div content. Since I can't tell if this bug is in lxml or one of the libs it uses, I'm posting to this list instead of the bug tracker. My setup: Python 2.5.1, lxml 1.3.6 and libxml2.so.2.6.28. If OTOH I'm doing something stupid, just tell me :) Thanks a lot, -- Bruno Barberi Gnecco <brunobg_at_users.sourceforge.net> ...the flaw that makes perfection perfect.

2 1

[lxml-dev] Restrict exception error log to local errors?
by Stefan Behnel 24 Jan '08

24 Jan '08

Hi, currently, the "error_log" property on exceptions raised by lxml represents a frozen snapshot of the global error log, thus aggregating loads of errors that have recently occurred, but that are not necessarily related to the problem that lead to the exception. However, we could often restrict that to an error log snapshot that is local to the operation that raised the exception. Question: should this be changed? Is there a reason this should not be changed for 2.0? Stefan

2 2

[lxml-dev] type of custom objects in XML-tree disappears
by Markus Hillebrand 23 Jan '08

23 Jan '08

Thanks for your fast response - with your help I was able to make my implementation work, but on the design side I'm not quite satisfied with the solution. You wrote "You do not have to call set_element_class_lookup() each time as it sticks with the parser." ... - but thats exactly what I want to do: I need a tree with objects of different(!) classes. And I want to put additional data in that objects. Yes, I read this one: "There is one thing to know up front. Element classes must not have a constructor, neither must there be any internal state (except for the data stored in the underlying XML tree). Element instances are created and garbage collected at need, so there is no way to predict when and how often a constructor would be called." on http://codespeak.net/lxml/element_classes.html So for me, it seems that lxml seems not to be designed to manage objects of different classes in an XML tree. With a hack (calling setElementClassLookup() before each creation of an element) I'm able to create such tree's and for some testcases it seems to work fine - despite of these nasty things I reported. But it's not quite satisficing: - it's not free of side-effects, when I change the default setElementClassLookup ... - I'm afraid to run in garbage collection bugs, like the one I reported. Finally my question: is it possible, that lxml supports that feature officially? For example by providing an explicit factory call like etree.createElement(class)? With best regards Markus

2 1

Re: [lxml-dev] lxml \ libxslt \ libxml2 leads to apache 2 crash : looks like fixed
by Dmitri Fedoruk 21 Jan '08

21 Jan '08

Hello everybody, I've been testing lxml2.0beta1 for a week already and it looks like the problem leading to apache2 crash disappeared. I've tried to reproduce the bug using the known situations which lead to crash, this did not happen. Our QA team made some more thorough tests, the crash did not happen. So I hope the bug is gone forever :) I guess I should thank Stefan for fixing whatever it was. Cheers, Dmitri

2 1

[lxml-dev] Problem installing lxml-2.0beta1 onto production server
by Brad Clements 21 Jan '08

21 Jan '08

I am installing lxml-2.0beta1 on python 2.3 on RHEL4. I brought my libxml2 and libxslt up to date: libxml2-2.6.30-1 libxslt-1.1.22-1 then easy-install -U lxml During the install process, I got this: easy_install-2.3 -U lxml Searching for lxml Reading http://pypi.python.org/simple/lxml/ Reading http://codespeak.net/lxml Best match: lxml 2.0beta1 Downloading http://cheeseshop.python.org/packages/source/l/lxml/lxml-2.0beta1.tar.gz Processing lxml-2.0beta1.tar.gz Running lxml-2.0beta1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SE9_WX/lxml-2.0beta1/egg-dist-tmp-onxV0q Building lxml version 2.0.beta1. NOTE: Trying to build without Cython, pre-generated 'src/lxml/etree.c' needs to be available. warning: no previously-included files found matching 'doc/pyrex.txt' warning: no previously-included files found matching 'src/lxml/etree.pxi' File "build/bdist.linux-i686/egg/lxml/html/diff.py", line 323 start.extend(tag for name, pos, tag in tag_stack) ^ SyntaxError: invalid syntax File "/usr/local/lib/python2.3/site-packages/lxml-2.0beta1-py2.3-linux-i686.egg/lxml/html/diff.py", line 323 start.extend(tag for name, pos, tag in tag_stack) ^ SyntaxError: invalid syntax Adding lxml 2.0beta1 to easy-install.pth file Installed /usr/local/lib/python2.3/site-packages/lxml-2.0beta1-py2.3-linux-i686.egg Processing dependencies for lxml Finished processing dependencies for lxml And when I try to run an application that uses lxml, I get this: @400000004793fafb29d86124 Traceback (most recent call last): @400000004793fafb29e1c764 File "/misc/home/bkc/src/Python/MurkWorks/Rating/RatingServer.py", line 13, in ? @400000004793fafb29e2b994 from MurkWorks.Carriers.Rater import Rater, get_carrier_service_mapper @400000004793fafb29ee36fc File "/misc/home/bkc/src/Python/MurkWorks/Carriers/Rater.py", line 22, in ? @400000004793fafb29eee2dc import UPS, BAX, CCX, Estes, FedexFreight, FedexExpress, FedexGround, USPS, Yellow, Aaction, LandAir, FedexNational, CentralTransport @400000004793fafb29f3d094 File "/misc/home/bkc/src/Python/MurkWorks/Carriers/CentralTransport.py", line 14, in ? @400000004793fafb29f4a76c from lxml import etree @400000004793fafb29f55734 ImportError: /usr/local/lib/python2.3/site-packages/lxml-2.0beta1-py2.3-linux-i686.egg/lxml/etree.so: undefined symbol: PyDict_Contains On my development server I'm running alpha3, so I'm trying to to install that now: http://cheeseshop.python.org/packages/source/l/lxml/lxml-2.0alpha3.tar.gz But that fails:: NOTE: Trying to build without Cython, pre-generated 'src/lxml/etree.c' needs to be available. warning: no previously-included files found matching 'doc/pyrex.txt' warning: no previously-included files found matching 'src/lxml/etree.pxi' src/lxml/etree.c:73843: error: `METH_COEXIST' undeclared here (not in a function) src/lxml/etree.c:73843: error: initializer element is not constant src/lxml/etree.c:73843: error: (near initialization for `__pyx_methods_5etree_QName[0].ml_flags') src/lxml/etree.c:73843: error: initializer element is not constant src/lxml/etree.c:73843: error: (near initialization for `__pyx_methods_5etree_QName[0]') same error with alpha4 Oops, and alpha5 is: Building lxml version 2.0.alpha5. NOTE: Trying to build without Cython, pre-generated 'src/lxml/etree.c' needs to be available. warning: no previously-included files found matching 'doc/pyrex.txt' warning: no previously-included files found matching 'src/lxml/etree.pxi' File "build/bdist.linux-i686/egg/lxml/html/diff.py", line 323 start.extend(tag for name, pos, tag in tag_stack) ^ SyntaxError: invalid syntax File "/usr/local/lib/python2.3/site-packages/lxml-2.0alpha5-py2.3-linux-i686.egg/lxml/html/diff.py", line 323 start.extend(tag for name, pos, tag in tag_stack) ^ SyntaxError: invalid syntax Adding lxml 2.0alpha5 to easy-install.pth file Installed /usr/local/lib/python2.3/site-packages/lxml-2.0alpha5-py2.3-linux-i686.egg Processing dependencies for lxml==2.0alpha5 Finished processing dependencies for lxml==2.0alpha5 It looks like it's going to be a late night for me re-writing to not use lxml on the production server. -- Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com AOL-IM: BKClements

3 2

Re: [lxml-dev] type of custom objects in XML-tree disappears
by Stefan Behnel 20 Jan '08

20 Jan '08

Hi, I'm responding here as I don't think this is a bug. It should be discussed on the list. mh wrote: > I want to use lxml to build up an XML tree with custom > element objects. > > Background: I want to build up an XML-compatible syntax tree > and provide additional methods and some non XML relevant > attributes in the tree. > > lxml provides etree.XMLParser.setElementClassLookup(...) > and etree.XMLParser.makeelement(...) methods to build up > such a tree. > > It works fine to that point, but after some operations on the > tree, the inserted elements are still there, but some of them > have changed their type from my custom classes to etree.Element > and my additional methods and attributes are lost. > > I tried repeat that kind of error, but it's not easy to make > it repeatable, so I wrote a brute force script to provocate that > kind of error (look at the end of this message). I found this error > on Mac OS X, Windows and Linux and on lxml 1.3.6 and lxml > 2beta1 with Python 2.5. > > Maybe it's not exactly an explicit use case lxml was built for, but > maybe it's worth thinking about that one. > > ----- SNIP ----- > from lxml import etree > import random > import sys > > class MyElement(etree.ElementBase): > TAG="MyElement" > > class Generator: > def __init__(self): > self.__oFactory=etree.XMLParser() > self.__oTree=etree.ElementTree(etree.Element("root"), None) Note that the root element is not taking your element class setup into account here, so it's using the standard class. > def CreateElement(self, oClass): > self.__oFactory.setElementClassLookup(etree.ElementDefaultClassLookup(oClass)) > oNew=self.__oFactory.makeelement(oClass.TAG) > return oNew You do not have to call set_element_class_lookup() each time as it sticks with the parser. I'd rather write __init__ like this: parser = etree.XMLParser() parser.set_element_class_lookup(etree.ElementDefaultClassLookup(the_class)) self.__makeelement = parser.makeelement self.__tree = etree.ElementTree(self.__makeelement("root")) > def Run(self): > try: > for i in range(0,200): > self.Visit(self.__oTree.getroot()) > except Exception, oError: > etree.dump(self.__oTree.getroot()) > > def Visit(self, oElement): > if oElement.tag!="root": > if not isinstance(oElement, MyElement): > raise Exception("Failed") > nRandom=random.randint(0,2) > if nRandom==0: > oNew=self.CreateElement(MyElement) > oElement.append(oNew) > elif nRandom==1: > oNew=self.CreateElement(MyElement) > oElement.insert(0, oNew) > for oSub in oElement: > self.Visit(oSub) > > oGen=Generator() > oGen.Run() This seems to work just fine for me (which actually surprises me). The problem is that your root element does not know about the factory you are using for your other elements, but when Element proxy(!) objects are created for a tree, they are based on the factory that is associated with the document that holds the root element. So when element proxy objects were garbage collected and then are recreated on demand, they will use the standard factory, not yours. The way to fix your code is to do something like the code I posted above. Stefan

1 0

[lxml-dev] Installing lxml 2.0beta1 via easy_install requires Cython; also, question about lxml.html.clean.clean_html
by Jon Rosebaugh 12 Jan '08

12 Jan '08

I attempted to install lxml 2.0beta1 via easy_install (easy_install lxml==2.0beta1), and it didn't work. After a bunch of experimentation, I discovered that the C files that are supposed to be present in the download were not present. After installing a patched version of Cython 0.9.6.10b (patched according to the directions I found on this list) lxml successfully installed. But I was very surprised at this requirement. Also, I'm not sure, but I think the lxml.html.clean.clean_html() function might not be working properly? I followed the example at http://codespeak.net/lxml/dev/lxmlhtml.html#cleaning-up-html but got different results. I expected this: <html> <body> <div> <style>/* deleted */</style> <a href="">a link</a> <a href="#">another link</a> <p>a paragraph</p> <div>secret EVIL!</div> of EVIL! Password: annoying EVIL! <a href="evil-site">spam spam SPAM!</a> <img src="evil!"> </div> </body> </html> But got this: <div><style>/* deleted */</style><body> <a href="">a link</a> <a href="#">another link</a> <p>a paragraph</p> <div>secret EVIL!</div> of EVIL! Password: annoying EVIL!<a href="evil-site">spam spam SPAM!</a> <img src="evil!"></body></div>

2 3

[lxml-dev] lxml \ libxslt \ libxml2 leads to apache 2 crash on freebsd/amd64
by Dmitri Fedoruk 09 Jan '08

09 Jan '08

Greetings! First of all, happy holidays. Looks like it's not the perfect time to report a problem, but I can't find any solution by myself. More than that, I'm not sure this is a lxml problem. But I really hope for somebody to take a look at this long message :) My application is a web service written in Python. It is running inside of mod_python handled by Apache2 (precise versions follow). Apache runs in prefork mode. I use xml to collect data from sources. I use xslt for their processing and, eventually, forming the html output. I use lxml library for this. Xslt transformations are compiled once for every instance of apache fork and then they run in their own thread independently. Up to one moment everything went smoothly and painlessly. Then I decided to upgrade all the software and came to the following situation. When I try to serialize an etree object ... transform = etree.XSLT(xslt_doc) result_tree = transform(data, **variables ) return etree.tostring(result_tree, 'utf-8') ( or return str(result_tree) ) I have a core dump of apache. The point is that it happens during serialization. I tried to omit 'utf-8', I tried to use str() - did not help. What is strange - this error occurs with only several xsl templates, and not all the time. What is even more strange: this happens only on amd64 machines _within_ apache. On my i386 desktop I can not reproduce this bug. Also I can not reproduce this from the console. Now the configuration details. I tried almost all possible combinations of this software. FreeBSD 6.2-20070330-SNAP Apache/2.0.61, Apache/2.2.6 mod_python-3.3.1 lxml 1.3.5, 1.3.6 libxml2-2.6.27, libxml2-2.6.30 libxslt-1.1.20, libxslt-1.1.22 I have to mention that I do also use some modules written in C that have pythonic binding. We suggested that they could be the reason of memory corruption. We run the application without their usage and still got the same core dump, so they are not the reason. The error messages are httpd in free(): error: chunk is already free httpd in free(): error: modified (chunk-) pointer httpd in free(): error: pointer to wrong page And the backtrace: (gdb) bt #0 0x00000008013824bc in kill () from /lib/libc.so.6 #1 0x000000080138134d in abort () from /lib/libc.so.6 #2 0x000000080131a265 in _UTF8_init () from /lib/libc.so.6 #3 0x000000080131a29c in _UTF8_init () from /lib/libc.so.6 #4 0x000000080131b23d in _UTF8_init () from /lib/libc.so.6 #5 0x00000008055dc3f9 in xmlFreeNodeList () from /usr/local/lib/libxml2.so.5 #6 0x00000008055dc29d in xmlFreeProp () from /usr/local/lib/libxml2.so.5 #7 0x00000008055dc2dc in xmlFreePropList () from /usr/local/lib/libxml2.so.5 #8 0x00000008055dc4bb in xmlFreeNodeList () from /usr/local/lib/libxml2.so.5 #9 0x00000008055dc385 in xmlFreeNodeList () from /usr/local/lib/libxml2.so.5 #10 0x00000008055dc385 in xmlFreeNodeList () from /usr/local/lib/libxml2.so.5 #11 0x00000008055dc385 in xmlFreeNodeList () from /usr/local/lib/libxml2.so.5 #12 0x00000008055dc385 in xmlFreeNodeList () from /usr/local/lib/libxml2.so.5 #13 0x00000008055dc385 in xmlFreeNodeList () from /usr/local/lib/libxml2.so.5 #14 0x00000008055dcad5 in xmlFreeDoc () from /usr/local/lib/libxml2.so.5 #15 0x00000008051a8399 in __pyx_tp_dealloc_5etree__Document () from /usr/local/lib/python2.5/site-packages/lxml-1.3.6-py2.5-freebsd-6.2-20070912-SNAP-amd64.egg/lxml/etree.so #16 0x00000008051be15b in __pyx_tp_dealloc_5etree__Element () from /usr/local/lib/python2.5/site-packages/lxml-1.3.6-py2.5-freebsd-6.2-20070912-SNAP-amd64.egg/lxml/etree.so #17 0x00000008051a9b8f in __pyx_tp_dealloc_5etree__ElementTree () from /usr/local/lib/python2.5/site-packages/lxml-1.3.6-py2.5-freebsd-6.2-20070912-SNAP-amd64.egg/lxml/etree.so #18 0x00000008036cbb4b in _PyFloat_Unpack8 () from /usr/local/lib/libpython2.5.so #19 0x0000000803723c03 in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.5.so #20 0x00000008037229eb in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #21 0x0000000803723c24 in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.5.so #22 0x00000008037229eb in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #23 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #24 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #25 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #26 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #27 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #28 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #29 0x0000000803723c24 in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.5.so #30 0x00000008036cd8ae in PyFunction_SetClosure () from /usr/local/lib/libpython2.5.so #31 0x00000008036b3c73 in PyObject_Call () from /usr/local/lib/libpython2.5.so #32 0x0000000803721262 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #33 0x0000000803723c24 in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.5.so #34 0x00000008037229eb in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #35 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #36 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #37 0x0000000803723326 in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #38 0x0000000803723c24 in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.5.so #39 0x00000008037229eb in PyEval_EvalFrameEx () from /usr/local/lib/libpython2.5.so #40 0x0000000803723c24 in PyEval_EvalCodeEx () from /usr/local/lib/libpython2.5.so #41 0x00000008036cd8ae in PyFunction_SetClosure () from /usr/local/lib/libpython2.5.so #42 0x00000008036b3c73 in PyObject_Call () from /usr/local/lib/libpython2.5.so #43 0x00000008036bbd64 in PyMethod_New () from /usr/local/lib/libpython2.5.so #44 0x00000008036b3c73 in PyObject_Call () from /usr/local/lib/libpython2.5.so #45 0x00000008036b3d09 in PyObject_Call () from /usr/local/lib/libpython2.5.so #46 0x00000008036b4024 in PyObject_CallMethod () from /usr/local/lib/libpython2.5.so #47 0x000000080356acf2 in python_handler () from /usr/local/libexec/apache2/mod_python.so #48 0x0000000000425e4a in ap_run_handler (r=0x190b090) at config.c:152 #49 0x0000000000426755 in ap_invoke_handler (r=0x190b090) at config.c:364 #50 0x0000000000422550 in ap_process_request (r=0x190b090) at http_request.c:249 #51 0x000000000041bbd7 in ap_process_http_connection (c=0x77f1b0) at http_core.c:253 #52 0x0000000000433c1a in ap_run_process_connection (c=0x77f1b0) at connection.c:43 #53 0x0000000000434075 in ap_process_connection (c=0x77f1b0, csd=0x77f090) at connection.c:176 #54 0x000000000042437f in child_main (child_num_arg=3) at prefork.c:610 #55 0x000000000042450b in make_child (s=0x5a94f8, slot=3) at prefork.c:704 #56 0x00000000004247b0 in perform_idle_server_maintenance (p=0x578028) at prefork.c:839 #57 0x0000000000424c37 in ap_mpm_run (_pconf=0x578028, plog=0x5a4028, s=0x5a94f8) at prefork.c:1040 #58 0x000000000042d27e in main (argc=1, argv=0x7fffffffea08) at main.c:656 The irony of it all is that I still can not downgrade all this packages to the version that did not have core dump... I've spent a week already trying to figure out the gist of the problem, combining different lxml, libxml2\xslt and apache versions. I start with this mailing list as the fatal call is made with the lxml function. I really hope there is an explanation and solution for this. I do not want to switch to another xslt processor or rebuild the architecture of the whole service in the worst case. I'd be grateful for any ideas and suggestions. Cheers, Dmitri

3 12

Re: [lxml-dev] cssselect and cssutils
by Stefan Behnel 07 Jan '08

07 Jan '08

Hi, Höke, Christof wrote: >> Von: Ian Bicking [mailto:ianb@colorstudy.com] >> ::first-letter is hard >> because it doesn't match any object in lxml. If it returned a string >> like "A" it would be very much out of context (e.g., no parent pointer), >> and it would be hard to do anything useful with it. To make it useful I >> think it would require some new stringish object that also looked nodeish >> (e.g., had a .getparent() method). I considered that a while ago, as it would also be interesting for XPath in general. However, currently, we use fast Python string creation functions to serve the API level. At the time I deducted that changing that to the instantiation of a custom string object would almost certainly slow things down and complicate them, just to serve a rather special use case. Although maybe I might want to take another look at that today... > What came to my mind was the DOM range spec stuff, but it is not really > finished, is it? I was reading about it some years (!) ago I think in the > Javascript Definitive Guide but I think it never went anywhere really. I had a discussion about that starting over at the XML-SIG list last summer. http://permalink.gmane.org/gmane.comp.python.lxml.devel/2763 That was the first time I heard about DOM ranges and when I dug into that a little deeper, I almost ran away screaming. IMVHO, that's an insane and horribly complicated spec. > :first-letter should actually be element.text[0] I guess (which would be a > string in lxml currently?) Yes. > I don't really know the lxml API but would it > be possible to define a subtype for element.text for this case? But you are > right, a more general approach would certainly be better. You could define a special string (and unicode) subtype for the result of an XPath expression, which is determined independent of the Python API level. The freedom is right there. However, it would mean you have to search and (in the worst case) instantiate the parent Element to make sure it won't go away while the string result exists. That's some overhead compared to a simple string creation. As I said, I might reconsider that, but I'm not very confident. Stefan

1 0

[lxml-dev] cssselect and cssutils
by Stefan Behnel 07 Jan '08

07 Jan '08

Hi Christof, Höke, Christof wrote: > You are the main developer for lxml, right? Yep, but not the only one. :) > I was trying the CSSSelect > facility for a Python CSS library I am developing > (http://code.google.com/p/cssutils/) Cool. I knew about cssutils, felt that its field of application was related to cssselect (and lxml in general) but not with too much of an overlap - and always thought it would be nice to have it working with lxml in some way. > and I think there are some minor > problems with "*" or "*|*" (I need to check again and I'll put them on the > bug tracker then) but a question regarding support for pseudo selectors: > Would it be possible to support stuff like :first-letter (currently not > working is it not?) with Python XPath extension functions which should be > able to do what XPath cannot? Are you maybe even working on it? I guess > things like :first-line are problematic but other should be ok. I'm not the primary person to ask here. cssselect was developed by Ian Bicking, he knows best what works, what doesn't, and how to fix it. :) > If I get the time I would try some things out and report back, this was > just an idea that I had while playing with CSSSelector... Go ahead, this is open source. Any help, testing and ideas are always appreciated. > Lxml is really great stuff BTW, it was actually quite simple using lxml and > a CSSStyleSheet on a given HTML. (Not released yet but an example is in the > SVN). Great. In case there's anything we can do on lxml's side, please ask on the list. > Also the XPath extension facility is really great, I used Pyana until > some time ago but now use lxml for most projects. Competition is best when you win. :) Stefan

2 1