Mailman 3 January 2012 - lxml - The Python XML Toolkit

Difference in behaviour: bug or bad usage?
by Willmer, Alex (PTS) Jan. 24, 2012

Jan. 24, 2012

The following is a simplified example of the behaviour we're seeing the Sunburnt library, which uses lxml: Python 2.4.3 (#1, Sep 21 2011, 19:55:41) [GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import lxml.etree >>> from lxml.builder import E >>> >>> class Foo(object): ... QUERY = E.query ... ... def test(self, q): ... return self.QUERY(q) ... >>> foo = Foo() >>> bar = foo.test('a') Traceback (most recent call last): File "<stdin>", line 1, in ? File "<stdin>", line 5, in test File "/opt/define/lib/python2.4/site-packages/lxml-2.2.7_r0-py2.4-linux-x86_64.egg/lxml/builder.py", line 47, in <lambda> return lambda *args, **kwargs: func(tag, *args, **kwargs) File "/opt/define/lib/python2.4/site-packages/lxml-2.2.7_r0-py2.4-linux-x86_64.egg/lxml/builder.py", line 220, in __call__ raise TypeError("bad argument type: %r" % item) TypeError: bad argument type: <__main__.Foo object at 0x2b1a8a79ee10> Python 2.7, lxml 2.7.7 >>> import lxml.etree >>> from lxml.builder import E >>> >>> class Foo(object): ... QUERY = E.query ... ... def test(self, q): ... return self.QUERY(q) ... >>> foo = Foo() >>> bar = foo.test('a') >>> print lxml.etree.tostring(bar) <query>a</query> I'm looking for guidance on whether the exception in Python 2.4 is due to a bug in lxml, incorrect usage in binding E.query to a class attribute, or perhaps an error on our part when building lxml on Redhat. The Sunburnt code in which this occurred is SoldrDelete.delete_queries() https://github.com/tow/sunburnt/blob/master/sunburnt/schema.py#L617 With thanks, Alex Think green - keep it on the screen. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

2 1

lxml/etree/XSLT and dynamic stylesheet variables
by Adam Tauno Williams Jan. 21, 2012

Jan. 21, 2012

I'm using etree to perform XSLT transforms, such as - from lxml import etree source = etree.parse(self.rfile) xslt = etree.fromstring(self._xslt) transform = etree.XSLT(xslt) result = transform(source) according to the docs at <http://lxml.de/xpathxslt.html#stylesheet-parameters> I can pass a dictionary of parameters to transform, such as - result = transform(doc_root, **{'non-python-identifier': '5'}) Can I pass a dictionary-like object? That doesn't seem to be working. I need to perform dynamic lookup of variables for the stylesheet. I've subclassed dictionary and overloaded [], get, has_key, and in to perform the required lookups; these work in testing. But passing the object to transform doesn't work. If that isn't possible is there an alternate XSLT implementation for python that would allow calling back for variable values?

1 1

encoding question
by Tim Arnold Jan. 20, 2012

Jan. 20, 2012

I have to follow a specification for producing xhtml files. The original files are in cp1252 encoding and I must reencode them to utf-8. Also, I have to replace certain characters with html entities. I think I've got this right, but I'd like to hear if there's something I'm doing that is dangerous or wrong. Please see the appended code, and thanks for any comments. I have two functions, translate (replaces high characters with entities) and reencode (um, reencodes): --------------------------------- import codecs, StringIO from lxml import etree high_chars = { 0x2014:'—', # 'EM DASH', 0x2013:'–', # 'EN DASH', 0x0160:'&Scaron;',# 'LATIN CAPITAL LETTER S WITH CARON', 0x201d:'”', # 'RIGHT DOUBLE QUOTATION MARK', 0x201c:'“', # 'LEFT DOUBLE QUOTATION MARK', 0x2019:"’", # 'RIGHT SINGLE QUOTATION MARK', 0x2018:"‘", # 'LEFT SINGLE QUOTATION MARK', 0x2122:'™', # 'TRADE MARK SIGN', 0x00A9:'©', # 'COPYRIGHT SYMBOL', } def translate(string): s = '' for c in string: if ord(c) in high_chars: c = high_chars.get(ord(c)) s += c return s def reencode(filename, in_encoding='cp1252',out_encoding='utf-8'): with codecs.open(filename,encoding=in_encoding) as f: s = f.read() sio = StringIO.StringIO(translate(s)) parser = etree.HTMLParser(encoding=in_encoding) tree = etree.parse(sio, parser) result = etree.tostring(tree.getroot(), method='html', pretty_print=True, encoding=out_encoding) with open(filename,'wb') as f: f.write(result) if __name__ == '__main__': fname = 'mytest.htm' reencode(fname)

1 0

lxml cutting tails of strings
by Motiejus Jakštys Jan. 20, 2012

Jan. 20, 2012

Hello, lxml community, I get string tails cut off in certain cases when I parse XML data. How to reproduce: Clone sample xml and python executable from here: $ git clone git://github.com/Motiejus/lxml_report.git and start it: $ ./debug.py demo.xml Actual output: Got data len: 752 Got data len: 752 Got data len: 2169 Got data len: 375 Expected output: Got data len: 752 Got data len: 752 Got data len: 2544 Got data len: 752 Is this a bug? Python and (LXML) versions I have tried on two amd64 installations: Debian Stable: python: 2.6.6-3+squeeze6 and 3.1.3-12 lxml: 2.2.8-2 libxslt1.1: 1.1.26-6 libxml2: 2.7.8.dfsg-2+squeeze1 Debian Sid: python: 2.7.2-9 and 3.2.2-1 lxml: 2.3.2-1 libxslt1.1: 1.1.26-8 libxml2: 2.7.8.dfsg-6 This also happens with xml.etree.cElementTree, xml.etree.ElementTree and xml.sax. However, data sets differ for every backend, and it was easiest to create (a small) one for lxml. Thank you for help. Motiejus Jakštys

2 2

objectify and namespaces weirdnes
by Piotr Oh Jan. 20, 2012

Jan. 20, 2012

Hi I'm scratching the surface of the lxml library and found objectify. I have xml file like this: http://pastebin.com/3bE1rSpW and would like to objectify it. The point is I have no idea how to deal with namespaces there. what I'm doing ... >>> tree=objectify.parse("file.xml",parser) >>> root=tree.getroot() >>> root.studenci.tag .... AttributeError: no such child: {urn:polon:plikImp:1_5}studenci or any other child. Well, what I'm missing here? I am a bit confused about the name space syntax. How can I get to the different children in this objectified file? P.

2 1

Multi threaded lxml usage causes crash in etree.so (lxml.etree.c:160651)
by Cal Leeming [Simplicity Media Ltd] Jan. 18, 2012

Jan. 18, 2012

Hey, Spent about a month trying to track down the cause of a crash in one of our webapps. When using uWSGI+Django, if you import lxml (without even using it) it will cause the worker to crash after the response has been sent back. I have replicated the same problem in a fresh vbox install. Attached is the gdb trace, and below is some info about the env + crash. Any advice on this would be *very* much appreciated, and will do anything I can to assist. Cal Leeming Simplicity Media Ltd ---- Tue Jan 17 20:01:51 2012 - *** Starting uWSGI 1.0.2.1 (64bit) on [Tue Jan 17 20:01:51 2012] *** Tue Jan 17 20:01:51 2012 - compiled with version: 4.4.5 on 16 January 2012 19:31:32 Tue Jan 17 20:01:51 2012 - current working directory: /home/duke Tue Jan 17 20:01:51 2012 - writing pidfile to /home/chroot/home/duke_local/webapps/ddcms/pid/uwsgi.pid Tue Jan 17 20:01:51 2012 - detected binary path: /home/duke/bin/uwsgi Tue Jan 17 20:01:51 2012 - limiting address space of processes... Tue Jan 17 20:01:51 2012 - your process address space limit is 536870912 bytes (512 MB) Tue Jan 17 20:01:51 2012 - your memory page size is 4096 bytes Tue Jan 17 20:01:51 2012 - allocated 7808 bytes (7 KB) for 8 cores per worker. Tue Jan 17 20:01:51 2012 - uwsgi socket 0 bound to UNIX address /home/chroot/home/duke_local/webapps/ddcms/sock/uwsgi.sock fd 3 Tue Jan 17 20:01:51 2012 - Python main interpreter initialized at 0x7c5e90 Tue Jan 17 20:01:51 2012 - threads support enabled Tue Jan 17 20:01:51 2012 - your server socket listen backlog is limited to 1024 connections Tue Jan 17 20:01:51 2012 - *** Operational MODE: threaded *** Tue Jan 17 20:01:51 2012 - added /home/chroot/home/duke_local/webapps/ddcms/webapp/ to pythonpath. Tue Jan 17 20:01:51 2012 - WSGI application 0 (mountpoint='/') ready on interpreter 0x7c5e90 pid: 3527 (default app) Tue Jan 17 20:01:51 2012 - spawned uWSGI master process (pid: 3527) Tue Jan 17 20:01:51 2012 - spawned uWSGI worker 1 (pid: 3528, cores: 8) Tue Jan 17 20:01:54 2012 - DAMN ! worker 1 (pid: 3528) died :( trying respawn ... Tue Jan 17 20:01:54 2012 - Respawned uWSGI worker 1 (new pid: 3541) Tue Jan 17 20:01:56 2012 - DAMN ! worker 1 (pid: 3541) died :( trying respawn ... Tue Jan 17 20:01:56 2012 - Respawned uWSGI worker 1 (new pid: 3553) [ 3276.149102] uwsgi[3531]: segfault at 27b00 ip 00007fefd0d802d4 sp 00007fefd952d6f0 error 4 in etree.so[7fefd0d10000+109000] [ 3278.533480] uwsgi[3542]: segfault at 27b00 ip 00007fefd0d802d4 sp 00007fefda52f6f0 error 4 in etree.so[7fefd0d10000+109000] Python version: 2.6.6 (r266:84292, Dec 26 2010, 22:48:11) [GCC 4.4.5] lxml.etree: (2, 3, 3, 0) libxml used: (2, 7, 8) libxml compiled: (2, 7, 8) libxslt used: (1, 1, 26) libxslt compiled: (1, 1, 26) Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f912f2df700 (LWP 3432)] 0x00007f9126b7e2d4 in initetree () at src/lxml/lxml.etree.c:160651 160651 src/lxml/lxml.etree.c: No such file or directory. in src/lxml/lxml.etree.c (gdb) bt full #0 0x00007f9126b7e2d4 in initetree () at src/lxml/lxml.etree.c:160651 __pyx_t_1 = 0x0 __pyx_t_5 = <value optimized out> __pyx_t_6 = 0x7f91335ec848 __pyx_t_7 = 0x0 __pyx_t_8 = 0x7f9126b75fe0 __pyx_t_9 = 0x7f91328baa60 __pyx_t_11 = <value optimized out> __pyx_t_12 = <value optimized out> __pyx_t_13 = <value optimized out> __pyx_t_14 = <value optimized out> #0 0x00007f9126b7e2d4 in initetree () at src/lxml/lxml.etree.c:160651 __pyx_t_7 = ((PyObject *)PyUnicode_DecodeASCII(xmlParserVersion, strlen(xmlParserVersion), NULL)); if (unlikely(!__pyx_t_7)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 232; __pyx_clineno = __LINE__; goto __pyx_L34_error;}

2 2

Namespace inconsistency with attributes
by Bartek Urbaniec Jan. 18, 2012

Jan. 18, 2012

Hello! I have a problem with namespaces for attributes. lxml takes always first value (which in my case is default) from list of namespaces and uses it for attributes. My xml file: <Root xmlns="urn:schemas-microsoft- com:office:spreadsheet" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet"> <Child ss:Color = "Blue"/> </Root> My program which adds one child element: from lxml import etree from StringIO import StringIO f = StringIO('<Root xmlns="urn:schemas-microsoft-com:office:spreadsheet" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet"><Child ss:Color = "Blue"/></Root>') parser = etree.parse(f) nameSpaces = {'ss': 'urn:schemas-microsoft-com:office:spreadsheet'} nameSpace = '{urn:schemas-microsoft-com:office:spreadsheet}' root = parser.xpath("/ss:Root", namespaces = nameSpaces)[0] child = etree.Element("Child") child.set(nameSpace+"Color","Red") root.append(child) print etree.tostring(root, pretty_print=True) And my result: <Root xmlns="urn:schemas-microsoft-com:office:spreadsheet" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet"> <Child ss:Color="Blue"/> <Child Color="Red"/> </Root> I would like to have: <Root xmlns="urn:schemas-microsoft-com:office:spreadsheet" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet"> <Child ss:Color="Blue"/> <Child ss:Color="Red"/> </Root> If i changed position of default namespace (I will put it at the end) everything will be OK. I will get attribute with prefix ss. But I cannot do this in that way. Is there other way to handle with that problem? Thanks in advance Bartek

3 2

parse problem
by contro opinion Jan. 18, 2012

Jan. 18, 2012

here is my code: import urllib import lxml.html down=" http://sc.hkex.com.hk/gb/www.hkex.com.hk/chi/market/sec_tradinfo/stockcode/… " file=urllib.urlopen(down). read() root=lxml.html.document_fromstring(file) data1 = root.xpath('//tr[@class="tr_normal" and .//img]') print "the row which contains img :" for u in data1: print u.text_content() data2 = root.xpath('//tr[@class="tr_normal" and not(.//img)]') print "the row which do not contain img :" for u in data2: print u.text_content() the output is :(i omit many lines ) the row which contains img : 00329 the row which do not contain img : 00001长江实业1,000#HOF ................many lines omitted 00327百富环球1,000#H 00328ALCO HOLDINGS2,000# i wondered why there are so many lines i can't get such as : (you can see in the web http://sc.hkex.com.hk/gb/www.hkex.com.hk/chi/market/sec_tradinfo/stockcode/… ) 00330 思捷环球 100 # H O F 00331 春天百货 2,000 # H 00332 NGAI LIK IND 4,000 # ...................many lines ommitted i want to know how can i get these ??

1 0

need help with finding elements
by Mag Gam Jan. 16, 2012

Jan. 16, 2012

Hello, I have a xml file like this, http://msdn.microsoft.com/en-us/library/windows/desktop/ms762271(v=vs.85).a… Lets say I want to get all the book titles authored by, ¨O'Brien, Tim¨. What is the best way to do that?

2 2

etree module for windows
by Tony Peña Jan. 7, 2012

Jan. 7, 2012

Hi to all... 1st at all, sorry by my english, I expecting be unserstand me :-) I making an app, and for the users without any tech-skill, I try to insert in my app, all libraries to can import locally without python and others tools, And 1 of them is lxml, and with linux, I insert in my modules path just "_elementpath.py" and "etree.so", and without python-lxml package for debian installed my app runs very fine... so my question.. How can i get the etree.so but for windows??? because my app is write in python+qt4, so i want to be make with py2exe for windows but when import etree linked in modules directory, windows not understand the .so file,... so how can i get the etree.dll?? or etree.py to can use in my app for windows??? Thanxs for your time -- Antonio Peña Secure email with PGP 0x8B021001 available at http://pgp.mit.edu Fingerprint: 74E6 2974 B090 366D CE71 7BB2 6476 FA09 8B02 1001

1 0