Mailman 3 June 2012 - lxml - The Python XML Toolkit

installing lxml 2.3.4 with Python 3.3
by David Beck June 21, 2012

June 21, 2012

I've been trying to install lxml 2.3.4 with Python 3.3 on a Mac running OS 10.7. I've followed the directions given at http://lxml.de/build.html and when I try to import etree from lxml, I get the following output: >>> from lxml import etree Traceback (most recent call last): File "<pyshell#1>", line 1, in <module> from lxml import etree File "<frozen importlib._bootstrap>", line 1082, in _handle_fromlist File "<frozen importlib._bootstrap>", line 1015, in _find_and_load File "<frozen importlib._bootstrap>", line 314, in _check_name_wrapper File "<frozen importlib._bootstrap>", line 234, in set_package_wrapper File "<frozen importlib._bootstrap>", line 247, in set_loader_wrapper File "<frozen importlib._bootstrap>", line 740, in load_module File "lxml.etree.pyx", line 48, in init lxml.etree (src/lxml/lxml.etree.c:159821) ValueError: level must be >= 0 The simple "import lxml" works fine. Note that I followed the same procedure and installedl lxml for Python 3.2, and it works fine there. David

2 1

lxml running on PyPy
by Stefan Behnel June 19, 2012

June 19, 2012

Hi, here's a little status update regarding lxml on PyPy. I got the basics of lxml.etree working so far, mostly by patching up Cython and tracking down bugs in PyPy's cpyext (CPython C-API compatibility) layer. I'm still getting crashes during error reporting and didn't care much about XPath or XSLT yet. But given that those do not have much Python interaction per se, I don't expect major surprises on that front. The results are very encouraging, given that PyPy lacks support for many of the tweaks and hacks that are possible in CPython. Here's a little parser benchmark: $ python2.7 -m timeit -s 'import lxml.etree as et' 'et.parse("hamlet.xml")' 100 loops, best of 3: 4.61 msec per loop $ pypy -m timeit -s 'import lxml.etree as et' 'et.parse("hamlet.xml")' 100 loops, best of 3: 5.74 msec per loop Pretty acceptable. That makes lxml the fastest XML parser that currently exists for PyPy. And here's a worst case benchmark for element proxy instantiation and iteration, likely the most heavily tuned parts of lxml when running in CPython: $ python2.7 -m timeit -s 'import lxml.etree as et; \ t=et.parse("hamlet.xml")' 'list(t.iter())' 100 loops, best of 3: 2.71 msec per loop $ pypy -m timeit -s 'import lxml.etree as et; \ t=et.parse("hamlet.xml")' 'list(t.iter())' 10 loops, best of 3: 28.2 msec per loop That's about a factor of 10. Sounds huge, but it's actually not bad, considering the amount of extra work that has to be done for PyPy here. Certainly doesn't render it unusable, we are still talking milliseconds after all. And so far, there hasn't gone any tuning into it, so it's not the final word. I'm pretty optimistic. BTW, if you're interested in improvements on this front, you can help getting this done faster by using the "donate" button on lxml's project home page. Any donation will help in freeing some of my time for this. Stefan

4 17

Separate the namespace URI and local name from the Python API
by Simon Sapin June 17, 2012

June 17, 2012

Hi, I would like to get the namespace URI and local part of an element’s name separately. I think I could manage with the C/Cython API, but I don’t see how to do it from Python. The best I can think of is: def split_ns(element): tag = element.tag if '}' in tag: ns, local = tag.split('}', 1) return ns[1:], local # remove '{' from ns else: return None, tag Is the above correct? (Ie. can the URI or local part ever contain '}'?) Is there a better way? These appear to work but using XPath for this seems overkill: etree.XPath('namespace-uri(.)') etree.XPath('local-name(.)') Regards, -- Simon Sapin

3 5

segfault - potential double free when using iterparse
by Alon Horev June 13, 2012

June 13, 2012

Hi All, First of all, I'm not opening a bug yet as I'm not certain whether this is a CPython bug or lxml bug. I'm getting a segfault within python's GC (garbage collector) module. here's the stack trace: #0 0x00007fc7e9f6b76e in gc_list_remove (op=0x7fc79cef3d98) at Modules/gcmodule.c:211 #1 PyObject_GC_Del (op=0x7fc79cef3d98) at Modules/gcmodule.c:1503 #2 0x00007fc7e9f2ac0f in PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2894 #3 0x00007fc7e9ea5b79 in gen_send_ex (arg=None, exc=<value optimized out>, gen=<value optimized out>) at Objects/genobject.c:84 #4 0x00007fc7e9ea6185 in gen_close (self=<generator at remote 0x7fc7c1ba73c0>) at Objects/genobject.c:130 #5 gen_del (self=<generator at remote 0x7fc7c1ba73c0>) at Objects/genobject.c:165 #6 0x00007fc7e9ea5a1b in gen_dealloc (gen=0x7fc7c1ba73c0) at Objects/genobject.c:32 In order to see what object the gc is freeing i tried casting it to a PyObject (we're freeing a lxml object): (gdb) p (PyObject*) op $17 = <ml.etree.DocInfo.standalone.__get__ at remote 0x7fc79cef3d98> Similar bugs (http://osdir.com/ml/python.bugs/2000-12/msg00214.html) blame the extension module for calling dealloc explicitly more than once or doing forbidden things in __del__. this is how i use lxml: from lxml.etree import iterparse def safe_iterparse(*args, **kwargs): for event, element in iterparse(*args, **kwargs): try: yield (event, element) finally: element.clear() I tried parsing the same data but It didn't reproduce.. It happens after a few hours running in production while generating etree using the builder and parsing using the attached code. anyone familiar with these kind of bugs in c extensions/cpython/lxml? could you give pointers to what I should be looking for? should I open a bug? how can I debug lxml? some version info: CPython version: 2.7.2 on linux. lxml: 2.3.3 thanks, Alon Horev

2 5

How to produce explicitly closed form of HTML tags
by Andreas Maier June 13, 2012

June 13, 2012

Hi, I am using lxml's XSLT support to transform XML to HTML. The XSLT specifies the following xsl:output directive: <xsl:output method="html" version="4.0" encoding="UTF-8" media-type="text/html" doctype-public="-//W3C//DTD HTML 4.01//EN" standalone="yes" omit-xml-declaration="yes" indent="no"/> I serialize the output element tree to a string using the etree.tostring() function: xslt_tree = etree.parse(xslt_url) xslt_transform = etree.XSLT(xslt_tree.getroot()) out_tree = xslt_transform(in_root_elem) fp.write(etree.tostring(out_tree)) This creates self-closing HTML tags, for example: <script type="text/javascript" src="tocgen.js"/> This hurts, because some browsers (including IE8 and FF10) do not support self-closing tags for some tags when the doctype is HTML. The script tag is a particularly nasty one because not recognizing that it is actually closed causes the whole rest of the document to be interpreted as (invalid) Javascript. But doing that with other tags may create problems as well. What works on all browsers I tested, is the explicitly closed form: <script type="text/javascript" src="tocgen.js"></script> See this thread for discussion: http://stackoverflow.com/questions/69913/why-dont-self-closing-script-tags-… BTW, Xalan produces the explicitly closed form for the same scenario. (Just to mention it as one other data point, not that I would consider going back to Xalan - lxml is much much faster and by comparing lxml and Xalan output beyond differences such as self-closing tags I found already one difference that seems to be a Xalan bug). -> Is there a way to get lxml's HTML serialization support to produce the explicitly closed form ? The lxml version I am using is lxml 2.3 with libxml2 statically linked, for Python 2.6, on Windows (lxml-2.3.win32-py2.6.exe <http://pypi.python.org/packages/2.6/l/lxml/lxml-2.3.win32-py2.6.exe#md5=413…>). -> Is there a later version that has a Windows installer and libxml2 statically linked ? (Other people need to be able to set that up for my stuff to work, and this way it is real simple). Andy

5 5

XPathFunctionError on second call with user functions
by Simon Sapin June 10, 2012

June 10, 2012

Hi, When using a custom XPath function (as lxml.cssselect does for lower-case) I get a strange XPathFunctionError error when using a compiled XPath object more than once. Here is a minimal test case: from lxml import etree, html def _make_lower_case(context, s): return s.lower() ns = etree.FunctionNamespace('http://codespeak.net/lxml/css/') ns.prefix = 'css' ns['lower-case'] = _make_lower_case #doc = html.parse('/home/simon/css3-backgrounds.html') doc = html.parse('http://www.w3.org/TR/css3-background/') expr = etree.XPath('//*[css:lower-case(name(.)) = "li"]') print('first call') expr(doc) print('second call') expr(doc) print('third call') expr(doc) And the output I get with lxml 2.3.4: first call second call Traceback (most recent call last): File "xpath_lower.py", line 20, in <module> expr(doc) File "xpath.pxi", line 466, in lxml.etree.XPath.__call__ (src/lxml/lxml.etree.c:119238) File "xpath.pxi", line 235, in lxml.etree._XPathEvaluatorBase._handle_result (src/lxml/lxml.etree.c:117011) File "lxml.etree.pyx", line 280, in lxml.etree._ExceptionContext._raise_if_stored (src/lxml/lxml.etree.c:7454) lxml.etree.XPathFunctionError: XPath function '{b'\xd0\x85?\x01'}b'lower-case'' not found Note that the '\xd0\x85?\x01' string looks random and changes every time I run the script. Is something wrong in my code? Regards, -- Simon Sapin

2 3

XPath: Allowed characters in element names
by Simon Sapin June 9, 2012

June 9, 2012

Hi, Is there something in the lxml API that can tell me if a particular Unicode string is a valid element/attribute name or prefix in an XPath expression? Obviously some characters like U+005B '[' have a special meaning in XPath, but others like U+00D7 MULTIPLICATION SIGN are note allowed for not apparent reason. Trying to make an Element object with that name and catching the ValueError is almost what I want, except that Element accepts the {namespace_URI}local_name syntax. Following links from the XPath spec I find that all of these are NCName tokens: http://www.w3.org/TR/REC-xml-names/#NT-NCName So I guess I will encode this grammar as a regexp. It should work but it won’t be pretty, with 17 or so character ranges. Questions: 1. Does the sets of allowed or not allowed names in the implementation match the grammar linked above? (I guess I should check the libxml2 source for this.) 2. Is there a better way than an ugly regexp? 3. When I get an invalid name, I can replace eg. 'foo' with something like '*[name() = "foo"]'. But if elements can never have this name anyway, is '*[0]' equivalent? (An expression that never matches.) Context: I want to make sure that XPath expressions built by cssselect are valid and correct. With the recently-fixed backshlash-escapes it is really easy to find a selector that currently translate to an invalid XPath expression, or even to a valid but incorrect expression. (Code injection in XPath is probably not as big of a security problem is it can with in SQL, but still.) Thanks, -- Simon Sapin

1 0

Suggestions for lxml topics at PyCon-DE ? Anyone for a code sprint?
by Stefan Behnel June 6, 2012

June 6, 2012

Hi, I gave a somewhat generic lxml talk at last year's PyCon-DE in Leipzig. In case some people on this list are going there, is there anything more special you'd want me to present this year? The deadline has been extended to Mai 15th, so there's some time left to come up with good ideas. I'm also considering to run an lxml sprint at the end of the conference. Anyone interested in learning how to work on lxml's code base? There's quite a number of open bugs that would lend themselves to getting your hands dirty, be it in Cython code or Python code (e.g. lxml.html). Working on those would be a very good way to give something back to the tool you are using. Stefan

1 0

objectify.parse() returns lxml.etree._ElementTree instead of lxml.objectify.ObjectifiedElement
by Roger Hoover June 3, 2012

June 3, 2012

Hi, Any idea why objectify.parse() is returning an lxml.etree._ElementTree object while objectify.parsestring() returns lxml.objectify.ObjectifiedElement? I've installed lxml 2.3.4 using easy_install on Ubuntu 10.04. from lxml import objectify import StringIO obj = objectify.parse(StringIO.StringIO("<foo><bar>1</bar></foo>")) print type(obj) #prints <type 'lxml.etree._ElementTree'> Thanks, Roger

2 2

xml install problems
by William Herry June 2, 2012

June 2, 2012

Hi, I am try to install xml on centos6.2 with command: pip-python install xml==2.3 it gives me this error: Downloading/unpacking lxml==2.3 Running setup.py egg_info for package lxml Building lxml version 2.3. Building with Cython 0.14.1. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib64 Installing collected packages: lxml Running setup.py install for lxml Building lxml version 2.3. Building with Cython 0.14.1. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib64 building 'lxml.etree' extension gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w {standard input}: Assembler messages: {standard input}:150508: Warning: end of file not at end of a line; newline inserted gcc: Internal error: Killed (program cc1) Please submit a full bug report. See <http://bugzilla.redhat.com/bugzilla> for instructions. error: command 'gcc' failed with exit status 1 Complete output from command /usr/bin/python -c "import setuptools;__file__='/root/build/lxml/setup.py';execfile(__file__)" install --single-version-externally-managed --record /tmp/pip-g_IvOB-record/install-record.txt: Building lxml version 2.3. Building with Cython 0.14.1. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib64 running install running build running build_py running build_ext building 'lxml.etree' extension gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/libxml2 -I/usr/include/python2.6 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.6/src/lxml/lxml.etree.o -w {standard input}: Assembler messages: {standard input}:150508: Warning: end of file not at end of a line; newline inserted gcc: Internal error: Killed (program cc1) Please submit a full bug report. See <http://bugzilla.redhat.com/bugzilla> for instructions. error: command 'gcc' failed with exit status 1 ---------------------------------------- Command /usr/bin/python -c "import setuptools;__file__='/root/build/lxml/setup.py';execfile(__file__)" install --single-version-externally-managed --record /tmp/pip-g_IvOB-record/install-record.txt failed with error code 1 Storing complete log in /root/.pip/pip.log I already installed libxml2-devel and libxslt-devel, can some body tell me what's going on? thanks in adv Regards -- =========================== William Herry WilliamHerryChina(a)Gmail.com

2 2