Mailman 3 November 2015 - lxml - The Python XML Toolkit

lxml.etree.XPathEvalError: Invalid expression for correct XPath expression on large XML file
by Dennis Walter 24 Nov '15

24 Nov '15

Hi, I am using Python/lxml to process large (~300MB) XML files and extract information with XPath. I stumbled upon a strange error that I cannot make any sense of: All XPath expressions using a "where" clause (square brackets) fail with the error message "lxml.etree.XPathEvalError: Invalid expression" (see stack traces below). Happens with different versions, 32-bit and 64-bit, and different OSs. I cannot reproduce this behaviour with small XML files, and I could not find any information about this. Has anyone experienced something similar? Can anybody determine some useful information from the stack trace? Many thanks, Dennis Tested environments: Linux ===== >>> xml = etree.parse('stammdaten.xml') >>> xml.xpath('//foo[@id]') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "lxml.etree.pyx", line 2115, in lxml.etree._ElementTree.xpath (src/lxml/lxml.etree.c:57654) File "xpath.pxi", line 370, in lxml.etree.XPathDocumentEvaluator.__call__ (src/lxml/lxml.etree.c:146564) File "xpath.pxi", line 238, in lxml.etree._XPathEvaluatorBase._handle_result (src/lxml/lxml.etree.c:144962) File "xpath.pxi", line 224, in lxml.etree._XPathEvaluatorBase._raise_eval_error (src/lxml/lxml.etree.c:144817) lxml.etree.XPathEvalError: Invalid expression >>> print("%-20s: %s" % ('Python', sys.version_info)) Python : sys.version_info(major=3, minor=4, micro=3, releaselevel='final', serial=0) >>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION)) lxml.etree : (3, 3, 3, 0) >>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION)) libxml used : (2, 9, 1) >>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION)) libxml compiled : (2, 9, 1) >>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION)) libxslt used : (1, 1, 28) >>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION)) libxslt compiled : (1, 1, 28) Windows ======= >>> xml = etree.parse('stammdaten.xml') >>> xml.xpath('//*[@id]') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "src/lxml/lxml.etree.pyx", line 2272, in lxml.etree._ElementTree.xpath (src\lxml\lxml.etree.c:70786) File "src/lxml/xpath.pxi", line 359, in lxml.etree.XPathDocumentEvaluator.__call__ (src\lxml\lxml.etree.c:179148) File "src/lxml/xpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_result (src\lxml\lxml.etree.c:177421) lxml.etree.XPathEvalError: Invalid expression >>> print("%-20s: %s" % ('Python', sys.version_info)) Python : sys.version_info(major=3, minor=4, micro=0, releaselevel='final', serial=0) >>> print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION)) lxml.etree : (3, 5, 0, 0) >>> print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION)) libxml used : (2, 9, 2) >>> print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION)) libxml compiled : (2, 9, 2) >>> print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION)) libxslt used : (1, 1, 28) >>> print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION)) libxslt compiled : (1, 1, 28)

2 1

Hot to remove empty tags
by Brijesh Bittu 19 Nov '15

19 Nov '15

Suppose I have this HTML Hello <strike>World</strike>. How can I strip empty tags and tags containing only spaces to get the following output? : Hello <strike>World</strike>.

2 1

lxml 3.5.0 final released
by Stefan Behnel 13 Nov '15

13 Nov '15

Hi all, since there were only minor complaints about the last beta, I just released the final version of the lxml 3.5.0. This is a feature release with many small and large improvements all over the place. Thanks to everyone who contributed. The documentation is here: http://lxml.de/ Download: http://lxml.de/files/lxml-3.5.0.tgz Signature: http://lxml.de/files/lxml-3.5.0.tgz.asc Changelog: http://lxml.de/3.5/changes-3.5.0.html Github: https://github.com/lxml/lxml/commit/b00892ed4ac551c39ff7ba42ff55524ad2516028 This release was built using Cython 0.23.4. If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.5.0 (2015-11-13) ================== Bugs fixed ---------- * Unicode string results failed XPath queries in PyPy. * LP#1497051: HTML target parser failed to terminate on exceptions and continued parsing instead. * Deprecated API usage in doctestcompare. 3.5.0b1 (2015-09-18) ==================== Features added -------------- * ``cleanup_namespaces()`` accepts a new argument ``keep_ns_prefixes`` that does not remove definitions of the provided prefix-namespace mapping from the tree. * ``cleanup_namespaces()`` accepts a new argument ``top_nsmap`` that moves definitions of the provided prefix-namespace mapping to the top of the tree. * LP#1490451: ``Element`` objects gained a ``cssselect()`` method as known from ``lxml.html``. Patch by Simon Sapin. * API functions and methods behave and look more like Python functions, which allows introspection on them etc. One side effect to be aware of is that the functions now bind as methods when assigned to a class variable. A quick fix is to wrap them in ``staticmethod()`` (as for normal Python functions). * ISO-Schematron support gained an option ``error_finder`` that allows passing a filter function for picking validation errors from reports. * LP#1243600: Elements in ``lxml.html`` gained a ``classes`` property that provides a set-like interface to the ``class`` attribute. Original patch by masklinn. * LP#1341964: The soupparser now handles DOCTYPE declarations, comments and processing instructions outside of the root element. Patch by Olli Pottonen. * LP#1421512: The ``docinfo`` of a tree was made editable to allow setting and removing the public ID and system ID of the DOCTYPE. Patch by Olli Pottonen. * LP#1442427: More work-arounds for quirks and bugs in pypy and pypy3. * ``lxml.html.soupparser`` now uses BeautifulSoup version 4 instead of version 3 if available. Bugs fixed ---------- * Memory errors that occur during tree adaptations (e.g. moving subtrees to foreign documents) could leave the tree in a crash prone state. * Calling ``process_children()`` in an XSLT extension element without an ``output_parent`` argument failed with a ``TypeError``. Fix by Jens Tröger. * GH#162: Image data in HTML ``data`` URLs is considered safe and no longer removed by ``lxml.html.clean`` JavaScript cleaner. * GH#166: Static build could link libraries in wrong order. * GH#172: Rely a bit more on libxml2 for encoding detection rather than rolling our own in some cases. Patch by Olli Pottonen. * GH#159: Validity checks for names and string content were tightened to detect the use of illegal characters early. Patch by Olli Pottonen. * LP#1421921: Comments/PIs before the DOCTYPE declaration were not serialised. Patch by Olli Pottonen. * LP#659367: Some HTML DOCTYPE declarations were not serialised. Patch by Olli Pottonen. * LP#1238503: lxml.doctestcompare is now consistent with stdlib's doctest in how it uses ``+`` and ``-`` to refer to unexpected and missing output. * Empty prefixes are explicitly rejected when a namespace mapping is used with ElementPath to avoid hiding bugs in user code. * Several problems with PyPy were fixed by switching to Cython 0.23.

1 0

Converting from Element to ObjectifiedElement
by Nathaniel Mishkin 10 Nov '15

10 Nov '15

I'm new to using lxml and not that much of a Python expert, so sorry in advance if this question has an obvious answer: I'm mostly using lxml.objectify (to parse some XML in an HTTP response) but in some cases I need to use lxml.etree (also to parse an HTTP response) because in those cases the XML I'm dealing with is less predictable/regular. However, once I've done some initial processing on the non-Objectified version, I'd like to make an Objectified version of it. I could convert through a string and back, but I'm looking for something (that I assume would be) faster. Any suggestions? Thanks. Nat

3 5

Fwd: [pypy-dev] Looking for clues to consistent Seg Fault in PyPy 2.6.1
by Jeff Doran 07 Nov '15

07 Nov '15

In light of this feedback from the PyPy dev team, is there any ongoing work on a cffi version of lxml? I know there is a github repo here ( https://github.com/amauryfa/lxml/tree/cffi) that lists PyPy 2.4 support, but it shows no activity in the last year so I'm guessing it's dormant. Thanks, - Jeff ---------- Forwarded message ---------- From: Armin Rigo <arigo(a)tunes.org> Date: Wed, Nov 4, 2015 at 5:09 AM Subject: Re: [pypy-dev] Looking for clues to consistent Seg Fault in PyPy 2.6.1 To: Jeff Doran <jdoran(a)lexmachina.com> Cc: Amaury Forgeot d'Arc <amauryfa(a)gmail.com>, PyPy Developer Mailing List < pypy-dev(a)python.org>, lxml(a)lxml.de Hi Jeff, On Tue, Nov 3, 2015 at 11:57 PM, Jeff Doran <jdoran(a)lexmachina.com> wrote: > Just an update on this problem. I retested using the latest lxml source > from here ( 3.5.0b2?) There were a few fixes to PyPy's cpyext layer. But there was no change to lxml itself. That means it is still using a broken approach to work around a CPython C API difference. Namely, this is the weakref stuff which is done incorrectly---although I don't know how to do that correctly. For now, lxml (the non-cffi version) is not compatible with PyPy at all. A bientôt, Armin.

2 1

Re: [lxml] [pypy-dev] Looking for clues to consistent Seg Fault in PyPy 2.6.1
by Jeff Doran 03 Nov '15

03 Nov '15

Hi, Just an update on this problem. I retested using the latest lxml source from here <https://github.com/lxml/lxml> ( 3.5.0b2?) and a version of PyPy-4.0 from here <https://bitbucket.org/pypy/pypy/downloads/pypy-4.0.0-src.tar.bz2> that I built for Debian 7.9. The good news is I got much farther in the lxml test suite then before, but the bad news is I still encountered my old friend SIGSEGV. This is consistently occuring during test_delslice_negative2 in lxml.tests.test_elementtree.ETreeTestCase which as I said is much further thru the suite of tests then previously. This is encouraging. For now I'm still unable to use this combination so if anyone has a suggestion on how I might proceed to utilize both PyPy and lxml I would be most appreciative. Thanks. - Jeff --------------------------------------------------------- nosetests -vv --nocapture nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] nose.selector: INFO: /home/jeff/lxml/src/lxml/etree.pypy-26.so is executable; skipped nose.selector: INFO: /home/jeff/lxml/src/lxml/objectify.pypy-26.so is executable; skipped Comparing with ElementTree 1.3.0 Comparing with cElementTree 1.3.0 TESTED VERSION: 3.5.0.beta1 Python: (major=2, minor=7, micro=10, releaselevel='final', serial=42) lxml.etree: (3, 5, 0, -99) libxml used: (2, 8, 0) libxml compiled: (2, 8, 0) libxslt used: (1, 1, 26) libxslt compiled: (1, 1, 26) lxml.html.tests.test_autolink.test_suite ... ok lxml.html.tests.test_basic.test_suite ... ok test_allow_tags (lxml.html.tests.test_clean.CleanerTest) ... ok test_clean_invalid_root_tag (lxml.html.tests.test_clean.CleanerTest) ... ok test_safe_attrs_excluded (lxml.html.tests.test_clean.CleanerTest) ... ok test_safe_attrs_included (lxml.html.tests.test_clean.CleanerTest) ... ok lxml.html.tests.test_clean.test_suite ... ok lxml.html.tests.test_diff.test_suite ... ok lxml.html.tests.test_elementsoup.test_suite ... ok ... lots of successful test output removed test_delitem (lxml.tests.test_elementtree.ETreeTestCase) ... ok test_delitem_tail (lxml.tests.test_elementtree.ETreeTestCase) ... ok test_delslice (lxml.tests.test_elementtree.ETreeTestCase) ... ok test_delslice_child_tail (lxml.tests.test_elementtree.ETreeTestCase) ... ok test_delslice_memory (lxml.tests.test_elementtree.ETreeTestCase) ... ok test_delslice_negative1 (lxml.tests.test_elementtree.ETreeTestCase) ... ok test_delslice_negative2 (lxml.tests.test_elementtree.ETreeTestCase) ... fish: Job 2, “nosetests -vv --nocapture ” terminated by signal SIGSEGV (Address boundary error) --------------------------------------------------------- On Mon, Oct 12, 2015 at 2:30 AM, Armin Rigo <arigo(a)tunes.org> wrote: > Hi, > > On Mon, Oct 12, 2015 at 11:24 AM, Amaury Forgeot d'Arc > <amauryfa(a)gmail.com> wrote: > > We define it as a string (in pypy/module/cpyext/include/patchlevel.h) > > #define PYPY_VERSION "2.6.0-alpha0" > > Yes, but i don't know how to do this check in Cython. Well, it's probably > easy. > > Armin >

1 0