Mailman 3 September 2017 - lxml - The Python XML Toolkit

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 11 Apr '23

11 Apr '23

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

9 9

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 10 Apr '23

10 Apr '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

6 6

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 31 Aug '22

31 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

thread-specific data for xslt callbacks
by Bob Kline 13 Oct '17

13 Oct '17

I need to make some thread-specific data available to my xslt function callbacks. I think I've sort of reverse-engineered a way to do it, by creating my own parser class, stuffing the data I need into an instance of that class, using the parser instance to parse the XSL/T filter, and then in the extension function following the context argument's context_node to its root tree, which in turn gets me back to my own parser object (and my thread-specific data). Is there a more elegant way to do it that I just haven't yet stumbled across in reading through the API docs? Thanks! -- Bob Kline http://www.rksystems.com mailto:bkline@rksystems.com

2 8

Strange results of xpath element search using string-value
by Алексей Алексей 30 Sep '17

30 Sep '17

2 1

xs:include not finding included file after XSLT transformation
by Markus Schöpflin 29 Sep '17

29 Sep '17

Using python-lxml-2.2.3-1.1.el6.x86_64 on RHEL6. I have a schema file using relative include statements like this: <xs:include schemaLocation="some/relative/path/blah.xsd"/> From this file an element tree object is created: schematree = etree.parse(schemafile) Then a schema object is created using: etree.XMLSchema(schematree) This works fine, the include statement is resolved relative to the locate of the schema file. Now I need to modify the schema tree object via an XSLT transformation, before the schema object is created: transform = etree.XSLT(...) schematree = transform(schematree) Now the creation of the schema object fails with the error message: lxml.etree.XMLSchemaParseError: Element '{http://www.w3.org/2001/XMLSchema}include': Failed to load the document 'some/relative/path/blah.xsd' for inclusion., line 3 I suppose this is caused by the fact that applying the XML transformation to the original element tree object creates a new element tree object (or rather _XSLTResultTree which is derived from _ElementTree), and the new object does not have a valid docinfo property. (It is empty.) So the include file is resolved relative to the current working directory, and not relative to the location of the original schema XML document. Now, is there a way to get the included file resolved relative to the location of the original schema. I tried setting the docinfo property on the result of the transformation, but this is not allowed. Is there any other way? TIA, Markus

2 5

C14N namespace prefixes
by Nagy László Zsolt 27 Sep '17

27 Sep '17

Hello, I need to calculate the SHA1 digest of an XML node. In order to do this correctly, I need to do C14N on the node. Specifically, http://www.w3.org/2001/10/xml-exc-c14n Here is a minimal working example: import io import copy from lxml import etree ORIGINAL_XML = b'''<soap:Envelope xmlns:ns="http://docs.oasis-open.org/ws-sx/ws-trust/200512" xmlns:soap="http://www.w3.org/2003/05/soap-envelope"> <soap:Header> <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1…" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-…"> <wsu:Timestamp wsu:Id="TS-1a8236fc-8e3a-9b71-495f-20f52709e893"> <wsu:Created>2017-09-19T07:38:45Z</wsu:Created><wsu:Expires>2017-09-19T08:38:45Z</wsu:Expires> </wsu:Timestamp> </wsse:Security> </soap:Header> <soap:Body xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-…" wsu:Id="id-1ea12929-0b1a-54d4-08db-b049cee527b5"> <ns:RequestSecurityToken> <ns:RequestType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</ns:RequestType> <ns:TokenType>http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV2.0</ns:TokenType> </ns:RequestSecurityToken> </soap:Body> </soap:Envelope> '''.replace(b'\n', b'') PREFIXES = ["soap", "wsse", "wsu"] # This is known to be the "good" C14N version of the wsu:Timestamp node GOOD_C14N_XML = b'''<wsu:Timestamp xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1…" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-…" wsu:Id="TS-1a8236fc-8e3a-9b71-495f-20f52709e893"><wsu:Created>2017-09-19T07:38:45Z</wsu:Created><wsu:Expires>2017-09-19T08:38:45Z</wsu:Expires></wsu:Timestamp>''' root_node = etree.fromstring(ORIGINAL_XML) test_node = root_node.find(".//{*}Timestamp") # This is the node to calculate digest for output = io.BytesIO() node = copy.copy(test_node) node.getroottree().write(output, method="c14n", inclusive_ns_prefixes=PREFIXES, exclusive=True, with_comments=False, pretty_print=False) output.seek(0) TEST_RESULT = output.read() assert TEST_RESULT == GOOD_C14N_XML The GOOD_C14N_XML contains the "good" canonical version of the Timestamp element. I say that it is "good" because for the ORIGINAL_XML, the C14N version created by a java program looks like this, and I must replicate this exact format. So by "good" I mean: this is the one that I need to replicate. In this example program, the assertion fails. The GOOD_C14N_XML contains all namespace declarations that were given in the PREFIXES, *in that specific order*. The TEST_RESULT only contains the wsu namespace. Questions: * Is it possible to achieve my goal with lxml? E.g. create a C14N format that matches the GOOD_C14N_XML in every bit. * I have noticed that the write method accepts pretty_print=True when write method is c14n. It has no effect. But shouldn't it throw an exception? (The same way it throws an exception when encoding is specified for c14n method.) Thanks, Laszlo

2 2

lxml 4.0.0 released
by Stefan Behnel 21 Sep '17

21 Sep '17

Hi all, I just released lxml 4.0.0 with several new features. Thanks to everyone who contributed. The documentation is here: http://lxml.de/ Download: https://pypi.python.org/packages/07/76/9f14811d3fb91ed7973a798ded15eda41607… Signature: https://pypi.python.org/packages/07/76/9f14811d3fb91ed7973a798ded15eda41607… Changelog: http://lxml.de/4.0/changes-4.0.0.html Github: https://github.com/lxml/lxml/releases/tag/lxml-4.0.0 This release was built using Cython 0.26.1. If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 4.0.0 (2017-09-17) ================== Features added -------------- * The ElementPath implementation is now compiled using Cython, which speeds up the ``.find*()`` methods quite significantly. * The modules ``lxml.builder``, ``lxml.html.diff`` and ``lxml.html.clean`` are also compiled using Cython in order to speed them up. * ``xmlfile()`` supports async coroutines using ``async with`` and ``await``. * ``iterwalk()`` has a new method ``skip_subtree()`` that prevents walking into the descendants of the current element. * ``RelaxNG.from_rnc_string()`` accepts a ``base_url`` argument to allow relative resource lookups. * The XSLT result object has a new method ``.write_output(file)`` that serialises output data into a file according to the ``<xsl:output>`` configuration. Bugs fixed ---------- * GH#251: HTML comments were handled incorrectly by the soupparser. Patch by mozbugbox. * LP#1654544: The html5parser no longer passes the ``useChardet`` option if the input is a Unicode string, unless explicitly requested. When parsing files, the default is to enable it when a URL or file path is passed (because the file is then opened in binary mode), and to disable it when reading from a file(-like) object. Note: This is a backwards incompatible change of the default configuration. If your code parses byte strings/streams and depends on character detection, please pass the option ``guess_charset=True`` explicitly, which already worked in older lxml versions. * LP#1703810: ``etree.fromstring()`` failed to parse UTF-32 data with BOM. * LP#1526522: Some RelaxNG errors were not reported in the error log. * LP#1567526: Empty and plain text input raised a TypeError in soupparser. * LP#1710429: Uninitialised variable usage in HTML diff. * LP#1415643: The closing tags context manager in ``xmlfile()`` could continue to output end tags even after writing failed with an exception. * LP#1465357: ``xmlfile.write()`` now accepts and ignores None as input argument. * Compilation under Py3.7-pre failed due to a modified function signature. Other changes ------------- * The main module source files were renamed from ``lxml.*.pyx`` to plain ``*.pyx`` (e.g. ``etree.pyx``) to simplify their handling in the build process. Care was taken to keep the old header files as fallbacks for code that compiles against the public C-API of lxml, but it might still be worth validating that third-party code does not notice this change.

3 10

Test failure(s) in lxml-4.0.0
by Ken Moffat 20 Sep '17

20 Sep '17

Hi, testing 4.0.0 I'm getting the following error in with both python2 and python3 on x86_64 linux (LFS : gcc-7.2, python-2.7.13, python-3.6.2, with libxml2-2.9.4, libxslt-1.1.29 AND on another machine with libxml2-2.9.5, libxslt-1.1.30 with a fix so that xslt-config --cflags works again), both with linux-4.13.1. I will admit that I take a jaundiced view of testsuites, but now that I've read your page on building from source I see you want to know about failures ;) ERROR: test_xmlschema_parse_fixed_attributes (lxml.tests.test_xmlschema.ETreeXMLSchemaTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.6/unittest/case.py", line 59, in testPartExecutor yield File "/usr/lib/python3.6/unittest/case.py", line 605, in run testMethod() File "/scratch/working/lxml-4.0.0/src/lxml/tests/test_xmlschema.py", line 217, in test_xmlschema_parse_fixed_attributes parser=parser) File "/scratch/working/lxml-4.0.0/src/lxml/tests/common_imports.py", line 164, in parse return etree.parse(f, parser=parser) File "src/lxml/etree.pyx", line 3441, in lxml.etree.parse doc = _parseDocument(source, parser, base_url) File "src/lxml/parser.pxi", line 1851, in lxml.etree._parseDocument return _parseMemoryDocument(source.getvalue(), url, parser) File "src/lxml/parser.pxi", line 1871, in lxml.etree._parseMemoryDocument c_doc = _parseDoc(text, url, parser) File "src/lxml/parser.pxi", line 1752, in lxml.etree._parseDoc return (<_BaseParser>parser)._parseUnicodeDoc(text, c_filename) File "src/lxml/parser.pxi", line 1066, in lxml.etree._BaseParser._parseUnicodeDoc return context._handleParseResultDoc(self, result, None) File "src/lxml/parser.pxi", line 598, in lxml.etree._ParserContext._handleParseResultDoc return _handleParseResult(self, self._c_ctxt, result, File "src/lxml/parser.pxi", line 709, in lxml.etree._handleParseResult _raiseParseError(c_ctxt, filename, context._error_log) File "src/lxml/parser.pxi", line 638, in lxml.etree._raiseParseError raise error_log._buildParseException( File "<string>", line 0 lxml.etree.XMLSyntaxError: Element 'b', attribute 'hardy': The value '(null)' does not match the fixed value constraint 'hey'. On one build on the 2.9.4/1.1.29 system I also got two other failures, which I cannot repeat (I ran it for real and logged it, testing before and after only ever showed the above 1 error in each). ERROR: test_etree_parse_io_error (lxml.tests.test_io.ETreeIOTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.6/unittest/case.py", line 59, in testPartExecutor yield File "/usr/lib/python3.6/unittest/case.py", line 605, in run testMethod() File "/scratch/working/lxml-4.0.0/src/lxml/tests/test_io.py", line 276, in test_etree_parse_io_error dn = tempfile.mkdtemp(prefix=dirnameRU) File "/usr/lib/python3.6/tempfile.py", line 368, in mkdtemp _os.mkdir(file, 0o700) UnicodeEncodeError: 'ascii' codec can't encode characters in position 5-18: ordinal not in range(128) and ERROR: test_etree_parse_io_error (lxml.tests.test_io.ElementTreeIOTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.6/unittest/case.py", line 59, in testPartExecutor yield File "/usr/lib/python3.6/unittest/case.py", line 605, in run testMethod() File "/scratch/working/lxml-4.0.0/src/lxml/tests/test_io.py", line 276, in test_etree_parse_io_error dn = tempfile.mkdtemp(prefix=dirnameRU) File "/usr/lib/python3.6/tempfile.py", line 368, in mkdtemp _os.mkdir(file, 0o700) UnicodeEncodeError: 'ascii' codec can't encode characters in position 5-18: ordinal not in range(128) and similar in 2.7 - unless I'm missing something, these 2 error reports are identical. To be clear, in that run python2 ran 1787 tests and reported FAILED (errors=3) while python3 ran 1800 tests with a similar count of errors. ĸen -- Truth, in front of her huge walk-in wardrobe, selected black leather boots with stiletto heels for such a barefaced truth. - Unseen Academicals

3 4

Documentation
by Gregory 17 Sep '17

17 Sep '17

A little while ago I though it would be fun to use Sphinx and a theme derived from "Read the Docs" to build the lxml docs, and it worked rather well I think. I put it on github: https://gregoryvigotorres.github.io/lxml_docs/index.html The way I did it breaks some things and has some issues, but I think it can be made to work. Is there any interest in pursuing this further and possibly replacing the endearing though somewhat dated looking and eccentric current documentation? Gregory

4 3

Re: [lxml] Fwd: Reg Libxml2 upgrade
by Christian Fersch 17 Sep '17

17 Sep '17

Hi Swarnalatha, the previous releases were build with 2.9.3, due to 2.9.4 having a bug regarding schema validation. Unfortunately, 2.9.3 has it's own bugs, too. You can see some previous discussion here: https://bugs.launchpad.net/lxml/+bug/1706375 That discussion was then moved to this mailing list – unfortunately all of its web-accessible archives seem to be broken :(. If you have a newsreader, you can have a look at nntp://news.gmane.org/gmane.comp.python.lxml.devel. The thread title is "Should (Linux) binary wheels include libxml2/libxslt". If you need a quick fix, you can just compile lxml yourself with your own version of libxml2 – that's what we ended up doing. On 29.08.17 8:43:15, "Swarnalatha Kannan" <swarna298(a)gmail.com> wrote: > >Team, > >We are using lxml (lxml-3.3.4.win-amd64-py2.7) in my project which gets >shipped with libxml2 2.9.0. We would like to upgrade libxml2 to 2.9.4 >to mitigate security vulnerabilities (CVE-2016-4448 > >). For this upgrade we have to update lxml which has 2.9.4 libxml2. But >latest lxml for 2.7 python 64 bit windows is having 2.9.3. Let me know >how to solve the issue? Thanks. > >Python version used: python-2.7.12.amd64.msi > >

4 8

Fwd: Fwd: Reg Libxml2 upgrade
by Swarnalatha Kannan 06 Sep '17

06 Sep '17

Since Python 2 and 3 are not compatible, we are little worried on doing the code change now. Is there any easy way to do the same? Also i could only see lxml 3.8.0 available only till python 3.4 for windows and it has only 2.9.3 libxml2 version and we required latest libxml2 which is 2.9.4. Kindly let us know if lxml windows installer is available for python 3.6. Also the steps to do the migration from Python 2 to Python3 without much code changes? Thanks. [image: Inline image 1] python installer that we use: pywin32-218.win-amd64-py2.7.exe lxml --> lxml-3.3.4.win-amd64-py2.7.exe (it comes with 2.9.0 version of libxml2) Regards, Swarna On Tue, Sep 5, 2017 at 11:59 PM, Stefan Behnel <stefan_ml(a)behnel.de> wrote: > Hi, > > Swarnalatha Kannan schrieb am 05.09.2017 um 17:55: > > I would like to know if security vulnerabilities on libxml2 holds good > for > > lxml. We are using lxml-3.3.4.win-amd64-py2.7.exe in our project. After > > installation, I could only see .h files inside libxml fodler of lxml. > > (C:\Python27\Lib\site-packages\lxml\includes\libxml). I would like to > know > > about the implementations of these header file functions. Because most of > > the vulnerabilities mentioned are part of .c files. Kindly get back. > > The header files are only there for external code that wants to compile > against lxml. What you are interested in is the libxml2 version that the > Windows binaries include. You can look that up in lxml itself: > > print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION)) > print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION)) > print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION)) > print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION)) > print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION)) > > lxml 3.3.4 is hugely old and probably comes with some 2.7.x version of > libxml2, maybe 2.7.8. These old versions have bugs that most likely include > security relevant ones. > > I recomment switching to Python 3.5 or 3.6. The corresponding lxml wheels > for Windows come with more recent library versions. > > Stefan > _________________________________________________________________ > Mailing list for the lxml Python XML toolkit - http://lxml.de/ > lxml(a)lxml.de > https://mailman-mail5.webfaction.com/listinfo/lxml >

1 0

lxml - The Python XML Toolkit September 2017