lxml - The Python XML Toolkit January 2019

lxml@python.org

6 participants
9 discussions

[lxml-dev] Reparenting a node
by Lawrence Oluyede 30 Jan '23

30 Jan '23

I have a doc A and a doc B, I'd like to put a node extracted from A in the document B but I always get a ValueError: ValueError: Element is not a child of this node. I didn't find any "setparent" in the API. How can I do this? -- Lawrence, oluyede.org - neropercaso.it "It is difficult to get a man to understand something when his salary depends on not understanding it" - Upton Sinclair

3 2

[lxml-dev] Checking whether a node is a comment/element
by Geoffrey Sneddon 11 Jan '23

11 Jan '23

Hi, What's the best way to check whether a given node is a comment or an element? For the former, I'm currently using isinstance(node, etree._Comment), which is rather obviously sub-optimal. -- Geoffrey Sneddon <http://gsnedders.com/>

4 4

[lxml-dev] lxml 2.0.5 released
by Stefan Behnel 11 Jan '23

11 Jan '23

Hi all, lxml 2.0.5 is on PyPI. This is a bug-fix-only release of the stable 2.0 series. Have fun, Stefan 2.0.5 (2008-05-01) Bugs fixed * Resolving to a filename in custom resolvers didn't work. * lxml did not honour libxslt's second error state "STOPPED", which let some XSLT errors pass silently. * Memory leak in Schematron with libxml2 >= 2.6.31.

3 4

[lxml-dev] lxml has its page on launchpad
by Stefan Behnel 20 Sep '22

20 Sep '22

Hi all, I added the lxml project to launchpad, the Ubuntu Bug-Tracker. It also has a FAQ engine and a couple of other goodies. https://launchpad.net/lxml It's easy to sign up for launchpad, BTW, no 90%-footnotes-contract. Have fun, Stefan

3 2

[lxml-dev] Building LXML Trunk
by Sidnei da Silva 30 Aug '22

30 Aug '22

Hi, I've tried to build lxml from trunk today, on Win32. Got the following error: src\lxml\etree.c(880) : error C2059: syntax error : ')' src\lxml\etree.c(881) : error C2059: syntax error : ')' src\lxml\etree.c(882) : error C2059: syntax error : ')' src\lxml\etree.c(883) : error C2059: syntax error : ')' Any clue? Smells like a Pyrex issue? -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214

4 4

Fwd: validation error message consistency across Python versions
by Tom Kralidis 15 Mar '19

15 Mar '19

Hi: I'm finding different behaviour when trying validate XML documents against lxml 4.3.0 in Python 2.7 and 3.x environments. Example validation error using 4.3.0 and Python 2.7.15, libxml2 2.9.3+dfsg1-1ubuntu0.6 (xenial) Error: Element '{http://www.opengis.net/foo}Filter': This element is not expected. Expected is one of ( {http://www.opengis.net/ogc}Filter, {http://www.opengis.net/cat/csw/2.0.2}CqlText ). (line 0) Example validation error using 4.3.0 and Python 3.4.8, libxml2 2.9.3+dfsg1-1ubuntu0.6 (xenial) Error: Element '{http://www.opengis.net/foo}Filter': This element is not expected. Expected is one of ( {http://www.opengis.net/ogc}Filter, {http://www.opengis.net/cat/csw/2.0.2}CqlText ). (<string>, line 0) As a result this causes our (strict) functional testing to fail. Any idea what can be done to keep error messages consistent? Thanks ..Tom

3 3

Can't process from string
by Ovnicraft 15 Mar '19

15 Mar '19

Hi I am trying to process this file https://github-production-repository-file-5c1aeb.s3.amazonaws.com/142435147… But I have problems with tags, it doesn't in correct form, 3th party system generate <> with lt/gt; in tags. How can i resolve this to load with fromstring() ? -- Sent from Gmail Mobile

2 1

Incremental parsing of XML
by Wietse Jacobs 28 Jan '19

28 Jan '19

Hello, I've run up against a problem and I've tried to create a "minimal" example to demonstrate. What I'm trying to achieve is to incrementally parse xml files but still use xpath expressions to extract "records" from the xml. A user would provide an xml file, a "record xpath" and a set of "column xpaths". My plan was to read the xml file incrementally by only reading "a" first child node of the root, extract records from that node, discard the node and continue with the next "first child". This works fine for most of the xml files we're importing, except when they get really big. Of course, the whole point of this exercise is to be able to parse *big* files without running out of memory. The problem I'm seeing is that *some* nodes aren't loaded correctly (they are "empty"), but I can't figure out why. python 3.6.5 lxml 4.3.0.0 libxml 2.9.9 libxslt 1.1.32 If I run the following script: ------8<----------------------------------------------------- from lxml import etree as ET def create_big_xml(file_path, node_count=1000): with ET.xmlfile(file_path, encoding='utf-8') as fh: fh.write_declaration() with fh.element('root'): for i in range(node_count): c0 = ET.Element('c0') c1 = ET.SubElement(c0, 'c1') c2 = ET.SubElement(c1, 'c2') c2.text = str(i) fh.write(c0) def iter_xml_records(file_path, record_xpath, column_xpaths): doc = ET.iterparse(file_path, events=('start', 'end')) try: _, root = next(doc) except StopIteration: return start_tag = None for event, element in doc: if event == 'start' and start_tag is None: start_tag = element.tag if event == 'end' and element.tag == start_tag: for record_node in record_xpath(root): yield [xp(record_node) for xp in column_xpaths] start_tag = None root.clear() fp = 'test-incr.xml' errors = 0 node_count = 100 while not errors: node_count *= 10 create_big_xml(fp, node_count) rec_xp = ET.XPath('./c0') col_xps = [ ET.XPath('./c1/c2/text()'), ] for i, record in enumerate(iter_xml_records(fp, rec_xp, col_xps)): if record != [[str(i)]]: errors += 1 print(node_count, "nodes:", errors, "errors") ------8<----------------------------------------------------- The output on my machine is: >>> python bug_lxml.py 1000 nodes: 0 errors 10000 nodes: 5 errors If I comment out the line "fh.write_declaration()", it takes longer to produce errors: >>> python bug_lxml.py 1000 nodes: 0 errors 10000 nodes: 0 errors 100000 nodes: 0 errors 1000000 nodes: 411 errors If your're still with me, thanks for taking the time! Is this a bug in lxml, or am I doing something that I shouldn't? Regards, Wietse

2 2

possible bug in lxml ? test_thread_error_log fail on openbsd
by Sebastien Marie 12 Jan '19

12 Jan '19

Hi, I am chasing a bug (python3.6 coredump, seems to be double-free) on python3 program of mine, which use lxml due to dependency (weboob in particular). I am still unsure if the problem is in lxml or in a dependency (libxml2 for example). I am able to reproduce the problem with the testsuite of lxml in both python2 and python3 environment, under OpenBSD. The failing test is test_thread_error_log. For simplicity I will speak only about test_thread_error_log. First, on OpenBSD, the malloc subsystem is very sensible to invalid use of malloc family function: it crashs (call abort(3)) early in order to avoid mis-use and/or possible exploitation of a bug. It could explain why the testsuite doesn't pass on OpenBSD whereas it is on other system. For good reproductibility of the problem (near 100%), I am using MALLOC_OPTIONS=S environment variable which enable malloc options for security auditing. In short, malloc(3) or free(3) will do more tests, and abort more easily on problem. See https://man.openbsd.org/malloc.3#MALLOC_OPTIONS for details. I am using lxml-3.7.0, as present on -current OpenBSD version. It seems to be compiled with standard options. $ MALLOC_OPTIONS=S python3.6 test.py -vv '.' test_thread_error_log Comparing with ElementTree 1.3.0 TESTED VERSION: 3.7.0 Python: sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0) lxml.etree: (3, 7, 0, 0) libxml used: (2, 9, 8) libxml compiled: (2, 9, 8) libxslt used: (1, 1, 32) libxslt compiled: (1, 1, 32) test_thread_error_log (lxml.tests.test_threading.ThreadingTestCase) ... python3.6(41533) in free(): chunk canary corrupted 0xf169db4b100 0x37@0x37 (double free?) Abort trap (core dumped) or $ MALLOC_OPTIONS=S python3.6 test.py -vv 'threading' test_thread_error_log test_thread_error_log (lxml.tests.test_threading.ThreadingTestCase) ... python3.6(79924) in free(): chunk is already free 0x9dccffba6c0 Abort trap (core dumped) or $ MALLOC_OPTIONS=S python3.6 test.py -vv 'threading' test_thread_error_log test_thread_error_log (lxml.tests.test_threading.ThreadingTestCase) ... python3.6(34868) in free(): chunk canary corrupted 0x1e598f5e9d80 0x37@0x37 (double free?) Abort trap (core dumped) $ gdb python3.6 python3.6.core ... Core was generated by `python3.6'. Program terminated with signal SIGABRT, Aborted. #0 thrkill () at -:3 3 -: No such file or directory. [Current thread is 1 (process 447114)] (gdb) bt #0 thrkill () at -:3 #1 0x00001e597f4d477e in _libc_abort () at /usr/src/lib/libc/stdlib/abort.c:51 #2 0x00001e597f444539 in wrterror (d=0x1e58f73c71f0, msg=0x1e597f3fe3e9 "chunk canary corrupted %p %#tx@%#zx%s") at /usr/src/lib/libc/stdlib/malloc.c:293 #3 0x00001e597f44748a in validate_canary (d=<optimized out>, ptr=<optimized out>, sz=1, allocated=<optimized out>) at /usr/src/lib/libc/stdlib/malloc.c:1025 #4 find_chunknum (d=0x0, info=<optimized out>, ptr=0x0, check=<optimized out>) at /usr/src/lib/libc/stdlib/malloc.c:1050 #5 0x00001e597f4449ec in ofree (argpool=<optimized out>, p=0x1e598f5e9d80, clear=<optimized out>, check=<optimized out>, argsz=0) at /usr/src/lib/libc/stdlib/malloc.c:1376 #6 0x00001e597f444628 in free (ptr=0x1e598f5e9d80) at /usr/src/lib/libc/stdlib/malloc.c:1433 #7 0x00001e593c9e7a71 in xmlCopyError () from /usr/local/lib/libxml2.so.16.1 #8 0x00001e593c9e76e6 in __xmlRaiseError () from /usr/local/lib/libxml2.so.16.1 #9 0x00001e593ca03123 in xmlFatalErrMsgStrIntStr () from /usr/local/lib/libxml2.so.16.1 #10 0x00001e593ca03628 in xmlParseEndTag2 () from /usr/local/lib/libxml2.so.16.1 #11 0x00001e593ca00f48 in xmlParseElement () from /usr/local/lib/libxml2.so.16.1 #12 0x00001e593ca04c4c in xmlParseDocument () from /usr/local/lib/libxml2.so.16.1 #13 0x00001e593ca0c3b7 in xmlDoRead () from /usr/local/lib/libxml2.so.16.1 #14 0x00001e5929ff77c2 in __pyx_f_4lxml_5etree_11_BaseParser__parseUnicodeDoc (__pyx_v_self=0x1e5955932cc0, __pyx_v_utext=0x1e58d710a800, __pyx_v_c_filename=0x0) at src/lxml/lxml.etree.c:110840 #15 0x00001e592a07b5a4 in __pyx_f_4lxml_5etree__parseDoc (__pyx_v_text=0x1e58d710a800, __pyx_v_filename=0x1e58baa4e198 <_Py_NoneStruct>, __pyx_v_parser=0x1e5955932cc0) at src/lxml/lxml.etree.c:116892 #16 0x00001e592a07916d in __pyx_f_4lxml_5etree__parseMemoryDocument (__pyx_v_text=0x1e58d710a800, __pyx_v_url=0x1e58baa4e198 <_Py_NoneStruct>, __pyx_v_parser=0x1e5955932cc0) at src/lxml/lxml.etree.c:118334 #17 0x00001e592a1f2f68 in __pyx_pf_4lxml_5etree_20XML (__pyx_self=0x1e58f8eff9a0, __pyx_v_text=0x1e58d710a800, __pyx_v_parser=0x1e5955932cc0, __pyx_v_base_url=0x1e58baa4e198 <_Py_NoneStruct>) at src/lxml/lxml.etree.c:78756 #18 0x00001e592a1f2c29 in __pyx_pw_4lxml_5etree_21XML (__pyx_self=0x1e58f8eff9a0, __pyx_args=0x1e58fedead48, __pyx_kwds=0x0) at src/lxml/lxml.etree.c:78642 #19 0x00001e592a214e96 in __Pyx_CyFunction_CallMethod (func=0x1e58f8eff9a0, self=0x1e58f8eff9a0, arg=0x1e58fedead48, kw=0x0) at src/lxml/lxml.etree.c:231798 #20 0x00001e592a21518a in __Pyx_CyFunction_Call (func=0x1e58f8eff9a0, arg=0x1e58fedead48, kw=0x0) at src/lxml/lxml.etree.c:231837 #21 0x00001e592a2140ee in __Pyx_CyFunction_CallAsMethod (func=0x1e58f8eff9a0, args=0x1e58fedead48, kw=0x0) at src/lxml/lxml.etree.c:231858 #22 0x00001e58ba8a8764 in _PyObject_FastCallDict () from /usr/local/lib/libpython3.6m.so.0.0 #23 0x00001e58ba98c799 in call_function () from /usr/local/lib/libpython3.6m.so.0.0 #24 0x00001e58ba986e82 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.6m.so.0.0 #25 0x00001e58ba98d15d in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.6m.so.0.0 #26 0x00001e58ba984822 in PyEval_EvalCodeEx () from /usr/local/lib/libpython3.6m.so.0.0 #27 0x00001e58ba8d92f2 in function_call () from /usr/local/lib/libpython3.6m.so.0.0 #28 0x00001e58ba8a84c6 in PyObject_Call () from /usr/local/lib/libpython3.6m.so.0.0 #29 0x00001e58ba987e52 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.6m.so.0.0 #30 0x00001e58ba98dd16 in fast_function () from /usr/local/lib/libpython3.6m.so.0.0 #31 0x00001e58ba98c7a0 in call_function () from /usr/local/lib/libpython3.6m.so.0.0 #32 0x00001e58ba986e82 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.6m.so.0.0 #33 0x00001e58ba98dd16 in fast_function () from /usr/local/lib/libpython3.6m.so.0.0 #34 0x00001e58ba98c7a0 in call_function () from /usr/local/lib/libpython3.6m.so.0.0 #35 0x00001e58ba986e82 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.6m.so.0.0 #36 0x00001e58ba98e17a in _PyFunction_FastCallDict () from /usr/local/lib/libpython3.6m.so.0.0 #37 0x00001e58ba8a86d5 in _PyObject_FastCallDict () from /usr/local/lib/libpython3.6m.so.0.0 #38 0x00001e58ba8a88a6 in _PyObject_Call_Prepend () from /usr/local/lib/libpython3.6m.so.0.0 #39 0x00001e58ba8a84c6 in PyObject_Call () from /usr/local/lib/libpython3.6m.so.0.0 #40 0x00001e58ba9de5c0 in t_bootstrap () from /usr/local/lib/libpython3.6m.so.0.0 #41 0x00001e58ba9d668d in pythread_wrapper () from /usr/local/lib/libpython3.6m.so.0.0 #42 0x00001e58c56bc96e in _rthread_start (v=0x0) at /usr/src/lib/librthread/rthread.c:96 #43 0x00001e597f4d48eb in __tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75 #44 0x0000000000000000 in ?? () (gdb) Any advice to ensure if the bug is in lxml or libxml2 would be welcome... Thanks. -- Sebastien Marie

1 1

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

lxml - The Python XML Toolkit January 2019