Mailman 3 December 2016 - lxml - The Python XML Toolkit

is there a way of getting around endless repetitions of "is not None" ?
by Martin Mueller 22 Dec '16

22 Dec '16

This may be a primitive Python question rather than an lxml question, but I’ll ask it anyhow. If I want to find out whether the attribute of some element has a particular value, I can’t just say If element.get(‘rend’) == ‘hi ‘ This may throw an “is None” exception. I can get around it by saying If element.get(‘rend’) is not None and element.get(‘rend’) == ‘hi’ Is there some way of getting around these tedious “is not None” explicitations or do I always have to say it. I know that computers are very literal animals.

3 5

Re: [lxml] Potential memory leak found in lxml
by Benoit Bernard 12 Dec '16

12 Dec '16

In-Reply-To=<4b9d9f1a-f836-f369-3420-f2afad10cff3(a)behnel.de> Thanks Stefan for your detailed explanations. This makes sense now. And this corresponds to what I found by attaching a debugger to the code. I wasn't sure whether all those names and attributes being added to a global dictionary was normal or not, and especially why umdh wasn't always reporting allocation numbers that I expected. But I was most likely not counting things properly. "Usually not something to worry about as new names quickly become less frequent, as long as the number of different document types is bounded." Well, I'm using lxml for a web crawler, so over a long enough period of time, this would perhaps be a problem. But I haven't crawled long enough yet to know. If it ever becomes problematic, I'll take a look at the _ParserDictionaryContext class, as you suggested. Thanks again!

1 0

a suggestion for error reporting
by Martin Mueller 11 Dec '16

11 Dec '16

In looping through a set of files I got this error report: Traceback (most recent call last): File "/Users/martinmueller/Dropbox/PycharmProjects/earlyprint/process_eebo_sample/try3.py", line 55, in <module> tree= etree.parse(filename, parser) File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:79841) File "src/lxml/parser.pxi", line 1793, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:116175) File "src/lxml/parser.pxi", line 1819, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:116525) File "src/lxml/parser.pxi", line 1723, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:115413) File "src/lxml/parser.pxi", line 1126, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:110110) File "src/lxml/parser.pxi", line 584, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:103584) File "src/lxml/parser.pxi", line 694, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:105238) File "src/lxml/parser.pxi", line 624, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:104147) lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: TEI line 2 and TanaEI, line 34853, column 11 It was clear from the report what was wrong, but not in what file it happened. I ran the script again asking it to print the filename before doing anything else, and this time the error report came right after the printing of the file name Would it be difficult to make the filename part of the error report so that in this case it would read A03429.xml Traceback (most recent call last): File "/Users/martinmueller/Dropbox/PycharmProjects/earlyprint/process_eebo_sample/try3.py", line 55, in <module> tree= etree.parse(filename, parser) File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:79841) File "src/lxml/parser.pxi", line 1793, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:116175) File "src/lxml/parser.pxi", line 1819, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:116525) File "src/lxml/parser.pxi", line 1723, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:115413) File "src/lxml/parser.pxi", line 1126, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:110110) File "src/lxml/parser.pxi", line 584, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:103584) File "src/lxml/parser.pxi", line 694, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:105238) File "src/lxml/parser.pxi", line 624, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:104147) lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: TEI line 2 and TanaEI, line 34853, column 11 It’s easy enough for me to log each file. But if yo work interactively with a set of files, you might look for five or fifty cases in a set of 5,000 files and if every filename gets printed, it clutters up the screen. Not a big thing, but one of those little things that make life easier. I recently switched from MySQL and noticed right away that the error reporting was more specific. Which is very useful for a Patzer liker myself.

1 0

Potential memory leak found in lxml
by Benoit Bernard 10 Dec '16

10 Dec '16

Hi all, Following my previous question <https://mailman-mail5.webfaction.com/pipermail/lxml/2016-November/007780.ht…>, I'm getting another potential memory leak when parsing multiple HTML pages in sequence (over a long time). Here's the allocation callstack identified by umdh: + 4064 ( 4096 - 32) 128 allocs BackTrace7F965A0 + 127 ( 128 - 1) BackTrace7F965A0 allocations ntdll! ?? ::FNODOBFM::`string'+000153EB ucrtbase!malloc+00000031 etree!xmlDictLookup+0000025E (c:\tmp\libxml2-win-binaries2\libxml2\dict.c, 933) etree!htmlParseName+000000C3 (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 2482) etree!htmlParseEntityRef+00000054 (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 2699) etree!htmlParseHTMLAttribute+0000012E (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 2602) etree!htmlParseAttValue+0000002B (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 2746) etree!htmlParseStartTag+0000039D (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 3834) etree!htmlParseElementInternal+00000069 (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 4467) etree!htmlParseContentInternal+000003D3 (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 4652) etree!htmlParseDocument+000002A2 (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 4818) etree!htmlDoRead+00000094 (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 6786) etree!htmlCtxtReadMemory+00000093 (c:\tmp\libxml2-win-binaries2\libxml2\htmlparser.c, 7072) etree!__pyx_f_4lxml_5etree_11_BaseParser__parseUnicodeDoc+0000028A (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 109222) etree!__pyx_f_4lxml_5etree__parseDoc+0000041F (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 115220) etree!__pyx_f_4lxml_5etree__parseMemoryDocument+000000E6 (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 116674) etree!__pyx_pf_4lxml_5etree_22fromstring+00000086 (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 77737) etree!__pyx_pw_4lxml_5etree_23fromstring+00000294 (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 77687) python33!PyCFunction_Call+000000F3 (c:\users\martin\33.amd64\python\objects\methodobject.c, 84) python33!PyObject_Call+00000061 (c:\users\martin\33.amd64\python\objects\abstract.c, 2036) python33!ext_do_call+00000295 (c:\users\martin\33.amd64\python\python\ceval.c, 4381) python33!PyEval_EvalFrameEx+00002041 (c:\users\martin\33.amd64\python\python\ceval.c, 2723) python33!PyEval_EvalCodeEx+0000065C (c:\users\martin\33.amd64\python\python\ceval.c, 3436) python33!function_call+0000015D (c:\users\martin\33.amd64\python\objects\funcobject.c, 639) python33!PyObject_Call+00000061 (c:\users\martin\33.amd64\python\objects\abstract.c, 2036) python33!ext_do_call+00000295 (c:\users\martin\33.amd64\python\python\ceval.c, 4381) python33!PyEval_EvalFrameEx+00002041 (c:\users\martin\33.amd64\python\python\ceval.c, 2723) python33!PyEval_EvalCodeEx+0000065C (c:\users\martin\33.amd64\python\python\ceval.c, 3436) python33!fast_function+0000014D (c:\users\martin\33.amd64\python\python\ceval.c, 4168) python33!call_function+00000339 (c:\users\martin\33.amd64\python\python\ceval.c, 4088) python33!PyEval_EvalFrameEx+00001F98 (c:\users\martin\33.amd64\python\python\ceval.c, 2681) python33!fast_function+000000D8 (c:\users\martin\33.amd64\python\python\ceval.c, 4155) This seems related to character entity references used in HTML attributes. Under these circumstances, in some cases, the xmlDictLookup() function seems to never release some values. But so far, I've been unable to identify the exact cause, or the exact set of HTML pages causing this issue. Anybody has an idea? FYI: Python : sys.version_info(major=3, minor=3, micro=5, releaselevel='final', serial=0) lxml.etree : (3, 6, 4, 0) libxml used : (2, 9, 4) libxml compiled : (2, 9, 4) libxslt used : (1, 1, 29) libxslt compiled : (1, 1, 29) Thanks!

2 1

lxml 3.7.0 released
by Stefan Behnel 10 Dec '16

10 Dec '16

Hi all, I just released lxml 3.7.0 with several new features. Most changes in this release came from pull requests. Thanks to everyone who contributed. The documentation is here: http://lxml.de/ Download: http://lxml.de/files/lxml-3.7.0.tgz Signature: http://lxml.de/files/lxml-3.7.0.tgz.asc Changelog: http://lxml.de/3.7/changes-3.7.0.html Github: https://github.com/lxml/lxml/commit/826ca604eccdf6f325c1c161e73132e765bb3855 This release was built using Cython 0.25.2. If you are interested in commercial support or customisations for the lxml package, please contact me directly. Have fun, Stefan 3.7.0 (2016-12-10) ================== Features added -------------- * GH#217: ``XMLSyntaxError`` now behaves more like its ``SyntaxError`` baseclass. Patch by Philipp A. * GH#216: ``HTMLParser()`` now supports the same ``collect_ids`` parameter as ``XMLParser()``. Patch by Burak Arslan. * GH#210: Allow specifying a serialisation method in ``xmlfile.write()``. Patch by Burak Arslan. * GH#203: New option ``default_doctype`` in ``HTMLParser`` that allows disabling the automatic doctype creation. Patch by Shadab Zafar. * GH#201: Calling the method ``.set('attrname')`` without value argument (or ``None``) on HTML elements creates an attribute without value that serialises like ``<div attrname></div>``. Patch by Daniel Holth. * GH#197: Ignore form input fields in ``form_values()`` when they are marked as ``disabled`` in HTML. Patch by Kristian Klemon. Bugs fixed ---------- * GH#206: File name and line number were missing from XSLT error messages. Patch by Marcus Brinkmann. Other changes ------------- * Log entries no longer allow anything but plain string objects as message text and file name. * ``zlib`` is included in the list of statically built libraries.

1 0

Re: [lxml] Memory leak when parsing XML files in sequence?
by Benoit Bernard 04 Dec '16

04 Dec '16

In-Reply-To=<4EA045E4.9020508(a)anaproy.nl> Has there been any advancements regarding this memory leak? I built the newest version of lxml (as well as its dependencies) and the problem is still there. I was able to track it down using umdh on Windows: etree!xmlDictLookup+0000025E (c:\tmp\libxml2-win-binaries\libxml2\dict.c, 933) etree!xmlHashAddEntry3+00000053 (c:\tmp\libxml2-win-binaries\libxml2\hash.c, 532) etree!xmlHashAddEntry+00000014 (c:\tmp\libxml2-win-binaries\libxml2\hash.c, 377) etree!xmlAddID+0000011D (c:\tmp\libxml2-win-binaries\libxml2\valid.c, 2632) etree!xmlSAX2AttributeInternal+0000078A (c:\tmp\libxml2-win-binaries\libxml2\sax2.c, 1411) etree!xmlSAX2StartElement+000002AE (c:\tmp\libxml2-win-binaries\libxml2\sax2.c, 1743) etree!htmlParseStartTag+00000579 (c:\tmp\libxml2-win-binaries\libxml2\htmlparser.c, 3926) etree!htmlParseElementInternal+00000069 (c:\tmp\libxml2-win-binaries\libxml2\htmlparser.c, 4467) etree!htmlParseContentInternal+000003D3 (c:\tmp\libxml2-win-binaries\libxml2\htmlparser.c, 4652) etree!htmlParseDocument+000002A2 (c:\tmp\libxml2-win-binaries\libxml2\htmlparser.c, 4818) etree!htmlDoRead+00000094 (c:\tmp\libxml2-win-binaries\libxml2\htmlparser.c, 6786) etree!htmlCtxtReadMemory+00000093 (c:\tmp\libxml2-win-binaries\libxml2\htmlparser.c, 7072) etree!__pyx_f_4lxml_5etree_11_BaseParser__parseUnicodeDoc+0000028A (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 109222) etree!__pyx_f_4lxml_5etree__parseDoc+0000041F (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 115220) etree!__pyx_f_4lxml_5etree__parseMemoryDocument+000000E6 (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 116674) etree!__pyx_pf_4lxml_5etree_22fromstring+00000086 (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 77737) etree!__pyx_pw_4lxml_5etree_23fromstring+00000294 (c:\tmp\lxml-3.6.4\src\lxml\lxml.etree.c, 77687) python33!PyCFunction_Call+000000F3 (c:\users\martin\33.amd64\python\objects\methodobject.c, 84) python33!PyObject_Call+00000061 (c:\users\martin\33.amd64\python\objects\abstract.c, 2036) python33!ext_do_call+00000295 (c:\users\martin\33.amd64\python\python\ceval.c, 4381) python33!PyEval_EvalFrameEx+00002041 (c:\users\martin\33.amd64\python\python\ceval.c, 2723) python33!PyEval_EvalCodeEx+0000065C (c:\users\martin\33.amd64\python\python\ceval.c, 3436) python33!function_call+0000015D (c:\users\martin\33.amd64\python\objects\funcobject.c, 639) python33!PyObject_Call+00000061 (c:\users\martin\33.amd64\python\objects\abstract.c, 2036) python33!ext_do_call+00000295 (c:\users\martin\33.amd64\python\python\ceval.c, 4381) python33!PyEval_EvalFrameEx+00002041 (c:\users\martin\33.amd64\python\python\ceval.c, 2723) python33!PyEval_EvalCodeEx+0000065C (c:\users\martin\33.amd64\python\python\ceval.c, 3436) python33!fast_function+0000014D (c:\users\martin\33.amd64\python\python\ceval.c, 4168) python33!call_function+00000339 (c:\users\martin\33.amd64\python\python\ceval.c, 4088) python33!PyEval_EvalFrameEx+00001F98 (c:\users\martin\33.amd64\python\python\ceval.c, 2681) For reference, here are my version numbers: Python : sys.version_info(major=3, minor=3, micro=5, releaselevel='final', serial=0) lxml.etree : (3, 6, 4, 0) libxml used : (2, 9, 4) libxml compiled : (2, 9, 4) libxslt used : (1, 1, 29) libxslt compiled : (1, 1, 29) Should I open a new bug? Thanks! Benoit Bernard https://benbernardblog.com

3 3