Mailman 3 April 2017 - lxml - The Python XML Toolkit

Missing tail in iterparse
by Jason Owen 22 Apr '17

22 Apr '17

Hello, I have encountered something that feels like a bug in lxml. Given this minimal parser: https://gist.github.com/jasonaowen/ 2c98ebe9515918eebc86eeeb3706e6cf and this HTML file: https://innovation.isotropic.org/gamelog/201307/18/game- 20130718-235322-1acbb5dc.html I expect running the parser on the HTML file to include, somewhere in its output, the string "splays their blue cards left", as it is present at line 289 in the HTML file, but it does not. Changing the file by deleting things before that line seems to avoid triggering this issue and causes this to be printed, which is the desired behavior: ('span', {'class': 'age e'}, '4', '.\n... Nnastya splays their blue cards left.\n') I have reproduced this on two machines: Ubuntu 16.04.2 LTS Linux 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Python 3.5.2 LIBXML_COMPILED_VERSION (2, 9, 3) LIBXML_VERSION (2, 9, 3) LIBXSLT_COMPILED_VERSION (1, 1, 29) LIBXSLT_VERSION (1, 1, 29) LXML_VERSION (3, 7, 3, 0) (lxml installed via pip) and Debian GNU/Linux 8.7 (jessie) Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u1 (2017-02-22) x86_64 GNU/Linux Python 3.4.2 LIBXML_COMPILED_VERSION (2, 9, 1) LIBXML_VERSION (2, 9, 1) LIBXSLT_COMPILED_VERSION (1, 1, 28) LIBXSLT_VERSION (1, 1, 28) LXML_VERSION (3, 7, 3, 0) (lxml installed via pip) I also reproduced on the Debian system with the system-installed python3-lxml package, version 3.4.0-1: LIBXML_COMPILED_VERSION (2, 9, 1) LIBXML_VERSION (2, 9, 1) LIBXSLT_COMPILED_VERSION (1, 1, 28) LIBXSLT_VERSION (1, 1, 28) LXML_VERSION (3, 4, 0, 0) Is this a bug in lxml, or am I using it wrong? Thanks, Jason

2 3

Problems with v3.7.3 for Python 2.7 and apache wsgi_mod
by EricWentz＠Eaton.com 21 Apr '17

21 Apr '17

Hello Version 3.7.3 crashes my apache instance with a segmentation fault - does not crash with 3.7.0, but there are missing features. Has anybody else seen this behavior? Is there currently a bug submitted for this? Thanks Eric W. Eric Wentz Sr. Threat Intelligence Analyst Eaton Mail Code 8T 1000 Eaton Boulevard Cleveland, Ohio 44122 tel: +1 440 523-2220 mobile: +1 216 533-6827 ericwentz(a)eaton.com<mailto:ericwentz@eaton.com> www.eaton.com<http://www.eaton.com/> [image001] The information in this message is proprietary and/or confidential and is intended only for the use of the individual(s) named above. This message should not be forwarded without permission from the author. If this message is received in error or otherwise by someone other than the addressee, please contact the author at the above number and return the message or destroy it.

1 0

"outdenting" pretty_print
by Martin Mueller 21 Apr '17

21 Apr '17

Is there a way of modifying pretty_print output so that it prints the lowest element nodes without any indentation and every other element name with just one tab stop, like the following: <l xml:id="A15310-e90"> <w lemma="to" pos="acp-p" reg="TO" xml:id="A15310-000450">TO</w> <w lemma="call" pos="vvi" reg="call" xml:id="A15310-000460">call</w> <w lemma="Clio" pos="n1-nn" reg="Clio" xml:id="A15310-000470">Clio</w> <w lemma="my" pos="po" reg="my" xml:id="A15310-000480">my</w> <w lemma="dulness" pos="n1" reg="dulness" xml:id="A15310-000490">dulnesse</w> <w lemma="to" pos="acp-cs" reg="to" xml:id="A15310-000500">to</w> <w lemma="redress" pos="vvi" reg="redress" xml:id="A15310-000510">redresse</w> </l> In my project—an enrichment of the 60,000 English books printed before 1700 and transcribed by the Text Creation partnership—every word or punctuation mark is wrapped in a <w> or <pc> element. Since the texts are long and the encoding is very verbose, there isn’t much point to using tab stops or spaces to express the position of an element in the hierarchy. On the other hand, there is value in having no indentation for <w> elements because they will in most cases fit in one line. It’s often necessary to look at output on a screen, and for the purpose of this project, this form of pretty-printing is easier to read. It also saves a lot of space: output formatted with lxml pretty_ print takes up 88 GB. With my form of “outdenting” it’s 67GB. Space is cheap, but not that cheap. So I need a formatting device with the following pseudocode: If element.tag is ‘w’ or ‘pc’: begin a new line without indentation else: indent by one tab stop I know how to write a postprocessing script that treats the pretty_printed file as a plain text file and use regulars expression to replace ‘\n\s*(<w|<pc)’ with ‘\n$1’. There must be a better way than this kludge, and I’ll be grateful for help.

3 2

problems with parent.remove(element)
by Martin Mueller 21 Apr '17

21 Apr '17

I’m puzzled by a problem to which I found a solution, but I don’t understand it. You can loop through text specifying different elements, as in for tree.iter(tei + ‘w’, tei + ‘pc’): If this: do ‘x’ If that: do ‘y’ But this doesn’t seem to work if the action involved is of the parent.remove(x) kind. In my work I make minor changes in linguistically annotated texts. In the first example below , two <w> elements are moved into one. In the other, a w hi sequence is turned into a single <w>. The result of these and similar operations is a garbage collection problem: there are redundant elements that you want to get rid of: <w lemma="himself" part="I" ">him</w> => <w lemma="himself"> =him selfe</w> <w lemma="himself" part="F" ">selfe</w> => DELETE <w y</w> => <w rendition=”#sup-right,1>ye</w> <hi rend="sup" > => DELETE <w >e</w> </hi> How to do this? Apparently, you can’t do it for each element at the point when it becomes redundant. if w.get(part=”I”): if w.getnext().get(part=”F”): w2 = w.getnext() w.text = w.text + w2.text w.attrib.pop(‘part’) parent= w2.getparent() parent.remove(w2) Code like this works for the first instance, but then stops. I really don’t understand why, but I know it is so. and has something with the behavior of lists. You can ‘pop’ attributes, but you can’t ‘pop’ list items in quite the same way. I’ve used a workaround in which I mark every redundant element with an n=’DELETE’ attribute. Then I loop through the text with a command like for hi in tree.iter(tei + ‘hi’): parent = hi.getparent() parent.remove(hi) This works for a single element. But in my work I generate redundant w, pc, and hi elements for each text. I tried to get rid of those with a command for element in tree.iter(tei + ‘w’, tei +’pc’, tei + ‘hi’): if element.get(‘n’) == ‘DELETE’: parent = element.getparent() parent.remove(element) This does not work. On the other hand, things do work if I iterate through the successively, with just one element for each iteration. That’s not a big deal, though I have 50,000 texts and some of them are very long. Is there a better way of handling this problem from the beginning?

2 2

Re: [lxml] Gracefully handling invalid XML characters when parsing documents
by Michal Petrucha 06 Apr '17

06 Apr '17

On Wed, Apr 05, 2017 at 11:02:14AM -0700, Peter Van Epp wrote: > On Wed, Apr 05, 2017 at 02:47:26PM +0200, Michal Petrucha wrote: > > Hi everyone, > > > > I'm finding myself in a situation where I need to process XML > > documents that aren't entirely valid, because they contain ASCII > > control characters, such as the vertical tab (), which are not > > allowed by the specification of XML1.0. The invalid characters > > themselves are not important to me at all, and I'm fine with just > > throwing them away from the input stream, and moving on. Other than > > those characters, the XML documents are valid. > > While I'm not at all an xml or lxml expert, is it not possible to run > the file through a preprocessor script (perl, python, sed or what ever you have > on your platform) and strip out the invalid charactes before the parser sees > them to keep the parser happy? This has crossed my mind, and it may be what I'll try to do if I don't discover a better solution, but it makes me uneasy, because there are different ways the same character can be encoded in an XML document (, , the raw character itself are just the ones I can immediately think of, but I suspect there's more magic you can do with XML entities). This is why I was hoping to handle this with a proper XML parser rather than just a regular expression. Cheers, Michal

1 0

Gracefully handling invalid XML characters when parsing documents
by Michal Petrucha 05 Apr '17

05 Apr '17

Hi everyone, I'm finding myself in a situation where I need to process XML documents that aren't entirely valid, because they contain ASCII control characters, such as the vertical tab (), which are not allowed by the specification of XML1.0. The invalid characters themselves are not important to me at all, and I'm fine with just throwing them away from the input stream, and moving on. Other than those characters, the XML documents are valid. When I try to parse such an XML document with lxml with its default settings, obviously, I get an error:: lxml.etree.XMLSyntaxError: xmlParseCharRef: invalid xmlChar value 11, line 6, column 17 In order to silence this error, and try to recover from it, I can use a custom parser with the “recover” option. This does get the job done in the sense that the error no longer gets raised, but it has significant side effects. Apparently, after the first invalid XML character is encountered, from that point on, the parser ignores *all* XML entities in the rest of the document. Here's a brief code sample that demonstrates the problem:: from lxml import etree broken_xml = """<?xml version="1.0"?> <root> <child> <something> & </child> <child></child> <child> <something> & </child> </root> """ recovering_parser = etree.XMLParser(recover=True) broken_tree = etree.fromstring(broken_xml, parser=recovering_parser) print(etree.tostring(broken_tree, pretty_print=True, encoding="unicode")) The output I get from this is the following:: <root> <child> <something> & </child> <child/> <child> something </child> </root> I've scoured the docs for anything that would give me more fine-grained control over what errors should be handled, and how, but I haven't found anything useful. What I need is a tree that contains all XML entities properly, and I don't really care about the invalid control characters. The use case is that we're getting these invalid XML documents from MS Exchange, where some emails happen to contain control characters in their bodies, and ignoring all remaining entities means that all HTML bodies turn into garbage. Does anyone have any pointers how I can get this to work? Cheers, Michal

1 0

meta element not being added to html output
by Bob Kline 03 Apr '17

03 Apr '17

According to https://www.w3.org/TR/xslt#section-HTML-Output-Method : "If there is a HEAD element, then the html output method should add a META element immediately after the start-tag of the HEAD element specifying the character encoding actually used. For example, <HEAD> <META http-equiv="Content-Type" content="text/html; charset=EUC-JP"> That's not happening with the lxml xslt processor. Is this a known bug? Thanks. -- Bob Kline http://www.rksystems.com mailto:bkline@rksystems.com

2 3

How to match schema validator information to nodes in parsed tree
by Bob Kline 02 Apr '17

02 Apr '17

When you pass a document tree to the XML Schema validator, the error logs identify the line and column numbers where each error is found, but the error object does not contain a reference to the Element object where the node was found. The Element objects in the parsed document tree have the line numbers for the elements, but not the column positions. The line numbers would not be sufficient for identifying which Element an error occurred in if there are multiple elements on the same line (a common occurrence). What I'm trying to do is insert an attribute into each element with a validation error, so that when the document is sent back to the editor, the user can invoke scripted commands to navigate through the errors. Is there any way to get the schema validator to provide a reference to the element where each error was found, or to get an element to tell the position on the line from which it was parsed? I realize I could use the line and column information that I get back from the validation error log to find and manipulate the string for the serialized document by hand (working backwards so I'm not changing the positions of the elements I haven't processed yet), but the reason we use real XML parsers is so we don't have to do such ugly, fragile, error-prone stuff, right? Thanks! -- Bob Kline http://www.rksystems.com mailto:bkline@rksystems.com

1 1