Thank you for looking into this! On Fri, Apr 21, 2017 at 9:37 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
This is a bit of a known quirk. It happens because the incremental parser sees the closing tag, and potentially but not necessarily more of the following content, and then yields the end event for the tag without making sure that the tail string is also completely parsed already.
Is this behavior documented somewhere? I found it pretty surprising, and searched but could not find anything about it. Also, as a user, is there a way to work around this quirk?
This could be fixed by making sure that the parser receives the complete tail text data before generating the end event for the element. But that means that there will be extreme cases where it needs to wait for a lot more data than currently that simply isn't going to be seen by anyone, especially because tail text is entirely irrelevant for many use cases.
I'll defer to your judgment here, because you have clearly thought about this problem more than I have, but I would love to understand more. Missing data feels like a big deal to me - why is tail text irrelevant for many use cases? In particular, is it often both irrelevant and present in the document to be parsed?
It could be argued that it's often relevant for HTML parsing, but introducing such a difference between the HTML and XML parsers would easily produce bugs on user side - see your way of shadowing the problem by passing slightly different data.
I'm not sure I follow; can you say more?
The implementation that creates parse events from SAX events is in saxparser.pxi. You can take a look at the spots where _pushSaxEndEvent() is called, but the whole machinery is a bit complex overall, e.g. because it allows matching only specific tag names (which shouldn't impact the "tail finished" detection). Postponing the end event creation might not be all that trivial.
I looked at that file, but it's not clear to me how or why the tail is sometimes missing, and I'm afraid I don't have much bandwidth to work on this. I've opened a bug report to track it: https://bugs.launchpad.net/lxml/+bug/1684273 . Thanks, Jason