Jason Owen schrieb am 17.04.2017 um 05:59:
I have encountered something that feels like a bug in lxml.
Given this minimal parser: https://gist.github.com/jasonaowen/ 2c98ebe9515918eebc86eeeb3706e6cf and this HTML file: https://innovation.isotropic.org/gamelog/201307/18/game- 20130718-235322-1acbb5dc.html
I expect running the parser on the HTML file to include, somewhere in its output, the string "splays their blue cards left", as it is present at line 289 in the HTML file, but it does not. Changing the file by deleting things before that line seems to avoid triggering this issue and causes this to be printed, which is the desired behavior:
('span', {'class': 'age e'}, '4', '.\n... Nnastya splays their blue cards left.\n')
This is a bit of a known quirk. It happens because the incremental parser sees the closing tag, and potentially but not necessarily more of the following content, and then yields the end event for the tag without making sure that the tail string is also completely parsed already. This could be fixed by making sure that the parser receives the complete tail text data before generating the end event for the element. But that means that there will be extreme cases where it needs to wait for a lot more data than currently that simply isn't going to be seen by anyone, especially because tail text is entirely irrelevant for many use cases. It could be argued that it's often relevant for HTML parsing, but introducing such a difference between the HTML and XML parsers would easily produce bugs on user side - see your way of shadowing the problem by passing slightly different data. The implementation that creates parse events from SAX events is in saxparser.pxi. You can take a look at the spots where _pushSaxEndEvent() is called, but the whole machinery is a bit complex overall, e.g. because it allows matching only specific tag names (which shouldn't impact the "tail finished" detection). Postponing the end event creation might not be all that trivial. Stefan