[Twisted-Python] trial.unitest-specific segfault with lxml
This is an odd one; I have a simple test using the incremental xmlfile() feature of lxml, that crashes *only* if the test case inherits from trial.unittest.TestCase; it works if inheriting from unittest.TestCase. SSCCE - well, hopefully correct - here: https://gist.github.com/philmayers/387597c7407ab98f159426cea5f44a69 With lxml 3.8.0 (tried both manylinux1 wheel from PyPI and locally-compiled) and a debug python, I get: $ bin/python2-debug -m twisted.trial test_repro.py test_repro Test test_one ... [OK] test_two ... [OK] ------------------------------------------------------------------------------- Ran 2 tests in 0.032s PASSED (successes=2) python2-debug: /builddir/build/BUILD/Python-2.7.13/Python/getargs.c:229: vgetargs1: Assertion `compat || (args != (PyObject*)NULL)' failed. Aborted (core dumped) Obviously lxml is a pretty big chunk of Cython, but the trial-specific nature of the crash has me curious which of the two is at fault. I have tried to debug it, and it basically seems to end up somewhere inside the Cython-generated code with an argument to a python function that is NULL when it shouldn't be. Any ideas?
On Fri, 18 Aug 2017 at 00:47 Phil Mayers <p.mayers@imperial.ac.uk> wrote:
SSCCE - well, hopefully correct - here:
https://gist.github.com/philmayers/387597c7407ab98f159426cea5f44a69
Note that I don't think you're allowed to catch GeneratorExit like this; normally doing so will crash your program with a RuntimeError after the generator fails to exit (possibly masked here by the generator living until the end of the program). However, I was able to reproduce the crash even after removing the try/except, so I don't think this issue is related. Given that the crash only occurs at exit, I suspect this has something to do with finalizers running at a bad time during interpreter shutdown, but I don't have time at the moment to look into this deeply (I just spotted the GeneratorExit issue and wanted to check if that was the problem). I suspect this is an lxml or Cython bug, ultimately, but changing the test base class changes some ordering of events arbitrarily resulting in this heisencrash. Alternatively, it might have something to do with logging.
On 18/08/2017 07:05, Tristan Seligmann wrote:
Note that I don't think you're allowed to catch GeneratorExit like this;
Interesting, I cribbed that from the lxml docs. Will look into it, thanks for the pointer.
You're spot on - I have managed to trim the test-case down to a plain python script, no twisted/trial. It seems to crash the interpreter on exit any time >1 xmlfile is live at that point, but it does not crash if you del the objects in certain orders - trial was obviously just revealing it. Thanks for the attention and apologies for the noise, seems this is nothing to do with Twisted. Regards, Phil
On Fri, 18 Aug 2017 at 00:47 Phil Mayers <p.mayers@imperial.ac.uk> wrote:
SSCCE - well, hopefully correct - here:
https://gist.github.com/philmayers/387597c7407ab98f159426cea5f44a69
Note that I don't think you're allowed to catch GeneratorExit like this; normally doing so will crash your program with a RuntimeError after the generator fails to exit (possibly masked here by the generator living until the end of the program). However, I was able to reproduce the crash even after removing the try/except, so I don't think this issue is related. Given that the crash only occurs at exit, I suspect this has something to do with finalizers running at a bad time during interpreter shutdown, but I don't have time at the moment to look into this deeply (I just spotted the GeneratorExit issue and wanted to check if that was the problem). I suspect this is an lxml or Cython bug, ultimately, but changing the test base class changes some ordering of events arbitrarily resulting in this heisencrash. Alternatively, it might have something to do with logging.
On 18/08/2017 07:05, Tristan Seligmann wrote:
Note that I don't think you're allowed to catch GeneratorExit like this;
Interesting, I cribbed that from the lxml docs. Will look into it, thanks for the pointer.
You're spot on - I have managed to trim the test-case down to a plain python script, no twisted/trial. It seems to crash the interpreter on exit any time >1 xmlfile is live at that point, but it does not crash if you del the objects in certain orders - trial was obviously just revealing it. Thanks for the attention and apologies for the noise, seems this is nothing to do with Twisted. Regards, Phil
participants (2)
-
Phil Mayers -
Tristan Seligmann