[issue17969] multiprocessing crash on exit

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Richard, I'll review implement your change.  It is a bit tricky to test this, since I can only tell after a few days if the particular (rare) problem has been fixed.  The crash is a rare one but consistently happens with some probability we use multiprocessing to perform certain batch processing.

About regressions:  The term means that problems, previously fixed, become broken again.  All fixes should reasonably have appropriate tests in the regression test suite.

I don't know what particular "regressions" you have been battling since 2.7.4.  I hope they are all testable by now.
I only know that there is a exit problem with multiprocessing that statistically happens and is problemematic.  It caused use to stop using multiprocessing and parallel jobs until I could diagnose the problem.  I have suggested several fixes to this particular problem.

I have two favorites, which would also help in the general case:
1) calling sys.exc_clear() at the beginning of the python finalize part, to throw out any lingering traceback and frame objects.  This will cause side effects such as gil-twiddling to happen early, and without consequence.  sys.exc_clear() is a safe api and used throughout the code, an extra call just prior to exit should be a very safe operation.
2) Turn off multi-threading at the start of python exit, by setting interpreter_lock to NULL.  Again, this is a no-brainer, since the NULL value is already well defined and works well.  It will cause all subsequent GIL releases to be no-ops.

I personally favor the second one.  It makes no sense to allow other threads to run after finalization has started and forcing them to stay dead is only prudent.

2.7 is not frozen, and it is in bug fix mode.  If a solid bug fix to an actual problem is proposed, we should embrace it, and deal with any eventual fallout appropriate as the price of doing business.  2.7 is a product that is alive and well, used by millions of people and thousands of businesses and will remain so for quite some time.  I know that most of the core devs have moved on to greener pastures, but I for one would like to stay loyal to this workhorse of a product and continue to make necessary improvements to it.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17969>
_______________________________________