[New-bugs-announce] [issue24544] Race condition and crash when embedding multi-thread script

[Oleksiy Markovets]

New submission from Oleksiy Markovets:

INTRODUCTION

While embedding python script in c++ application I faced random crashes 
with error:

Fatal Python error: Py_EndInterpreter: not the last thread                                                                                                                                                                                                                     
 Aborted  

Even though I explicitly join each thread created with *threading* 
module. Please see attachment, this is simplest python script +
c++ code which demonstrates problem.

By using threading.setprofiler I was able to make problem appear each run.
If you uncomment  time.sleep(1) problem won't reproduce.

Note:
In main.cpp I manually create separate interpreter because in real-life
application is multi-threaded and different thread uses different interpreters
for at least some sand-boxing.

INVESTIGATION

I did some investigation, and here is what I found out:

* Each new thread is started by calling *thread_PyThread_start_new_thread* from threadmodule.c.
Basically this function creates new thread which executes *t_bootstrap* and adds this
thread to interpreter's thread list.

* *t_bootstrap* executes python function *Thread.__bootstrap* and when it's done, removes
thread from interpreter's thread list.

* *Thread.__bootstrap* runs *Thread.run()* (actually python code which should be executed
in separate thread) and when it's done calls *Thread.__stop* (which by mean of condition
variable sets Boolean flag Thread.__stopped)

* *Thread.join* doesn't wait thread to exit, it only waits for *Thread.__stopped* flag
to be set.


So here is race condition:
When main thread finished *Thread.join* it's only guaranteed that *Thread.run*
is finished, but *t_bootstrap* still may be running (interpreter's thread list
is not cleared).

POSSIBLE SOLUTION

To call Thread.__stop from *t_bootstrap* (instead of Thread.__bootstrap) after
removing thread from interpreter's thread list.

Or do not use detached threads and call something like *pthread_join* in *Thread.join*.
Which is IMHO much clearer approach but also requires much more efforts.

----------
components: Extension Modules
files: main.cpp
messages: 246035
nosy: Oleksiy Markovets
priority: normal
severity: normal
status: open
title: Race condition and crash when embedding multi-thread script
type: crash
versions: Python 2.7
Added file: http://bugs.python.org/file39840/main.cpp

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue24544>
_______________________________________