[Python-ideas] Threading hooks and disable gc per thread

[Nick Coghlan]

On Thu, May 12, 2011 at 9:58 AM, Christian Heimes <lists at cheimes.de> wrote:
> on thread start hook
> --------------------
>
> Similar to the atexit module, third party modules can register a
> callable with *args and **kwargs. The functions are called inside the
> newly created thread just before the target is called. The best place
> for the hook list is threading.Thread._bootstrap_inner() right before
> the try: self.run() except: block. Exceptions are ignored during the
> call but reported to the user at the end (same as atexit's
> atexit_callfunc())
>
>
> on thread end hook
> ------------------
>
> Same as on thread start hook but the callables are called inside the
> dying thread after self.run().

So the plan is to have threading.Thread support the hooks, while
_thread.start_new_thread and creation of thread states at the C level
(including via PyGILState_Ensure) will bypass them?

That actually sounds reasonable to me (+0), but the PEP should at
least discuss the rationale for the choice of level for the new
feature. I also suggest storing the associated hook lists at the
threading.Thread class object level rather than at the threading
module level (supporting such modularity of state being a major
advantage of only providing this feature at the higher level).

The PEP should also go into detail as to why having these hooks in a
custom Thread subclass isn't sufficient (e.g. needing to support
threads created by third party libraries, but note that such a
rationale has a problem due to the _thread.start_new_thread loophole).

Composability through inheritance should also be discussed - the hook
invocation should probably walk the MRO so it is easy to create Thread
subclasses that include class specific hooks without inadvertently
skipping the hooks installed on threading.Thread.

The possibility of passing exception information to thread_end hooks
(ala __exit__ methods) should be considered, along with the general
relationship between the threading hooks and the context management
protocol.

> gc.disable_thread(), gc.enable_thread(), gc.isenabled_thread()
> --------------------------------------------------------------
>
> Right now almost any code can trigger a gc.collect() run
> non-deterministicly. Some application like JCC want to control if
> gc.collect() is wanted on a thread level. This could be solved with a
> new flat in PyThreadState. PyThreadState->gc_enabled is enabled by
> default. When the flag is false, _PyObject_GC_Malloc() doesn't start a
> gc.collect() run for that thread. The collection is delayed until
> another thread or the main thread triggers it.
>
> The three functions should also have a C equivalent so C code can
> prevent gc in a thread.

The default setting for this should go in the interpreter state object
rather than in a static variable (subinterpreters can then inherit the
state of their parent interpreter when they are first created).

Otherwise sounds reasonable. (+0)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia