[Python-ideas] Threading hooks and disable gc per thread

[Christian Heimes]

Am 15.05.2011 13:13, schrieb Nick Coghlan:

(Sorry for the delay, I was swamped with work again)

> So the plan is to have threading.Thread support the hooks, while
> _thread.start_new_thread and creation of thread states at the C level
> (including via PyGILState_Ensure) will bypass them?
> 
> That actually sounds reasonable to me (+0), but the PEP should at
> least discuss the rationale for the choice of level for the new
> feature. I also suggest storing the associated hook lists at the
> threading.Thread class object level rather than at the threading
> module level (supporting such modularity of state being a major
> advantage of only providing this feature at the higher level).

I've considered both places, too. _thread.start_new_thread() as well as
PyGILState_Ensure() would require a considerable amount of C coding for
a feature that won't affect performance in a noticeable way.

This is my answer against an C implementation in
_thread.start_new_thread(). It's far too much work for a feature that
can be implemented in Python easily. An implementation in the pure
Python threading module will work on PyPy, IronPython and Jython
instantly. I consider any library, that bypasses the threading module,
broken, too.

PyGILState_Ensure() or PyThreadState_New() are a different beast. I
concur, it would the best place for the hooks if I could think of a way
to implement the on-thread-stop hook. I don't see a way to execute some
code at the end of a thread without cooperation from the calling code.

> The PEP should also go into detail as to why having these hooks in a
> custom Thread subclass isn't sufficient (e.g. needing to support
> threads created by third party libraries, but note that such a
> rationale has a problem due to the _thread.start_new_thread loophole).

Understood.

> Composability through inheritance should also be discussed - the hook
> invocation should probably walk the MRO so it is easy to create Thread
> subclasses that include class specific hooks without inadvertently
> skipping the hooks installed on threading.Thread.

Good idea!

Do you think, it's sufficient to have hook methods like

class Thread:
    _start_hooks = []

    def on_thread_starting(self):
        for hook, args, kwargs in self._start_hooks:
            hook(*args, **kwargs)

? Subclasses of threading.Thread can easily overwrite the hook method
and call its parent's on_thread_starting().

> The possibility of passing exception information to thread_end hooks
> (ala __exit__ methods) should be considered, along with the general
> relationship between the threading hooks and the context management
> protocol.

That's an interesting idea! I'll consider it.

>> gc.disable_thread(), gc.enable_thread(), gc.isenabled_thread()
>> --------------------------------------------------------------
>
> The default setting for this should go in the interpreter state object
> rather than in a static variable (subinterpreters can then inherit the
> state of their parent interpreter when they are first created).
> 
> Otherwise sounds reasonable. (+0)

A subinterpreter flag isn't enough. All subinterpreters share a common
GC list. A gc.collect() inside a subinterpreter run affects the entire
interpreter and not just the one subinterpreter. I've to think about the
issue of subinterpreters ...

If I understand the code correctly, gc.get_objects() punches a hole in
the subinterpreter isolation. It returns all tracked objects of the
current process -- from all subinterpreters. Is this a design issue? The
fact isn't mentioned in
http://docs.python.org/c-api/init.html#bugs-and-caveats.

Christian