[Python-ideas] New PyThread_tss_ C-API for CPython

[Erik Bray]

On Wed, Dec 21, 2016 at 2:10 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 21 December 2016 at 01:35, Masayuki YAMAMOTO <ma3yuki.8mamo10 at gmail.com>
> wrote:
>>
>> 2016-12-20 22:30 GMT+09:00 Erik Bray <erik.m.bray at gmail.com>:
>>>
>>> This is probably an implementation detail, but ISTM that even with
>>> PyThread_call_once, it will be necessary to reset any used once_flags
>>> manually in PyOS_AfterFork, essentially for the same reason the
>>> autoTLSkey is reset there currently...
>>
>>
>> Deleting threads key is executed on *_Fini functions, but Py_FinalizeEx
>> function that calls *_Fini functions doesn't terminate CPython interpreter.
>> Furthermore, source comment and document have said description about
>> reinitialization after calling Py_FinalizeEx. [1] [2] That is to say there
>> is an implicit possible that is reinitialization contrary to name
>> "call_once" on a process level. Therefore, if CPython interpreter continues
>> to allow reinitialization, I'd suggest to rename the call_once API to avoid
>> misreading semantics. (for example, safe_init, check_init)
>
>
> Ouch, I'd missed that, and I agree it's not a negligible implementation
> detail - there are definitely applications embedding CPython out there that
> rely on being able to run multiple Initialize/Finalize cycles in the same
> process and have everything "just work". It also means using the
> "PyThread_*" prefix for the initialisation tracking aspect would be
> misleading, since the life cycle details are:
>
> 1. Create the key for the first time if it has never been previously set in
> the process
> 2. Destroy and reinit if Py_Finalize gets called
> 3. Destroy and reinit if a new subprocess is forked
>
> It also means we can't use pthread_once even in the pthread TLS
> implementation, since it doesn't provide those semantics.
>
> So I see two main alternatives here.
>
> Option 1: Modify the proposed PyThread_tss_create and PyThread_tss_delete
> APIs to accept a "bool *init_flag" pointer in addition to their current
> arguments.
>
> If *init_flag is true, then PyThread_tss_create is a no-op, otherwise it
> sets the flag to true after creating the key.
> If *init_flag is false, then PyThread_tss_delete is a no-op, otherwise it
> sets the flag to false after deleting the key.
>
> Option 2: Similar to option 1, but using a custom type alias, rather than
> using a C99 bool directly
>
> The closest API we have to these semantics at the moment would be
> PyGILState_Ensure, so the following API naming might work for option 2:
>
>     Py_ensure_t
>     Py_ENSURE_NEEDS_INIT
>     Py_ENSURE_INITIALIZED
>
> Respectively, these would just be aliases for bool, false, and true.
>
> And then modify the proposed PyThread_tss_create and PyThread_tss_delete
> APIs to accept a "Py_ensure_t *init_flag" in addition to their current
> arguments.

That all sounds good--between the two option 2 looks a bit more explicit.

Though what about this?  Rather than adding another type, the original
proposal could be changed slightly so that Py_tss_t *is* partially
defined as a struct consisting of a bool, with whatever the native TLS
key is.   E.g.

typedef struct {
    bool init_flag;
    #if defined(_POSIX_THREADS)
    pthreat_key_t key;
    #elif defined (NT_THREADS)
    DWORD key;
    /* etc... */
} Py_tss_t;

Then it's just taking Masayuki's original patch, with the global bool
variables, and formalizing that by combining the initialized flag with
the key, and requiring the semantics you described above for
PyThread_tss_create/delete.

For Python's purposes it seems like this might be good enough, with
the more general purpose pthread_once-like functionality not required.

Best,
Erik