Calling PyState_AddModule in module init?
Hello, Currently it's not necessary to call PyState_AddModule in a module PyInit_* function. For single-phase init modules, Python calls it automatically. However, documentation of PyState_FindModule sugests that you need to call PyState_AddModule.
I wrote what I think is the expected behavior in PR 16101, but I'm not sure that's actually it. If anyone who knows the original intent is reading this, could you chime in? :)
On Sat, 14 Sep 2019 at 01:59, Petr Viktorin <encukou@gmail.com> wrote:
Poking around with the git commit annotations brought me to https://bugs.python.org/issue15042
So the issue is that the import machinery calls PyState_AddModule *after* the init function returns. That means that if you want to be able to use PyState_FindModule *from* the init function (directly or indirectly), then you need to call PyState_AddModule explicitly before the calls that need it.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
So the issue is that the import machinery calls PyState_AddModule after the init function returns.
There's a bit more to it. It seems that the fact that PyState_AddModule is called after the init function is an implementation detail of the default import mechanism of CPython.
For instance, builtins.__import__ can be replaced with an alternate import mechanism that does not use PyState_AddModule. Then, calling PyImport_Import twice on a C will execute the init function twice. If this is a C Extension that holds module state then you wouldn't want that init function to be called twice.
Therefore, PyState_AddModule should always be added to a C Extension that initializes module state in the init function. Then, PyState_FindModule should be used to avoid the re-initialization.
- Eddie
On Wed, 18 Sep 2019 at 18:01, Petr Viktorin <encukou@gmail.com> wrote:
Right, I think the documentation you've already added is correct on that front (all extension module import implementations should be automatically calling PyState_AddModule), but we should also mention the reasons why a module might want to call it explicitly:
- in order to use PyState_FindModule (directly or indirectly) from within the extension module's own init function
- in order to work around an existing alternate extension module import function that doesn't call it automatically
At least some standard library extension modules do the latter (they call PyState_AddModule as the last thing they do before returning).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 14 Sep 2019 at 01:59, Petr Viktorin <encukou@gmail.com> wrote:
Poking around with the git commit annotations brought me to https://bugs.python.org/issue15042
So the issue is that the import machinery calls PyState_AddModule *after* the init function returns. That means that if you want to be able to use PyState_FindModule *from* the init function (directly or indirectly), then you need to call PyState_AddModule explicitly before the calls that need it.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
So the issue is that the import machinery calls PyState_AddModule after the init function returns.
There's a bit more to it. It seems that the fact that PyState_AddModule is called after the init function is an implementation detail of the default import mechanism of CPython.
For instance, builtins.__import__ can be replaced with an alternate import mechanism that does not use PyState_AddModule. Then, calling PyImport_Import twice on a C will execute the init function twice. If this is a C Extension that holds module state then you wouldn't want that init function to be called twice.
Therefore, PyState_AddModule should always be added to a C Extension that initializes module state in the init function. Then, PyState_FindModule should be used to avoid the re-initialization.
- Eddie
On Wed, 18 Sep 2019 at 18:01, Petr Viktorin <encukou@gmail.com> wrote:
Right, I think the documentation you've already added is correct on that front (all extension module import implementations should be automatically calling PyState_AddModule), but we should also mention the reasons why a module might want to call it explicitly:
- in order to use PyState_FindModule (directly or indirectly) from within the extension module's own init function
- in order to work around an existing alternate extension module import function that doesn't call it automatically
At least some standard library extension modules do the latter (they call PyState_AddModule as the last thing they do before returning).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (3)
-
Eddie Elizondo -
Nick Coghlan -
Petr Viktorin