__module__ of newly-created extension classes
When creating new-style classes from within an extension module, the current behavior results in the __module__ attribute being set to the name of the Python module which first imports the extension module. I recall having a similar problem with my own extension classes in Boost.Python v1, but was a little surprised to find that it persists in Boost.Python v2, when creating new-style classes. A trivial inspection reveals this bit of code in typeobject.c: /* Set __module__ in the dict */ if (PyDict_GetItemString(dict, "__module__") == NULL) { tmp = PyEval_GetGlobals(); if (tmp != NULL) { tmp = PyDict_GetItemString(tmp, "__name__"); if (tmp != NULL) { if (PyDict_SetItemString(dict, "__module__", tmp) < 0) return NULL; } } } I can work around this problem, but I wonder if it would be a good idea if Python set __name__ automatically during the import of an extension module? I realize that there are ways to undermine this, e.g. explicitly creating a nested module object. +---------------------------------------------------------------+ David Abrahams C++ Booster (http://www.boost.org) O__ == Pythonista (http://www.python.org) c/ /'_ == resume: http://users.rcn.com/abrahams/resume.html (*) \(*) == email: david.abrahams@rcn.com +---------------------------------------------------------------+
Hi David!
When creating new-style classes from within an extension module, the current behavior results in the __module__ attribute being set to the name of the Python module which first imports the extension module. [...]
I'm not sure I understood your problem. If I got it correctly, you will get the right behavior by setting tp_name to something like "module.TypeName". -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ]
From: "Gustavo Niemeyer" <niemeyer@conectiva.com>
Hi David!
When creating new-style classes from within an extension module, the current behavior results in the __module__ attribute being set to the name of the Python module which first imports the extension module. [...]
I'm not sure I understood your problem. If I got it correctly, you will get the right behavior by setting tp_name to something like "module.TypeName".
Well, that wouldn't fix the __module__ attribute AFAICT. I'm creating new classes by invoking the metaclass with the name, bases, and dictionary. -Dave
When creating new-style classes from within an extension module, the current behavior results in the __module__ attribute being set to the name of the Python module which first imports the extension module.
I recall having a similar problem with my own extension classes in Boost.Python v1, but was a little surprised to find that it persists in Boost.Python v2, when creating new-style classes. A trivial inspection reveals this bit of code in typeobject.c:
/* Set __module__ in the dict */ if (PyDict_GetItemString(dict, "__module__") == NULL) { tmp = PyEval_GetGlobals(); if (tmp != NULL) { tmp = PyDict_GetItemString(tmp, "__name__"); if (tmp != NULL) { if (PyDict_SetItemString(dict, "__module__", tmp) < 0) return NULL; } } }
I can work around this problem, but I wonder if it would be a good idea if Python set __name__ automatically during the import of an extension module? I realize that there are ways to undermine this, e.g. explicitly creating a nested module object.
The __name__ attribute is already set as soon as a module is created. I think the problem is that the extension's dict is not the "current globals dict" as returned by PyEval_GetGlobals(). There's no convenient way to make this so, since the "current globals dict" is taken from the frame (and I don't want to change that). What was your workaround? I thought that the "correct" way to set __module__ is to use a dotted name as the type name (the initializer for tp_name). Can you do that? --Guido van Rossum (home page: http://www.python.org/~guido/)
From: "Guido van Rossum" <guido@python.org>
I can work around this problem, but I wonder if it would be a good idea if Python set __name__ automatically during the import of an extension module? I realize that there are ways to undermine this, e.g. explicitly creating a nested module object.
The __name__ attribute is already set as soon as a module is created.
I think the problem is that the extension's dict is not the "current globals dict" as returned by PyEval_GetGlobals().
Ah, thanks. I only did a very quick analysis, and didn't remember the way it turned out when I looked at this in past years for v1.
There's no convenient way to make this so, since the "current globals dict" is taken from the frame (and I don't want to change that).
Fine, I understand, I think.
What was your workaround?
If I recall correctly, what I was doing in v1 was to keep track of the module being initialized in some kind of stack under-the-covers, and to grab its __name__ attribute.
I thought that the "correct" way to set __module__ is to use a dotted name as the type name (the initializer for tp_name).
You would know. Didn't you write that code?
Can you do that?
Hmm, I see that you have a property which grabs the __module__ from the class name... Since I am invoking the metaclass to create classes, I don't set tp_name directly, but I could manufacture the right name and pass it to the metaclass. That requires pretty much the same sort of under-the-covers collaboration between the code that creates the module and the code that creates the class. That's OK, but (minor point) it means that my library, which is meant to be a framework of re-usable components, isn't really as modular as it should be: you couldn't just use the class generation facilities in the context of someone else's module creation. Now I'm wondering if there isn't a better way for Python to get ahold of the current module's __name__ in this case, if you don't want to change what PyEval_GetGlobals() does. -Dave
Now I'm wondering if there isn't a better way for Python to get ahold of the current module's __name__ in this case, if you don't want to change what PyEval_GetGlobals() does.
If you can rigorously define "current module" I can propose an API. --Guido van Rossum (home page: http://www.python.org/~guido/)
From: "Guido van Rossum" <guido@python.org>
Now I'm wondering if there isn't a better way for Python to get ahold of the current module's __name__ in this case, if you don't want to change what PyEval_GetGlobals() does.
If you can rigorously define "current module" I can propose an API.
A rigorous definition is easy; I can't promise it would always meet everyone's expectations: The "current module" is given by the current formula (PyEval_GetGlobals(), etc.) except when loading an extension module, in which case the current module is the one most recently created. I'm not sure, but this might be simplifiable to: the "current module" is the one most recently created. -Dave
If you can rigorously define "current module" I can propose an API.
A rigorous definition is easy; I can't promise it would always meet everyone's expectations:
The "current module" is given by the current formula (PyEval_GetGlobals(), etc.) except when loading an extension module, in which case the current module is the one most recently created.
If the init function of an extension module calls PyImport_Import() to import another extension which has to be loaded freshly, does that mean that after that point the definition current module is left to that other module, or does it revert to the first extension? What if an extension imports a Python module which loads an extension module?
I'm not sure, but this might be simplifiable to:
the "current module" is the one most recently created.
No, because in a Python function defined in a Python module, when that function is executing that module is the current module. A possible implementation could maintain a per-thread global which is NULL when we're not loading an extension. Py_InitModule() sets this global to the module it creates. When the import mechanism calls an extension's initxxx function, it saves the value of this per-thread global; when the function returns, it restores it from the saved value. Then there could be a new function Py_GetGlobals() that looks in this per-thread global, and returns its dict if it is not NULL; if NULL, it falls back on PyEval_GetGlobals(). (You can't promise to return the current *module*, because that isn't kept track of; only the current globals are kept track of on the execution stack.) --Guido van Rossum (home page: http://www.python.org/~guido/)
From: "Guido van Rossum" <guido@python.org>
The "current module" is given by the current formula (PyEval_GetGlobals(), etc.) except when loading an extension module, in which case the current module is the one most recently created.
If the init function of an extension module calls PyImport_Import() to import another extension which has to be loaded freshly, does that mean that after that point the definition current module is left to that other module, or does it revert to the first extension?
It reverts of course (glad you asked)!
What if an extension imports a Python module which loads an extension module?
Of course there's a similar reversion after the extension module finishes loading.
I'm not sure, but this might be simplifiable to:
the "current module" is the one most recently created.
No, because in a Python function defined in a Python module, when that function is executing that module is the current module.
A possible implementation could maintain a per-thread global which is NULL when we're not loading an extension. Py_InitModule() sets this global to the module it creates. When the import mechanism calls an extension's initxxx function, it saves the value of this per-thread global; when the function returns, it restores it from the saved value. Then there could be a new function Py_GetGlobals() that looks in this per-thread global, and returns its dict if it is not NULL; if NULL, it falls back on PyEval_GetGlobals(). (You can't promise to return the current *module*, because that isn't kept track of; only the current globals are kept track of on the execution stack.)
Sounds pretty good to me. -Dave
participants (3)
-
David Abrahams -
Guido van Rossum -
Gustavo Niemeyer