how to find the file path to an extension module at init time?
Hi, in Python modules, the "__file__" attribute is provided by the runtime before executing the module body. For extension modules, it is set only after executing the init function. I wonder if there's any way to figure out where an extension module is currently being loaded from. The _PyImport_LoadDynamicModule() function obviously knows it, but it does not pass that information on into the module init function. I'm asking specifically because I'd like to properly implement __file__ in Cython modules at module init time. There are cases where it could be faked (when compiling modules on the fly during import), but in general, it's not known at compile time where a module will get installed and run from, so I currently don't see how to do it without dedicated runtime support. That's rather unfortunate, because it's not so uncommon for packages to look up bundled data files relative to their own position using __file__, and that is pretty much always done in module level code. Another problem is that package local imports from __init__.py no longer work when it's compiled, likely because __path__ is missing on the new born module object in sys.modules. Here, it would also help if the path to the module (and to its package) was known early enough. Any ideas how this could currently be achieved? Or could this become a new feature in the future? Stefan
I'm asking specifically because I'd like to properly implement __file__ in Cython modules at module init time.
Why do you need to implement __file__? Python will set it eventually to its correct value, no?
Another problem is that package local imports from __init__.py no longer work when it's compiled
Does it actually work to have __init__ be an extension module?
Any ideas how this could currently be achieved?
Currently, for Cython? I don't think that can work.
Or could this become a new feature in the future?
Certainly. An approach similar to _Py_PackageContext should be possible. Regards, Martin
"Martin v. Löwis", 13.11.2011 21:46:
I'm asking specifically because I'd like to properly implement __file__ in Cython modules at module init time.
Why do you need to implement __file__? Python will set it eventually to its correct value, no?
Well, yes, eventually. However, almost all real world usages are at module init time, not afterwards. So having CPython set it after running through the module global code doesn't help much.
Another problem is that package local imports from __init__.py no longer work when it's compiled
Does it actually work to have __init__ be an extension module?
I'm just starting to try it, and the problems I found so far were __file__ (in general), __path__ and relative imports (specifically).
Any ideas how this could currently be achieved?
Currently, for Cython? I don't think that can work.
Hmm, it might work to put an empty module next to the 'real' extension and to import it to figure out the common directory of both. As long as it's still there after installation and the right one gets imported, that is. A relative import should help on versions that support it. Although that won't help in the __init__ case because a relative import will likely depend on __path__ being available first. Chicken and egg... Support in CPython would definitely help.
Or could this become a new feature in the future?
Certainly. An approach similar to _Py_PackageContext should be possible.
Yes, and a "_Py_ModuleImportContext" would be rather trivial to do. Could that go into 3.3? What about 2.7? Could an exception be made there regarding new "features"? It's not likely to break anything, but it would help Cython. Stefan
On Mon, Nov 14, 2011 at 6:18 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Certainly. An approach similar to _Py_PackageContext should be possible.
Yes, and a "_Py_ModuleImportContext" would be rather trivial to do. Could that go into 3.3? What about 2.7? Could an exception be made there regarding new "features"? It's not likely to break anything, but it would help Cython.
Hmm, interesting call - fixing this doesn't actually require a new public API, since it's just a new data attribute protected by the import lock that is used to pass state information from importdl.c to moduleobject.c. I'm inclined to say "no" myself, but it's a close run thing. So +1 for fixing it in 3.3, and -0 for calling it a bug rather than a missing feature and also fixing it in 2.7 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 14 November 2011 08:18, Stefan Behnel <stefan_ml@behnel.de> wrote:
"Martin v. Löwis", 13.11.2011 21:46:
I'm asking specifically because I'd like to properly implement __file__ in Cython modules at module init time.
Why do you need to implement __file__? Python will set it eventually to its correct value, no?
Well, yes, eventually. However, almost all real world usages are at module init time, not afterwards. So having CPython set it after running through the module global code doesn't help much.
Perhaps Cython could detect use of __file__ at module scope (if this package context function is not available), and if it's used it tries to use something akin to imp.find_module(__name__) to find the path to the file and set __file__ manually. It should handle dots out of the box and perhaps not rely on any __path__ attributes of packages (I think not many people change __path__ or use pkgutil anyway). Would this be feasible for python < 3.3?
Another problem is that package local imports from __init__.py no longer work when it's compiled
Does it actually work to have __init__ be an extension module?
I'm just starting to try it, and the problems I found so far were __file__ (in general), __path__ and relative imports (specifically).
Any ideas how this could currently be achieved?
Currently, for Cython? I don't think that can work.
Hmm, it might work to put an empty module next to the 'real' extension and to import it to figure out the common directory of both. As long as it's still there after installation and the right one gets imported, that is. A relative import should help on versions that support it. Although that won't help in the __init__ case because a relative import will likely depend on __path__ being available first. Chicken and egg...
Support in CPython would definitely help.
Or could this become a new feature in the future?
Certainly. An approach similar to _Py_PackageContext should be possible.
Yes, and a "_Py_ModuleImportContext" would be rather trivial to do. Could that go into 3.3? What about 2.7? Could an exception be made there regarding new "features"? It's not likely to break anything, but it would help Cython.
Stefan
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/markflorisson88%40gmail.co...
mark florisson, 14.11.2011 12:55:
On 14 November 2011 08:18, Stefan Behnel wrote:
"Martin v. Löwis", 13.11.2011 21:46:
I'm asking specifically because I'd like to properly implement __file__ in Cython modules at module init time.
Why do you need to implement __file__? Python will set it eventually to its correct value, no?
Well, yes, eventually. However, almost all real world usages are at module init time, not afterwards. So having CPython set it after running through the module global code doesn't help much.
Perhaps Cython could detect use of __file__ at module scope (if this package context function is not available), and if it's used it tries to use something akin to imp.find_module(__name__) to find the path to the file and set __file__ manually.
It's problematic. Depending on the import hooks that are in use, a second search for the already loaded but not yet instantiated module may potentially trigger a second try to load it. And if the module is already put into sys.modules, a second search may not return a helpful result. Also, running a full import of "imp" along the way may have additional side effects, and the C-API doesn't have a way to just search for the path of a module.
Would this be feasible for python < 3.3?
It would certainly not be more than a work-around, but it could be made to work "well enough" in many cases. I'd definitely prefer having something that really works in both 2.7.x and 3.3+ over a half-baked solution in all of Py2.x. Stefan
Currently, for Cython? I don't think that can work.
Hmm, it might work to put an empty module next to the 'real' extension and to import it to figure out the common directory of both. As long as it's still there after installation and the right one gets imported, that is. A relative import should help on versions that support it. Although that won't help in the __init__ case because a relative import will likely depend on __path__ being available first. Chicken and egg...
If there was an actual __init__.py that just had import __cinit__ then __cinit__ could copy __path__ from the already-loaded __init__, no?
Certainly. An approach similar to _Py_PackageContext should be possible.
Yes, and a "_Py_ModuleImportContext" would be rather trivial to do. Could that go into 3.3?
If somebody contributes a patch: sure.
What about 2.7?
Certainly not. It would be a new feature, and there can't be new features in 2.7.
Could an exception be made there regarding new "features"? It's not likely to break anything, but it would help Cython.
"Not likely to break anything" really means it practice "it probably will break somebody's code". Policies are there to be followed, and this one I personally feel strongly about. If it means that users can get certain Cython features only with Python 3.x, the better. Regards, Martin
"Martin v. Löwis", 15.11.2011 01:33:
Currently, for Cython? I don't think that can work.
Hmm, it might work to put an empty module next to the 'real' extension and to import it to figure out the common directory of both. As long as it's still there after installation and the right one gets imported, that is. A relative import should help on versions that support it. Although that won't help in the __init__ case because a relative import will likely depend on __path__ being available first. Chicken and egg...
If there was an actual __init__.py that just had
import __cinit__
or rather from .__cinit__ import * (relative import only for CPythons that support it)
then __cinit__ could copy __path__ from the already-loaded __init__, no?
Hmm, right - that should work. __init__ would be in sys.modules with a properly set __file__ and __path__, and __cinit__ (knowing its own package name anyway) could look it up there to find out its file system path. I don't think there's much code out there that actually uses __file__ to find out the name of the module rather than just its package directory. So it would (more or less) fix both problems for __init__ files, and it should work with Py2.4+.
Certainly. An approach similar to _Py_PackageContext should be possible.
Yes, and a "_Py_ModuleImportContext" would be rather trivial to do. Could that go into 3.3?
If somebody contributes a patch: sure.
Ok, cool.
What about 2.7?
Certainly not. It would be a new feature, and there can't be new features in 2.7.
Could an exception be made there regarding new "features"? It's not likely to break anything, but it would help Cython.
"Not likely to break anything" really means it practice "it probably will break somebody's code". Policies are there to be followed, and this one I personally feel strongly about. If it means that users can get certain Cython features only with Python 3.x, the better.
Understandable. Stefan
Stefan Behnel, 15.11.2011 09:01:
"Martin v. Löwis", 15.11.2011 01:33:
An approach similar to _Py_PackageContext should be possible.
Yes, and a "_Py_ModuleImportContext" would be rather trivial to do. Could that go into 3.3?
If somebody contributes a patch: sure.
Ok, cool.
Patch(es) uploaded to the bug tracker. http://bugs.python.org/issue13429 Stefan
participants (4)
-
"Martin v. Löwis" -
mark florisson -
Nick Coghlan -
Stefan Behnel