PyCapsule_Import semantics, relative imports, module names etc.
While porting several existing CPython extension modules that form a package to be 2.7 and 3.x compatible the existing PyObject_* API was replaced with PyCapsule_*. This introduced some issues the existing CPython docs are silent on. I'd like clarification on a few issues and wish to raise some questions. 1. Should an extension module name as provided in PyModule_Create (Py3) or Py_InitModule3 (Py2) be fully package qualified or just the module name? I believe it's just the module name (see item 5 below) Yes/No? 2. PyCapsule_Import does not adhere to the general import semantics. The module name must be fully qualified, relative imports are not supported. 3. PyCapsule_Import requires the package (e.g. __init__.py) to import *all* of it's submodules which utilize the PyCapsule mechanism preventing lazy on demand loading. This is because PyCapsule_Import only imports the top level module (e.g. the package). From there it iterates over each of the module names in the module path. However the parent module (e.g. globals) will not contain an attribute for the submodule unless it's already been loaded. If the submodule has not been loaded into the parent PyCapsule_Import throws an error instead of trying to load the submodule. The only apparent solution is for the package to load every possible submodule whether required or not just to avoid a loading error. The inability to load modules on demand seems like a design flaw and change in semantics from the prior use of PyImport_ImportModule in combination with PyObject. [One of the nice features with normal import loading is setting the submodule name in the parent, the fact this step is omitted is what causes PyCapsule_Import to fail unless all submodules are unconditionally loaded). Shouldn't PyCapsule_Import utilize PyImport_ImportModule? 4. Relative imports seem much more useful for cooperating submodules in a package as opposed to fully qualified package names. Being able to import a C_API from the current package (the package I'm a member of) seems much more elegant and robust for cooperating modules but this semantic isn't supported (in fact the leading dot syntax completely confuses PyCapsule_Import, doc should clarify this). 5. The requirement that a module specifies it's name as unqualified when it is initializing but then also has to use a fully qualified package name for PyCapsule_New, both of which occur inside the same initialization function seems like an odd inconsistency (documentation clarification would help here). Also, depending on your point of view package names could be considered a deployment/packaging decision, a module obtains it's fully qualified name by virtue of it's position in the filesystem, something at compile time the module will not be aware of, another reason why relative imports make sense. Note the identical comment regarding _Py_PackageContext in modsupport.c (Py2) and moduleobject.c (Py3) regarding how a module obtains it's fully qualified package name (see item 1). Thanks! -- John
PyCapsule_Import() is a simple helper function, a slightly-updated analogue to PyCObject_Import(). It's not particularly sophisticated, and I'm not surprised it's bewildered by complicated scenarios like relative imports in subpackages. For now all I can recommend is that you not try and torture PyCapsule_Import(). And, as always... patches welcome. /arry On Sat, Jul 25, 2015 at 1:41 AM, John Dennis <jdennis@redhat.com> wrote:
While porting several existing CPython extension modules that form a package to be 2.7 and 3.x compatible the existing PyObject_* API was replaced with PyCapsule_*. This introduced some issues the existing CPython docs are silent on. I'd like clarification on a few issues and wish to raise some questions.
1. Should an extension module name as provided in PyModule_Create (Py3) or Py_InitModule3 (Py2) be fully package qualified or just the module name? I believe it's just the module name (see item 5 below) Yes/No?
2. PyCapsule_Import does not adhere to the general import semantics. The module name must be fully qualified, relative imports are not supported.
3. PyCapsule_Import requires the package (e.g. __init__.py) to import *all* of it's submodules which utilize the PyCapsule mechanism preventing lazy on demand loading. This is because PyCapsule_Import only imports the top level module (e.g. the package). From there it iterates over each of the module names in the module path. However the parent module (e.g. globals) will not contain an attribute for the submodule unless it's already been loaded. If the submodule has not been loaded into the parent PyCapsule_Import throws an error instead of trying to load the submodule. The only apparent solution is for the package to load every possible submodule whether required or not just to avoid a loading error. The inability to load modules on demand seems like a design flaw and change in semantics from the prior use of PyImport_ImportModule in combination with PyObject. [One of the nice features with normal import loading is setting the submodule name in the parent, the fact this step is omitted is what causes PyCapsule_Import to fail unless all submodules are unconditionally loaded). Shouldn't PyCapsule_Import utilize PyImport_ImportModule?
4. Relative imports seem much more useful for cooperating submodules in a package as opposed to fully qualified package names. Being able to import a C_API from the current package (the package I'm a member of) seems much more elegant and robust for cooperating modules but this semantic isn't supported (in fact the leading dot syntax completely confuses PyCapsule_Import, doc should clarify this).
5. The requirement that a module specifies it's name as unqualified when it is initializing but then also has to use a fully qualified package name for PyCapsule_New, both of which occur inside the same initialization function seems like an odd inconsistency (documentation clarification would help here). Also, depending on your point of view package names could be considered a deployment/packaging decision, a module obtains it's fully qualified name by virtue of it's position in the filesystem, something at compile time the module will not be aware of, another reason why relative imports make sense. Note the identical comment regarding _Py_PackageContext in modsupport.c (Py2) and moduleobject.c (Py3) regarding how a module obtains it's fully qualified package name (see item 1).
Thanks!
-- John _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/larry%40hastings.org
On 27 July 2015 at 01:21, Larry Hastings <larry@hastings.org> wrote:
PyCapsule_Import() is a simple helper function, a slightly-updated analogue to PyCObject_Import(). It's not particularly sophisticated, and I'm not surprised it's bewildered by complicated scenarios like relative imports in subpackages. For now all I can recommend is that you not try and torture PyCapsule_Import(). And, as always... patches welcome.
In this case, there are actually a lot of limitations related to the fact that extension modules generally have far more limited information about where they live in the package hierarchy than normal Python modules do. PEP 489 addressed quite a few of those with multi-phase initialisation.
On Sat, Jul 25, 2015 at 1:41 AM, John Dennis <jdennis@redhat.com> wrote:
While porting several existing CPython extension modules that form a package to be 2.7 and 3.x compatible the existing PyObject_* API was replaced with PyCapsule_*. This introduced some issues the existing CPython docs are silent on. I'd like clarification on a few issues and wish to raise some questions.
1. Should an extension module name as provided in PyModule_Create (Py3) or Py_InitModule3 (Py2) be fully package qualified or just the module name? I believe it's just the module name (see item 5 below) Yes/No?
Fully qualified is generally better (if you know the ultimate location), but it's mainly for introspection support, so most things will work fine even if you set the name to something like "<this is mostly ignored>".
2. PyCapsule_Import does not adhere to the general import semantics. The module name must be fully qualified, relative imports are not supported.
Correct, as it has no knowledge of the current module name to anchor a relative import.
3. PyCapsule_Import requires the package (e.g. __init__.py) to import *all* of it's submodules which utilize the PyCapsule mechanism preventing lazy on demand loading. This is because PyCapsule_Import only imports the top level module (e.g. the package). From there it iterates over each of the module names in the module path. However the parent module (e.g. globals) will not contain an attribute for the submodule unless it's already been loaded. If the submodule has not been loaded into the parent PyCapsule_Import throws an error instead of trying to load the submodule. The only apparent solution is for the package to load every possible submodule whether required or not just to avoid a loading error. The inability to load modules on demand seems like a design flaw and change in semantics from the prior use of PyImport_ImportModule in combination with PyObject. [One of the nice features with normal import loading is setting the submodule name in the parent, the fact this step is omitted is what causes PyCapsule_Import to fail unless all submodules are unconditionally loaded). Shouldn't PyCapsule_Import utilize PyImport_ImportModule?
This sounds like it may be a bug in PyCapsule_Import, but I don't know the capsule API very well myself (I've never had a reason to use it - all the extension modules I've worked with personally have been self-contained).
4. Relative imports seem much more useful for cooperating submodules in a package as opposed to fully qualified package names. Being able to import a C_API from the current package (the package I'm a member of) seems much more elegant and robust for cooperating modules but this semantic isn't supported (in fact the leading dot syntax completely confuses PyCapsule_Import, doc should clarify this).
Until PEP 489 (multi-phase initialisation) was implemented for Python 3.5, extension modules didn't know their actual runtime place in the module hierarchy, so there was no easy way to provide a module name to the API to anchor relative lookups. Given PEP 489, it may be feasible to offer a PyCapsule_ImportRelative for 3.6+, but it would require someone interested in working through the details of such an API.
5. The requirement that a module specifies it's name as unqualified when it is initializing but then also has to use a fully qualified package name for PyCapsule_New, both of which occur inside the same initialization function seems like an odd inconsistency (documentation clarification would help here). Also, depending on your point of view package names could be considered a deployment/packaging decision, a module obtains it's fully qualified name by virtue of it's position in the filesystem, something at compile time the module will not be aware of, another reason why relative imports make sense. Note the identical comment regarding _Py_PackageContext in modsupport.c (Py2) and moduleobject.c (Py3) regarding how a module obtains it's fully qualified package name (see item 1).
Yes, these weird limitations were the genesis of Petr Viktorin's efforts in implementing a new approach to import extension modules for Python 3.5: https://www.python.org/dev/peps/pep-0489/ While the old single-phase initialisation mechanism is still supported for backwards compatibility, the new multi-phase initialisation approach brings the capabilities of extension modules much closer to their pure Python counterparts. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (3)
-
John Dennis -
Larry Hastings -
Nick Coghlan