Usefulness of binary compatibility accross Python versions?
Hello, Nowadays we have an official mechanism for third-party C extensions to be binary-compatible accross feature releases of Python: the stable ABI. But, for non-stable ABI-using C extensions, there are also mechanisms in place to *try* and ensure binary compatibility. One of them is the way in which we add tp_ slots to the PyTypeObject structure. Typically, when adding a tp_XXX slot, you also need to add a Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures that have been compiled against a recent enough PyTypeObject definition. This way, extensions compiled against Python N-1 are supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set, the core Python runtime won't try to access the (non-existing) tp_XXX member. However, beside internal code complication, it means you need to add a new Py_TPFLAGS_HAVE_XXX each time we add a slot. Since we have only 32 such bits available (many of them already taken), it is a very limited resource. Is it worth it? (*) Can an extension compiled against Python N-1 really claim to be compatible with Python N, despite other possible differences? (*) we can't extend the tp_flags field to 64 bits, precisely because of the binary compatibility problem... Regards Antoine.
I think it's more acceptable to require matching versions now than it was 10 years ago -- people are much more likely to use installer tools like pip and conda that can check version compatibility. I think I'd be okay with dropping the flag-based mechanism you describe if we were to introduce a clear mechanism that always rejected a dynamically loaded module if it was compiled for a different Python version. This should happen without any cooperation from the module. Perhaps in Python.h we can introduce a reference to a variable whose name varies by version (major.minor, I think) and which is defined only by the interpreter itself. Or perhaps the version should be based on a separate ABI version. On Sat, Dec 16, 2017 at 5:22 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Hello,
Nowadays we have an official mechanism for third-party C extensions to be binary-compatible accross feature releases of Python: the stable ABI.
But, for non-stable ABI-using C extensions, there are also mechanisms in place to *try* and ensure binary compatibility. One of them is the way in which we add tp_ slots to the PyTypeObject structure.
Typically, when adding a tp_XXX slot, you also need to add a Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures that have been compiled against a recent enough PyTypeObject definition. This way, extensions compiled against Python N-1 are supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set, the core Python runtime won't try to access the (non-existing) tp_XXX member.
However, beside internal code complication, it means you need to add a new Py_TPFLAGS_HAVE_XXX each time we add a slot. Since we have only 32 such bits available (many of them already taken), it is a very limited resource. Is it worth it? (*) Can an extension compiled against Python N-1 really claim to be compatible with Python N, despite other possible differences?
(*) we can't extend the tp_flags field to 64 bits, precisely because of the binary compatibility problem...
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Sat, 16 Dec 2017 09:34:02 -0800 Guido van Rossum <guido@python.org> wrote:
I think it's more acceptable to require matching versions now than it was 10 years ago -- people are much more likely to use installer tools like pip and conda that can check version compatibility.
I think I'd be okay with dropping the flag-based mechanism you describe if we were to introduce a clear mechanism that always rejected a dynamically loaded module if it was compiled for a different Python version. This should happen without any cooperation from the module. Perhaps in Python.h we can introduce a reference to a variable whose name varies by version (major.minor, I think) and which is defined only by the interpreter itself. Or perhaps the version should be based on a separate ABI version.
Interestingly, Python 2 had such an API version check (though it would only emit a warning), it was removed as part of PEP 3121: https://github.com/python/cpython/commit/1a21451b1d73b65af949193208372e86bf3... I haven't been able to find the pre-approval discussion around PEP 3121, so I'm not sure why the API check was removed. The PEP (quite short!) also says nothing about it. Currently, you can pass a `module_api_version` to PyModule_Create2(), but that function is for specialists only :-) ("""Most uses of this function should be using PyModule_Create() instead; only use this if you are sure you need it.""") And the new multi-phase initialization API doesn't seem to support passing an API version: https://docs.python.org/3/c-api/module.html#multi-phase-initialization Fortunately, nowadays all major platforms (Windows, Linux, macOS) tag C extension filenames with the interpreter version (*), e.g.: - "XXX.cpython-35m-darwin.so" on macOS - "XXX?.cp35-win32.pyd" on Windows - "XXX?.cpython-37dm-x86_64-linux-gnu.so" on Linux and others So in practice there should be little potential for confusion, except when renaming the extension file? (*) references: https://bugs.python.org/issue22980 https://docs.python.org/3.7/whatsnew/3.5.html#build-and-c-api-changes Regards Antoine.
On Sat, 16 Dec 2017 19:37:54 +0100 Antoine Pitrou <solipsis@pitrou.net> wrote:
Currently, you can pass a `module_api_version` to PyModule_Create2(), but that function is for specialists only :-)
("""Most uses of this function should be using PyModule_Create() instead; only use this if you are sure you need it.""")
Ah, it turns out I misunderstood that piece of documentation and also what PEP 3121 really did w.r.t the module API check. PyModule_Create() is actually a *macro* calling PyModule_Create2() with the version number is was compiled against! #ifdef Py_LIMITED_API #define PyModule_Create(module) \ PyModule_Create2(module, PYTHON_ABI_VERSION) #else #define PyModule_Create(module) \ PyModule_Create2(module, PYTHON_API_VERSION) #endif And there's already a check for that version number in moduleobject.c: https://github.com/python/cpython/blob/master/Objects/moduleobject.c#L114 That check is always invoked when calling PyModule_Create() and PyModule_Create2(). Currently it merely invokes a warning, but we can easily turn that into an error. (with apologies to Martin von Löwis for not fully understanding what he did at the time :-)) Regards Antoine.
On Sat, Dec 16, 2017 at 11:14 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sat, 16 Dec 2017 19:37:54 +0100 Antoine Pitrou <solipsis@pitrou.net> wrote:
Currently, you can pass a `module_api_version` to PyModule_Create2(), but that function is for specialists only :-)
("""Most uses of this function should be using PyModule_Create() instead; only use this if you are sure you need it.""")
Ah, it turns out I misunderstood that piece of documentation and also what PEP 3121 really did w.r.t the module API check.
PyModule_Create() is actually a *macro* calling PyModule_Create2() with the version number is was compiled against!
#ifdef Py_LIMITED_API #define PyModule_Create(module) \ PyModule_Create2(module, PYTHON_ABI_VERSION) #else #define PyModule_Create(module) \ PyModule_Create2(module, PYTHON_API_VERSION) #endif
And there's already a check for that version number in moduleobject.c: https://github.com/python/cpython/blob/master/Objects/moduleobject.c#L114
That check is always invoked when calling PyModule_Create() and PyModule_Create2(). Currently it merely invokes a warning, but we can easily turn that into an error.
(with apologies to Martin von Löwis for not fully understanding what he did at the time :-))
If it's only a warning, I worry that if we stop checking the flag bits it can cause wild pointer following. This sounds like it would be a potential security issue (load a module, ignore the warning, try to use a certain API on a class it defines, boom). Also, could there still be 3rd party modules out there that haven't been recompiled in a really long time and use some older backwards compatible module initialization API? (I guess we could stop supporting that and let them fail hard.) -- --Guido van Rossum (python.org/~guido)
On Sat, 16 Dec 2017 11:42:15 -0800 Guido van Rossum <guido@python.org> wrote:
If it's only a warning, I worry that if we stop checking the flag bits it can cause wild pointer following. This sounds like it would be a potential security issue (load a module, ignore the warning, try to use a certain API on a class it defines, boom). Also, could there still be 3rd party modules out there that haven't been recompiled in a really long time and use some older backwards compatible module initialization API? (I guess we could stop supporting that and let them fail hard.)
As far as I can tell, all the legacy APIs were removed when PEP 3121 was implemented (Python 3 allowed us to do a clean break here). Regards Antoine.
Well then, maybe you can propose some specific set of changes? (I'm about to go on vacation and I'd like to focus on other things for the next two weeks though, so don't count on me too much.) On Sat, Dec 16, 2017 at 3:49 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sat, 16 Dec 2017 11:42:15 -0800 Guido van Rossum <guido@python.org> wrote:
If it's only a warning, I worry that if we stop checking the flag bits it can cause wild pointer following. This sounds like it would be a
potential
security issue (load a module, ignore the warning, try to use a certain API on a class it defines, boom). Also, could there still be 3rd party modules out there that haven't been recompiled in a really long time and use some older backwards compatible module initialization API? (I guess we could stop supporting that and let them fail hard.)
As far as I can tell, all the legacy APIs were removed when PEP 3121 was implemented (Python 3 allowed us to do a clean break here).
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Dec 16, 2017 11:44 AM, "Guido van Rossum" <guido@python.org> wrote: On Sat, Dec 16, 2017 at 11:14 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sat, 16 Dec 2017 19:37:54 +0100 Antoine Pitrou <solipsis@pitrou.net> wrote:
Currently, you can pass a `module_api_version` to PyModule_Create2(), but that function is for specialists only :-)
("""Most uses of this function should be using PyModule_Create() instead; only use this if you are sure you need it.""")
Ah, it turns out I misunderstood that piece of documentation and also what PEP 3121 really did w.r.t the module API check.
PyModule_Create() is actually a *macro* calling PyModule_Create2() with the version number is was compiled against!
#ifdef Py_LIMITED_API #define PyModule_Create(module) \ PyModule_Create2(module, PYTHON_ABI_VERSION) #else #define PyModule_Create(module) \ PyModule_Create2(module, PYTHON_API_VERSION) #endif
And there's already a check for that version number in moduleobject.c: https://github.com/python/cpython/blob/master/Objects/moduleobject.c#L114
That check is always invoked when calling PyModule_Create() and PyModule_Create2(). Currently it merely invokes a warning, but we can easily turn that into an error.
(with apologies to Martin von Löwis for not fully understanding what he did at the time :-))
If it's only a warning, I worry that if we stop checking the flag bits it can cause wild pointer following. This sounds like it would be a potential security issue (load a module, ignore the warning, try to use a certain API on a class it defines, boom). Also, could there still be 3rd party modules out there that haven't been recompiled in a really long time and use some older backwards compatible module initialization API? (I guess we could stop supporting that and let them fail hard.) I think there's a pretty simple way to avoid this kind of problem. Since PEP 3149 (Python 3.2), the import system has (IIUC) checked for: foo.cpython-XYm.so foo.abi3.so foo.so If we drop foo.so from this list, then we're pretty much guaranteed not to load anything into a python that it wasn't intended for. How disruptive would this be? AFAICT there hasn't been any standard way to build python extensions named like 'foo.so' since 3.2 was released, so we're talking about modules from 3.1 and earlier (or else people who are manually hacking around the compatibility checking system, who can presumably take care of themselves). We've at a minimum been issuing warnings about these modules for 5 versions now (based on Antoine's analysis above), and I'd be really surprised if a module built for 3.1 works on 3.7 anyway. So this change seems pretty reasonable to me. -n
On Sat, Dec 16, 2017, at 08:22, Antoine Pitrou wrote:
Typically, when adding a tp_XXX slot, you also need to add a Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures that have been compiled against a recent enough PyTypeObject definition. This way, extensions compiled against Python N-1 are supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set, the core Python runtime won't try to access the (non-existing) tp_XXX member.
Is there any practical for of having the flag off for one slot and on for another slot that's been added later? Could this be replaced (that is, a slot for such a thing added before it's too late) with a simple counter that goes up with each version, and any "unused" slot should have NULL or some other sentinel value? If it really is important to have the flags themselves, just add another set of flags - Py_TPFLAGS_HAVE_MORE_FLAGS.
On Sun, 17 Dec 2017 21:07:02 -0500 Random832 <random832@fastmail.com> wrote:
Is there any practical for of having the flag off for one slot and on for another slot that's been added later?
Could this be replaced (that is, a slot for such a thing added before it's too late) with a simple counter that goes up with each version, and any "unused" slot should have NULL or some other sentinel value?
Any replacement here would break binary compatibility, which is what those flags are precisely meant to avoid.
If it really is important to have the flags themselves, just add another set of flags - Py_TPFLAGS_HAVE_MORE_FLAGS.
Yes, we could... but it's more complication again. That said, if we decide to drop cross-version compatibility, we then are allowed to enlarge the tp_flags field to, say, 64 bits. Regards Antoine.
On Mon, Dec 18, 2017, at 05:23, Antoine Pitrou wrote:
On Sun, 17 Dec 2017 21:07:02 -0500 Random832 <random832@fastmail.com> wrote:
Is there any practical for of having the flag off for one slot and on for another slot that's been added later?
Could this be replaced (that is, a slot for such a thing added before it's too late) with a simple counter that goes up with each version, and any "unused" slot should have NULL or some other sentinel value?
Any replacement here would break binary compatibility, which is what those flags are precisely meant to avoid.
I meant replacing the mechanism for new fields, rather than existing ones.
If it really is important to have the flags themselves, just add another set of flags - Py_TPFLAGS_HAVE_MORE_FLAGS.
Yes, we could... but it's more complication again.
Hmm, maybe that could be eased with macros... /* Doing this preprocessor trick because if we used a ternary * operator, dummy macros needed to prevent compile errors may * become an attractive nuisance */ #define Py_TPFLAG_CHK(tp, flagname) \ Py__TPFCHK_##flagname(tp, flagname) #define Py__TPFCHK_OLD(tp, flagname) \ ((tp).tp_flags & Py_TPFLAGS_##flagname) #define Py__TPFCHK_NEW(tp, flagname) \ ((tp).tp_flags & Py_TPFLAGS_TPFLAGVER \ && (tp).tp_flagver >= Py_TPFLAGVER_##flagname \ && Py_TPFLAGCHK_##flagname(tp)) #define Py__TPFCHK_HEAPTYPE Py__TPFCHK_OLD #define Py_TPFLAGS_HEAPTYPE (1UL<<9) #define Py__TPFCHK_TPFLAGVER Py__TPFCHK_OLD #define Py_TPFLAGS_TPFLAGVER (1UL<<31) #define Py__TPFCHK_NEWFLD Py__TPFCHK_NEW #define Py_TPFLAGVER_NEWFLD 32 #define Py_TPFLAGCHK_NEWFLD(tp) ((tp).tp_newfld != NULL) So to check heap type you get Py_TPFLAG_CHK(tp, HEAPTYPE) ((tp).tp_flags & (1UL<<9)) And to check newfld1 you get Py_TPFLAG_CHK(tp, NEWFLD) ((tp).tp_flags & (1UL<<31) && (tp).tp_flagver >= 32 && ((tp).tp_newfld != ((void *)0))) Or in a "more flags" scenario: #define Py__TPFCHK_NEW(tp, flagname) \ ((tp).tp_flags & Py_TPFLAGS_TPMOREFLAGS \ && (tp).tp_moreflag & PY_TPMOREFLAGS_##flagname)
Following this discussion, I opened two issues: * https://bugs.python.org/issue32387: "Disallow untagged C extension import on major platforms" * https://bugs.python.org/issue32388: "Remove cross-version binary compatibility" Regards Antoine. On Sat, 16 Dec 2017 14:22:57 +0100 Antoine Pitrou <solipsis@pitrou.net> wrote:
Hello,
Nowadays we have an official mechanism for third-party C extensions to be binary-compatible accross feature releases of Python: the stable ABI.
But, for non-stable ABI-using C extensions, there are also mechanisms in place to *try* and ensure binary compatibility. One of them is the way in which we add tp_ slots to the PyTypeObject structure.
Typically, when adding a tp_XXX slot, you also need to add a Py_TPFLAGS_HAVE_XXX type flag to signal those static type structures that have been compiled against a recent enough PyTypeObject definition. This way, extensions compiled against Python N-1 are supposed to "still work": as they don't have Py_TPFLAGS_HAVE_XXX set, the core Python runtime won't try to access the (non-existing) tp_XXX member.
However, beside internal code complication, it means you need to add a new Py_TPFLAGS_HAVE_XXX each time we add a slot. Since we have only 32 such bits available (many of them already taken), it is a very limited resource. Is it worth it? (*) Can an extension compiled against Python N-1 really claim to be compatible with Python N, despite other possible differences?
(*) we can't extend the tp_flags field to 64 bits, precisely because of the binary compatibility problem...
Regards
Antoine.
participants (4)
-
Antoine Pitrou -
Guido van Rossum -
Nathaniel Smith -
Random832