[Python-Dev] Usefulness of binary compatibility accross Python versions?

[Nathaniel Smith]

On Dec 16, 2017 11:44 AM, "Guido van Rossum" <guido at python.org> wrote:

On Sat, Dec 16, 2017 at 11:14 AM, Antoine Pitrou <solipsis at pitrou.net>
wrote:

> On Sat, 16 Dec 2017 19:37:54 +0100
> Antoine Pitrou <solipsis at pitrou.net> wrote:
> >
> > Currently, you can pass a `module_api_version` to PyModule_Create2(),
> > but that function is for specialists only :-)
> >
> > ("""Most uses of this function should be using PyModule_Create()
> > instead; only use this if you are sure you need it.""")
>
> Ah, it turns out I misunderstood that piece of documentation and also
> what PEP 3121 really did w.r.t the module API check.
>
> PyModule_Create() is actually a *macro* calling PyModule_Create2() with
> the version number is was compiled against!
>
> #ifdef Py_LIMITED_API
> #define PyModule_Create(module) \
>         PyModule_Create2(module, PYTHON_ABI_VERSION)
> #else
> #define PyModule_Create(module) \
>         PyModule_Create2(module, PYTHON_API_VERSION)
> #endif
>
> And there's already a check for that version number in moduleobject.c:
> https://github.com/python/cpython/blob/master/Objects/moduleobject.c#L114
>
> That check is always invoked when calling PyModule_Create() and
> PyModule_Create2().  Currently it merely invokes a warning, but we can
> easily turn that into an error.
>
> (with apologies to Martin von Löwis for not fully understanding what he
> did at the time :-))
>

If it's only a warning, I worry that if we stop checking the flag bits it
can cause wild pointer following. This sounds like it would be a potential
security issue (load a module, ignore the warning, try to use a certain API
on a class it defines, boom). Also, could there still be 3rd party modules
out there that haven't been recompiled in a really long time and use some
older backwards compatible module initialization API? (I guess we could
stop supporting that and let them fail hard.)


I think there's a pretty simple way to avoid this kind of problem.

Since PEP 3149 (Python 3.2), the import system has (IIUC) checked for:

foo.cpython-XYm.so
foo.abi3.so
foo.so

If we drop foo.so from this list, then we're pretty much guaranteed not to
load anything into a python that it wasn't intended for.

How disruptive would this be? AFAICT there hasn't been any standard way to
build python extensions named like 'foo.so' since 3.2 was released, so
we're talking about modules from 3.1 and earlier (or else people who are
manually hacking around the compatibility checking system, who can
presumably take care of themselves). We've at a minimum been issuing
warnings about these modules for 5 versions now (based on Antoine's
analysis above), and I'd be really surprised if a module built for 3.1
works on 3.7 anyway. So this change seems pretty reasonable to me.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20171217/31e4ddbf/attachment.html>