On Sun, 2019-10-20 at 07:32 +0300, Matti Picus wrote:
Hello,
I've noticed that the compiled module suffix keeps changing between PyPy3 releases: it's been .pypy3-71-*.so for 7.1, now it's .pypy3-72- *.so (also .pyc). However, this is a bit surprising to me given that for PyPy2 it's still at .pypy-41.so.
Is the bytecode generated by successive PyPy3 releases really incompatible between them? Or are the suffix changes only incidental? They cause quite some trouble for us, since they make it necessary to recompile installed modules on Gentoo, and PyPy's overzealous compiling causes access violations for our users.
TIA for any help. For c-extensions the major-minor version becomes the PEP 425 ABI tag https://www.python.org/dev/peps/pep-0425/#abi-tag. My operating assumption is that any change in the ABI requires a change in that field. Our support for the massive CPython C-API changes between versions, which changes the ABI. In particular, between 7.1 and 7.2 the PyDateTime_CAPI structure size changed, and we changed the order of fields in Py_Buffer, as well as adding many missing functions and macros. The complete log is here http://doc.pypy.org/en/latest/release-v7.2.0.html#c-api-cpyext-and-c-extensi... for both versions and here http://doc.pypy.org/en/latest/release-v7.2.0.html#python-3-6-c-api for
On 19/10/19 11:55 pm, Michał Górny wrote: the pypy3.6 specific ones.
The fact that the PyPy2 ABI tag did **not** change is most likely a bug in the release process. I think there will be cases where pypy2-v7.1.1 and pypy2-v7.2.0 c-extension modules will be slightly incompatible with eachother, although a cursory test of mixing NumPy across versions (building on 7.2, copying into 7.1) seems to pass "np.test()". Perhaps a better test would be to mix the pygolang c-extension modules, which seem to stress-test the C-API more extensively given the number of issues it exposes.
Thanks for the explanation. This answers my question. Do you need me to file a bug for the ABI tag not changing in PyPy2, or will you take it from here?
The pyc files should always be rebuilt in each python environment, so I am not sure what problems could be caused by bumping the ABI tag. Does Gentoo somehow mix the byte-compiled pyc files across versions?
I am not sure what you mean by "compiling causes access violations for our users", could you point to a discussion of the problem?
It's related to circular dependencies between packages. For example, when rebuild setuptools for the new version, it tries to load its plugins and byte-compile them as well. Since they don't belong to the setuptools package, our PM catches that as illegal access. It's a generic problem with Python, not something you need to worry about. I've just been pointed out that we already have a hack for it, I just need to add PyPy3 to it. -- Best regards, Michał Górny