[pypy-dev] PyPy3: is bytecode really incompatible between releases?

[Matti Picus]

On 19/10/19 11:55 pm, Michał Górny wrote:
> Hello,
>
> I've noticed that the compiled module suffix keeps changing between
> PyPy3 releases: it's been .pypy3-71-*.so for 7.1, now it's .pypy3-72-
> *.so (also .pyc).  However, this is a bit surprising to me given that
> for PyPy2 it's still at .pypy-41.so.
>
> Is the bytecode generated by successive PyPy3 releases really
> incompatible between them?  Or are the suffix changes only incidental?
> They cause quite some trouble for us, since they make it necessary to
> recompile installed modules on Gentoo, and PyPy's overzealous compiling
> causes access violations for our users.
>
> TIA for any help.
For c-extensions the major-minor version becomes the PEP 425 ABI tag 
https://www.python.org/dev/peps/pep-0425/#abi-tag. My operating 
assumption is that any change in the ABI requires a change in that 
field. Our support for the massive CPython C-API changes between 
versions, which changes the ABI. In particular, between 7.1 and 7.2 the 
PyDateTime_CAPI structure size changed, and we changed the order of 
fields in Py_Buffer, as well as adding many missing functions and 
macros. The complete log is here 
http://doc.pypy.org/en/latest/release-v7.2.0.html#c-api-cpyext-and-c-extensions 
for both versions and here 
http://doc.pypy.org/en/latest/release-v7.2.0.html#python-3-6-c-api for 
the pypy3.6 specific ones.


The fact that the PyPy2 ABI tag did **not** change is most likely a bug 
in the release process. I think there will be cases where pypy2-v7.1.1 
and pypy2-v7.2.0 c-extension modules will be slightly incompatible with 
eachother, although a cursory test of mixing NumPy across versions 
(building on 7.2, copying into 7.1) seems to pass "np.test()". Perhaps a 
better test would be to mix the pygolang c-extension modules, which seem 
to stress-test the C-API more extensively given the number of issues it 
exposes.


The pyc files should always be rebuilt in each python environment, so I 
am not sure what problems could be caused by bumping the ABI tag. Does 
Gentoo somehow mix the byte-compiled pyc files across versions?


I am not sure what you mean by "compiling causes access violations for 
our users", could you point to a discussion of the problem?

Matti