[pypy-dev] PyPy3: is bytecode really incompatible between releases?

[Michał Górny]

On Sun, 2019-10-20 at 07:32 +0300, Matti Picus wrote:
> On 19/10/19 11:55 pm, Michał Górny wrote:
> > Hello,
> > 
> > I've noticed that the compiled module suffix keeps changing between
> > PyPy3 releases: it's been .pypy3-71-*.so for 7.1, now it's .pypy3-72-
> > *.so (also .pyc).  However, this is a bit surprising to me given that
> > for PyPy2 it's still at .pypy-41.so.
> > 
> > Is the bytecode generated by successive PyPy3 releases really
> > incompatible between them?  Or are the suffix changes only incidental?
> > They cause quite some trouble for us, since they make it necessary to
> > recompile installed modules on Gentoo, and PyPy's overzealous compiling
> > causes access violations for our users.
> > 
> > TIA for any help.
> For c-extensions the major-minor version becomes the PEP 425 ABI tag 
> https://www.python.org/dev/peps/pep-0425/#abi-tag. My operating 
> assumption is that any change in the ABI requires a change in that 
> field. Our support for the massive CPython C-API changes between 
> versions, which changes the ABI. In particular, between 7.1 and 7.2 the 
> PyDateTime_CAPI structure size changed, and we changed the order of 
> fields in Py_Buffer, as well as adding many missing functions and 
> macros. The complete log is here 
> http://doc.pypy.org/en/latest/release-v7.2.0.html#c-api-cpyext-and-c-extensions 
> for both versions and here 
> http://doc.pypy.org/en/latest/release-v7.2.0.html#python-3-6-c-api for 
> the pypy3.6 specific ones.
> 
> 
> The fact that the PyPy2 ABI tag did **not** change is most likely a bug 
> in the release process. I think there will be cases where pypy2-v7.1.1 
> and pypy2-v7.2.0 c-extension modules will be slightly incompatible with 
> eachother, although a cursory test of mixing NumPy across versions 
> (building on 7.2, copying into 7.1) seems to pass "np.test()". Perhaps a 
> better test would be to mix the pygolang c-extension modules, which seem 
> to stress-test the C-API more extensively given the number of issues it 
> exposes.

Thanks for the explanation.  This answers my question.  Do you need me
to file a bug for the ABI tag not changing in PyPy2, or will you take it
from here?

> The pyc files should always be rebuilt in each python environment, so I 
> am not sure what problems could be caused by bumping the ABI tag. Does 
> Gentoo somehow mix the byte-compiled pyc files across versions?
> 
> 
> I am not sure what you mean by "compiling causes access violations for 
> our users", could you point to a discussion of the problem?
> 

It's related to circular dependencies between packages.  For example,
when rebuild setuptools for the new version, it tries to load its
plugins and byte-compile them as well.  Since they don't belong to
the setuptools package, our PM catches that as illegal access.

It's a generic problem with Python, not something you need to worry
about.  I've just been pointed out that we already have a hack for it,
I just need to add PyPy3 to it.

-- 
Best regards,
Michał Górny

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 618 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20191020/ef394b6d/attachment.sig>