How have you found compiling in OpenSSL and other libraries into your relocatable builds?
I noticed that https://www.pypy.org/download.html says that "the linux binaries ship an OpenSSL library". Have you found that to be an issue in any way? Have users been accepting of it? I ask as I'm contemplating trying to get relocatable builds for CPython on Linux (to start), and I suspect the OpenSSL question is going to be one of the bigger/stickier discussion points and having some real-world experience/information on the topic would probably help.
I think there are two different topics here: - macOS, windows: PyPy is not doing a good job with the native installers. We should be following CPython and provide real packages, not a tarball. This is issue 3697 [0] for macOS and issue 2894 [5] for windows. - linux: (aarch64, x86_64, i686): we create a portable build of PyPy. It starts in a docker file based on manylinux2014 [1] (note the various install scripts to get updated versions of some libraries). There is an additional wrinkle in that some of the python modules we ship are based on cffi c-extensions (like _ssl, _gdbm, lzma). For these we have an additional build step to download and build the relevant support library on every build [6]. Then when packaging, we use ldd to check dependencies [2], set RPATH appropriately, and bundle those into the tarball [4]. We make sure that not only libpypy3-c.so is checked with ldd, but also the c-extensions [3]. There is a tradeoff between putting libraries in the Docker file and rebuilding the buildbots or building them each time, and openssl is actually on both build paths (probably this could be simplified). CI caching could probably be used to lessen the burden, we have a different build/test workflow from CPython where we typically build only once a day. [0] https://foss.heptapod.net/pypy/pypy/-/issues/3697 [1] https://foss.heptapod.net/pypy/buildbot/-/tree/branch/default/docker [2] https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/... [3] https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/... [4] https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/... [5] https://foss.heptapod.net/pypy/pypy/-/issues/2894 [6] https://foss.heptapod.net/pypy/pypy/-/blob/branch/py3.8/lib_pypy/pypy_tools/... As for complaints about OpenSSL versions: we release 2-3 times a year so can respond "quickly" if security concerns arise. My impression, based on the little feedback we get, is that shipping portable manylinux2014-based builds solves more end user problems than it causes. Sophisticated end users can always rebuild to suit their purposes. Matti On 1/4/22 01:47, matt--- via pypy-dev wrote:
I think there are two different topics here: - macOS, windows: PyPy is not doing a good job with the native installers. We should be following CPython and provide real packages, not a tarball. This is issue 3697 [0] for macOS and issue 2894 [5] for windows. - linux: (aarch64, x86_64, i686): we create a portable build of PyPy. It starts in a docker file based on manylinux2014 [1] (note the various install scripts to get updated versions of some libraries). There is an additional wrinkle in that some of the python modules we ship are based on cffi c-extensions (like _ssl, _gdbm, lzma). For these we have an additional build step to download and build the relevant support library on every build [6]. Then when packaging, we use ldd to check dependencies [2], set RPATH appropriately, and bundle those into the tarball [4]. We make sure that not only libpypy3-c.so is checked with ldd, but also the c-extensions [3]. There is a tradeoff between putting libraries in the Docker file and rebuilding the buildbots or building them each time, and openssl is actually on both build paths (probably this could be simplified). CI caching could probably be used to lessen the burden, we have a different build/test workflow from CPython where we typically build only once a day. [0] https://foss.heptapod.net/pypy/pypy/-/issues/3697 [1] https://foss.heptapod.net/pypy/buildbot/-/tree/branch/default/docker [2] https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/... [3] https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/... [4] https://foss.heptapod.net/pypy/pypy/-/blob/branch/default/pypy/tool/release/... [5] https://foss.heptapod.net/pypy/pypy/-/issues/2894 [6] https://foss.heptapod.net/pypy/pypy/-/blob/branch/py3.8/lib_pypy/pypy_tools/... As for complaints about OpenSSL versions: we release 2-3 times a year so can respond "quickly" if security concerns arise. My impression, based on the little feedback we get, is that shipping portable manylinux2014-based builds solves more end user problems than it causes. Sophisticated end users can always rebuild to suit their purposes. Matti On 1/4/22 01:47, matt--- via pypy-dev wrote:
participants (3)
-
Brett Cannon -
matt＠vazor.com -
Matti Picus