python interpreter docker builder
Hi is there any interest (or anyone has done it before), building the python interpreter using docker? The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows. Thanks
https://devguide.python.org/buildbots/ These run in Docker containers: - https://github.com/python/cpython/blob/master/.travis.yml - https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh These are all of the current builds; are you proposing another? https://github.com/python/buildmaster-config/blob/master/master/custom/build... https://github.com/python/buildmaster-config/blob/master/worker_example.Dock... https://github.com/buildbot/buildbot_travis :
https://github.com/python/core-workflow/issues On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense. Thanks PS. I didn't know about the core workflow, thanks On Thu, 2 Apr 2020 at 19:55, Wes Turner <wes.turner@gmail.com> wrote:
Would e.g. pyinstaller or constructor solve the problem? https://pyinstaller.readthedocs.io/en/stable/
https://github.com/conda/constructor :
Constructor is a tool which allows constructing an installer for a
collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
Yes, I'm aware of those.. providing an application wouldn't be what I have in mind. The point would be providing a python sdk, similar to what node/java/.net provide. something it would possible to build upon. Those are distributed as stand alone "zip" to the general public. On Thu, 2 Apr 2020 at 20:15, Wes Turner <wes.turner@gmail.com> wrote:
To distribute python binaries (interpreter + extensions) on linux one need to compile it using the lowest common denominator (eg. some linux distro really old, like rhel6). Later when a user need to build an extension that might not be possible (because the running host might not have a compatible compiler). So I think a good idea would be starting from the toolchain (eg. in my case I use gcc 9.2.x). These days if I need to use java/node/etc. I won't go through the re-building the interpreter/jvm. Does that make sense? On Thu, 2 Apr 2020 at 20:39, Wes Turner <wes.turner@gmail.com> wrote:
No, I think you might be doing it wrong. FWIW, I haven't needed to compile CPython in 10 years (iot to build something other than CPython) because official builds of Conda, DEB, RPM, or Docker containers are generally sufficient. Are you familiar with the manylinux specs? I should've thought to mention that you can build maximally-portable packages with the manylinux docker containers as your base image: https://github.com/pypa/manylinux https://github.com/pypa/manylinux/blob/master/README.rst#docker-images https://pypi.org/project/auditwheel/ : the PEP 513 manylinux1, PEP 571 manylinux2010 and PEP 599 manylinux2014 platform tags. On Thu, Apr 2, 2020, 8:48 PM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
It should be possible to do rootless manylinux builds in containers with podman/buildah: https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-... Last I checked, rootless within rootless as you'd need for CI builds is problematic due to overlayfs within overlayfs not working. Otherwise, You *can* specify a nonroot --prefix/--installroot to install (build-dep) packages to But then you'd be building a chroot without seccomp, cgroups, namespaces On Thu, Apr 2, 2020, 9:34 PM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
Hi Antonio, could you please spell out your requirements a bit more clearly ? I have a hard time trying to understand what you're after. It seems to be about some sort of Python extension building environment for Linux, but I could be wrong. You normally don't need to build Python itself just to build an extension. If you're looking for a standardized Python build for Linux, PyRun might be what you want: http://pyrun.org/ Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 03 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Hi Marc-Andre, yes pyrun is sort of what I had in mind :) That would be like the jre part of the jdk if you're familiar with java. Ideally users will download the jre and use for running applications (eg. pyrun) but if there's a need for an extension build one need to have a compiler.. now given the plethora of linux distros that's a logistic nightmare. So my little project does that (following the linux from scratch guidelines): it builds the toolchain (minus libc) the compiler and the python interpreter in one giant (at the moment) tarball. It should be easy to extract from it the runtime part (eg. the pyrun equivalent or the miniconda). I hope this makes a bit more sense. On Fri, 3 Apr 2020 at 04:46, M.-A. Lemburg <mal@egenix.com> wrote:
Conda can supply (in principle ) anything. And it certainly supply compilers. And conda-forge has them, at least for some packages. It seems that conda could provide your SDK. And if you want it, you could use constructor to build an installer with everything in one package. -CHB On Fri, Apr 3, 2020 at 6:22 AM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Hi Antonio, you may want to have a look at the Alpine images for Python and this optimized variant: https://github.com/jfloff/alpine-python They also come with dev tools installed. Still, they are overall rather clunky to ship around. On the plus side, you can layer images in docker, so that applications could reuse the base images, once downloaded. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 07 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ On 4/3/2020 3:18 PM, Antonio Cavallo wrote:
On Tuesday, 7 April 2020 10:52:36 CEST M.-A. Lemburg wrote:
It's a bad idea to use alpine with Python. See: - https://github.com/iron-io/dockers/issues/42#issuecomment-290763088 - https://github.com/pypa/manylinux/issues/37 - https://superuser.com/questions/1219609/why-is-the-alpine-docker-image-over-...
Re: manylinux wheels in Alpine docker (musl (with glibc optionally also installed)) "Clarify that pip wheels are incompatible with alpine-based images" https://github.com/docker-library/docs/issues/904 "Install manylinux wheel although it's not officially supported by the host" https://github.com/pypa/pip/issues/3969#issuecomment-495991727 :
```bash RUN echo 'manylinux1_compatible = True' >
/usr/local/lib/python3.7/site-packages/_manylinux.py
RUN python -c 'import sys; sys.path.append(r"/_manylinux.py")' ```
On Tue, Apr 7, 2020, 10:58 AM João Santos <jmcs@jsantos.eu> wrote:
https://devguide.python.org/buildbots/ These run in Docker containers: - https://github.com/python/cpython/blob/master/.travis.yml - https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh These are all of the current builds; are you proposing another? https://github.com/python/buildmaster-config/blob/master/master/custom/build... https://github.com/python/buildmaster-config/blob/master/worker_example.Dock... https://github.com/buildbot/buildbot_travis :
https://github.com/python/core-workflow/issues On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense. Thanks PS. I didn't know about the core workflow, thanks On Thu, 2 Apr 2020 at 19:55, Wes Turner <wes.turner@gmail.com> wrote:
Would e.g. pyinstaller or constructor solve the problem? https://pyinstaller.readthedocs.io/en/stable/
https://github.com/conda/constructor :
Constructor is a tool which allows constructing an installer for a
collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
Yes, I'm aware of those.. providing an application wouldn't be what I have in mind. The point would be providing a python sdk, similar to what node/java/.net provide. something it would possible to build upon. Those are distributed as stand alone "zip" to the general public. On Thu, 2 Apr 2020 at 20:15, Wes Turner <wes.turner@gmail.com> wrote:
To distribute python binaries (interpreter + extensions) on linux one need to compile it using the lowest common denominator (eg. some linux distro really old, like rhel6). Later when a user need to build an extension that might not be possible (because the running host might not have a compatible compiler). So I think a good idea would be starting from the toolchain (eg. in my case I use gcc 9.2.x). These days if I need to use java/node/etc. I won't go through the re-building the interpreter/jvm. Does that make sense? On Thu, 2 Apr 2020 at 20:39, Wes Turner <wes.turner@gmail.com> wrote:
No, I think you might be doing it wrong. FWIW, I haven't needed to compile CPython in 10 years (iot to build something other than CPython) because official builds of Conda, DEB, RPM, or Docker containers are generally sufficient. Are you familiar with the manylinux specs? I should've thought to mention that you can build maximally-portable packages with the manylinux docker containers as your base image: https://github.com/pypa/manylinux https://github.com/pypa/manylinux/blob/master/README.rst#docker-images https://pypi.org/project/auditwheel/ : the PEP 513 manylinux1, PEP 571 manylinux2010 and PEP 599 manylinux2014 platform tags. On Thu, Apr 2, 2020, 8:48 PM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
It should be possible to do rootless manylinux builds in containers with podman/buildah: https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-... Last I checked, rootless within rootless as you'd need for CI builds is problematic due to overlayfs within overlayfs not working. Otherwise, You *can* specify a nonroot --prefix/--installroot to install (build-dep) packages to But then you'd be building a chroot without seccomp, cgroups, namespaces On Thu, Apr 2, 2020, 9:34 PM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
Hi Antonio, could you please spell out your requirements a bit more clearly ? I have a hard time trying to understand what you're after. It seems to be about some sort of Python extension building environment for Linux, but I could be wrong. You normally don't need to build Python itself just to build an extension. If you're looking for a standardized Python build for Linux, PyRun might be what you want: http://pyrun.org/ Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 03 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Hi Marc-Andre, yes pyrun is sort of what I had in mind :) That would be like the jre part of the jdk if you're familiar with java. Ideally users will download the jre and use for running applications (eg. pyrun) but if there's a need for an extension build one need to have a compiler.. now given the plethora of linux distros that's a logistic nightmare. So my little project does that (following the linux from scratch guidelines): it builds the toolchain (minus libc) the compiler and the python interpreter in one giant (at the moment) tarball. It should be easy to extract from it the runtime part (eg. the pyrun equivalent or the miniconda). I hope this makes a bit more sense. On Fri, 3 Apr 2020 at 04:46, M.-A. Lemburg <mal@egenix.com> wrote:
Conda can supply (in principle ) anything. And it certainly supply compilers. And conda-forge has them, at least for some packages. It seems that conda could provide your SDK. And if you want it, you could use constructor to build an installer with everything in one package. -CHB On Fri, Apr 3, 2020 at 6:22 AM Antonio Cavallo <antonio.cavallo.71@gmail.com> wrote:
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Hi Antonio, you may want to have a look at the Alpine images for Python and this optimized variant: https://github.com/jfloff/alpine-python They also come with dev tools installed. Still, they are overall rather clunky to ship around. On the plus side, you can layer images in docker, so that applications could reuse the base images, once downloaded. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 07 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ On 4/3/2020 3:18 PM, Antonio Cavallo wrote:
On Tuesday, 7 April 2020 10:52:36 CEST M.-A. Lemburg wrote:
It's a bad idea to use alpine with Python. See: - https://github.com/iron-io/dockers/issues/42#issuecomment-290763088 - https://github.com/pypa/manylinux/issues/37 - https://superuser.com/questions/1219609/why-is-the-alpine-docker-image-over-...
Re: manylinux wheels in Alpine docker (musl (with glibc optionally also installed)) "Clarify that pip wheels are incompatible with alpine-based images" https://github.com/docker-library/docs/issues/904 "Install manylinux wheel although it's not officially supported by the host" https://github.com/pypa/pip/issues/3969#issuecomment-495991727 :
```bash RUN echo 'manylinux1_compatible = True' >
/usr/local/lib/python3.7/site-packages/_manylinux.py
RUN python -c 'import sys; sys.path.append(r"/_manylinux.py")' ```
On Tue, Apr 7, 2020, 10:58 AM João Santos <jmcs@jsantos.eu> wrote:
participants (5)
-
Antonio Cavallo -
Christopher Barker -
João Santos -
M.-A. Lemburg -
Wes Turner