python interpreter docker builder
Hi is there any interest (or anyone has done it before), building the python interpreter using docker?
The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows.
Thanks
https://devguide.python.org/buildbots/
These run in Docker containers: - https://github.com/python/cpython/blob/master/.travis.yml - https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh
These are all of the current builds; are you proposing another? https://github.com/python/buildmaster-config/blob/master/master/custom/build... https://github.com/python/buildmaster-config/blob/master/worker_example.Dock...
https://github.com/buildbot/buildbot_travis :
Basically we provide a compatibility shim in buildbot that allows it to
consume a .travis.yml file.
buildbot_travis does however not support the full .travis.yml format.
https://github.com/python/core-workflow/issues
On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
Hi is there any interest (or anyone has done it before), building the python interpreter using docker?
The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows.
Thanks _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... Code of Conduct: http://python.org/psf/codeofconduct/
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense.
Thanks
PS. I didn't know about the core workflow, thanks
On Thu, 2 Apr 2020 at 19:55, Wes Turner wes.turner@gmail.com wrote:
https://devguide.python.org/buildbots/
These run in Docker containers:
https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh
These are all of the current builds; are you proposing another?
https://github.com/python/buildmaster-config/blob/master/master/custom/build...
https://github.com/python/buildmaster-config/blob/master/worker_example.Dock...
https://github.com/buildbot/buildbot_travis :
Basically we provide a compatibility shim in buildbot that allows it to
consume a .travis.yml file.
buildbot_travis does however not support the full .travis.yml format.
https://github.com/python/core-workflow/issues
On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
Hi is there any interest (or anyone has done it before), building the python interpreter using docker?
The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows.
Thanks _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... Code of Conduct: http://python.org/psf/codeofconduct/
Would e.g. pyinstaller or constructor solve the problem?
https://pyinstaller.readthedocs.io/en/stable/
PyInstaller bundles a Python application and all its dependencies into a
single package. The user can run the packaged app without installing a Python interpreter or any modules. PyInstaller supports Python 2.7 and Python 3.5+, and correctly bundles the major Python packages such as numpy, PyQt, Django, wxPython, and others.
PyInstaller is tested against Windows, Mac OS X, and GNU/Linux. However,
it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD but testing against them is not part of our continuous integration tests.
https://github.com/conda/constructor :
Constructor is a tool which allows constructing an installer for a
collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython
On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense.
Thanks
PS. I didn't know about the core workflow, thanks
On Thu, 2 Apr 2020 at 19:55, Wes Turner wes.turner@gmail.com wrote:
https://devguide.python.org/buildbots/
These run in Docker containers:
https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh
These are all of the current builds; are you proposing another?
https://github.com/python/buildmaster-config/blob/master/master/custom/build...
https://github.com/python/buildmaster-config/blob/master/worker_example.Dock...
https://github.com/buildbot/buildbot_travis :
Basically we provide a compatibility shim in buildbot that allows it to
consume a .travis.yml file.
buildbot_travis does however not support the full .travis.yml format.
https://github.com/python/core-workflow/issues
On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Hi is there any interest (or anyone has done it before), building the python interpreter using docker?
The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows.
Thanks _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... Code of Conduct: http://python.org/psf/codeofconduct/
Yes, I'm aware of those.. providing an application wouldn't be what I have in mind.
The point would be providing a python sdk, similar to what node/java/.net provide. something it would possible to build upon. Those are distributed as stand alone "zip" to the general public.
On Thu, 2 Apr 2020 at 20:15, Wes Turner wes.turner@gmail.com wrote:
Would e.g. pyinstaller or constructor solve the problem?
https://pyinstaller.readthedocs.io/en/stable/
PyInstaller bundles a Python application and all its dependencies into a
single package. The user can run the packaged app without installing a Python interpreter or any modules. PyInstaller supports Python 2.7 and Python 3.5+, and correctly bundles the major Python packages such as numpy, PyQt, Django, wxPython, and others.
PyInstaller is tested against Windows, Mac OS X, and GNU/Linux. However,
it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD but testing against them is not part of our continuous integration tests.
https://github.com/conda/constructor :
Constructor is a tool which allows constructing an installer for a
collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython
On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense.
Thanks
PS. I didn't know about the core workflow, thanks
On Thu, 2 Apr 2020 at 19:55, Wes Turner wes.turner@gmail.com wrote:
https://devguide.python.org/buildbots/
These run in Docker containers:
https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh
These are all of the current builds; are you proposing another?
https://github.com/python/buildmaster-config/blob/master/master/custom/build...
https://github.com/python/buildmaster-config/blob/master/worker_example.Dock...
https://github.com/buildbot/buildbot_travis :
Basically we provide a compatibility shim in buildbot that allows it
to consume a .travis.yml file.
buildbot_travis does however not support the full .travis.yml format.
https://github.com/python/core-workflow/issues
On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Hi is there any interest (or anyone has done it before), building the python interpreter using docker?
The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows.
Thanks _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... Code of Conduct: http://python.org/psf/codeofconduct/
So you want to import from / call into what would be in $prefix/lib & $prefix/include?
How is the SDK archive use case different from the package archive use case?
On Thu, Apr 2, 2020, 8:32 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
Yes, I'm aware of those.. providing an application wouldn't be what I have in mind.
The point would be providing a python sdk, similar to what node/java/.net provide. something it would possible to build upon. Those are distributed as stand alone "zip" to the general public.
On Thu, 2 Apr 2020 at 20:15, Wes Turner wes.turner@gmail.com wrote:
Would e.g. pyinstaller or constructor solve the problem?
https://pyinstaller.readthedocs.io/en/stable/
PyInstaller bundles a Python application and all its dependencies into
a single package. The user can run the packaged app without installing a Python interpreter or any modules. PyInstaller supports Python 2.7 and Python 3.5+, and correctly bundles the major Python packages such as numpy, PyQt, Django, wxPython, and others.
PyInstaller is tested against Windows, Mac OS X, and GNU/Linux.
However, it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD but testing against them is not part of our continuous integration tests.
https://github.com/conda/constructor :
Constructor is a tool which allows constructing an installer for a
collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython
On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense.
Thanks
PS. I didn't know about the core workflow, thanks
On Thu, 2 Apr 2020 at 19:55, Wes Turner wes.turner@gmail.com wrote:
https://devguide.python.org/buildbots/
These run in Docker containers:
https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh
These are all of the current builds; are you proposing another?
https://github.com/python/buildmaster-config/blob/master/master/custom/build...
https://github.com/python/buildmaster-config/blob/master/worker_example.Dock...
https://github.com/buildbot/buildbot_travis :
Basically we provide a compatibility shim in buildbot that allows it
to consume a .travis.yml file.
buildbot_travis does however not support the full .travis.yml format.
https://github.com/python/core-workflow/issues
On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Hi is there any interest (or anyone has done it before), building the python interpreter using docker?
The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows.
Thanks _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... Code of Conduct: http://python.org/psf/codeofconduct/
To distribute python binaries (interpreter + extensions) on linux one need to compile it using the lowest common denominator (eg. some linux distro really old, like rhel6). Later when a user need to build an extension that might not be possible (because the running host might not have a compatible compiler).
So I think a good idea would be starting from the toolchain (eg. in my case I use gcc 9.2.x).
These days if I need to use java/node/etc. I won't go through the re-building the interpreter/jvm. Does that make sense?
On Thu, 2 Apr 2020 at 20:39, Wes Turner wes.turner@gmail.com wrote:
So you want to import from / call into what would be in $prefix/lib & $prefix/include?
How is the SDK archive use case different from the package archive use case?
On Thu, Apr 2, 2020, 8:32 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
Yes, I'm aware of those.. providing an application wouldn't be what I have in mind.
The point would be providing a python sdk, similar to what node/java/.net provide. something it would possible to build upon. Those are distributed as stand alone "zip" to the general public.
On Thu, 2 Apr 2020 at 20:15, Wes Turner wes.turner@gmail.com wrote:
Would e.g. pyinstaller or constructor solve the problem?
https://pyinstaller.readthedocs.io/en/stable/
PyInstaller bundles a Python application and all its dependencies into
a single package. The user can run the packaged app without installing a Python interpreter or any modules. PyInstaller supports Python 2.7 and Python 3.5+, and correctly bundles the major Python packages such as numpy, PyQt, Django, wxPython, and others.
PyInstaller is tested against Windows, Mac OS X, and GNU/Linux.
However, it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD but testing against them is not part of our continuous integration tests.
https://github.com/conda/constructor :
Constructor is a tool which allows constructing an installer for a
collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython
On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense.
Thanks
PS. I didn't know about the core workflow, thanks
On Thu, 2 Apr 2020 at 19:55, Wes Turner wes.turner@gmail.com wrote:
https://devguide.python.org/buildbots/
These run in Docker containers:
https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh
These are all of the current builds; are you proposing another?
https://github.com/python/buildmaster-config/blob/master/master/custom/build...
https://github.com/python/buildmaster-config/blob/master/worker_example.Dock...
https://github.com/buildbot/buildbot_travis :
Basically we provide a compatibility shim in buildbot that allows it
to consume a .travis.yml file.
buildbot_travis does however not support the full .travis.yml format.
https://github.com/python/core-workflow/issues
On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Hi is there any interest (or anyone has done it before), building the python interpreter using docker?
The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows.
Thanks _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... Code of Conduct: http://python.org/psf/codeofconduct/
No, I think you might be doing it wrong. FWIW, I haven't needed to compile CPython in 10 years (iot to build something other than CPython) because official builds of Conda, DEB, RPM, or Docker containers are generally sufficient.
Are you familiar with the manylinux specs? I should've thought to mention that you can build maximally-portable packages with the manylinux docker containers as your base image: https://github.com/pypa/manylinux
https://github.com/pypa/manylinux/blob/master/README.rst#docker-images
https://pypi.org/project/auditwheel/ :
auditwheel is a command line tool to facilitate the creation of Python
wheel packages for Linux (containing pre-compiled binary extensions) that are compatible with a wide variety of Linux distributions, consistent with the PEP 513 manylinux1, PEP 571 manylinux2010 and PEP 599 manylinux2014 platform tags.
On Thu, Apr 2, 2020, 8:48 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
To distribute python binaries (interpreter + extensions) on linux one need to compile it using the lowest common denominator (eg. some linux distro really old, like rhel6). Later when a user need to build an extension that might not be possible (because the running host might not have a compatible compiler).
So I think a good idea would be starting from the toolchain (eg. in my case I use gcc 9.2.x).
These days if I need to use java/node/etc. I won't go through the re-building the interpreter/jvm. Does that make sense?
On Thu, 2 Apr 2020 at 20:39, Wes Turner wes.turner@gmail.com wrote:
So you want to import from / call into what would be in $prefix/lib & $prefix/include?
How is the SDK archive use case different from the package archive use case?
On Thu, Apr 2, 2020, 8:32 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Yes, I'm aware of those.. providing an application wouldn't be what I have in mind.
The point would be providing a python sdk, similar to what node/java/.net provide. something it would possible to build upon. Those are distributed as stand alone "zip" to the general public.
On Thu, 2 Apr 2020 at 20:15, Wes Turner wes.turner@gmail.com wrote:
Would e.g. pyinstaller or constructor solve the problem?
https://pyinstaller.readthedocs.io/en/stable/
PyInstaller bundles a Python application and all its dependencies
into a single package. The user can run the packaged app without installing a Python interpreter or any modules. PyInstaller supports Python 2.7 and Python 3.5+, and correctly bundles the major Python packages such as numpy, PyQt, Django, wxPython, and others.
PyInstaller is tested against Windows, Mac OS X, and GNU/Linux.
However, it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD but testing against them is not part of our continuous integration tests.
https://github.com/conda/constructor :
Constructor is a tool which allows constructing an installer for a
collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython
On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense.
Thanks
PS. I didn't know about the core workflow, thanks
On Thu, 2 Apr 2020 at 19:55, Wes Turner wes.turner@gmail.com wrote:
https://devguide.python.org/buildbots/
These run in Docker containers:
https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh
These are all of the current builds; are you proposing another?
https://github.com/python/buildmaster-config/blob/master/master/custom/build...
https://github.com/python/buildmaster-config/blob/master/worker_example.Dock...
https://github.com/buildbot/buildbot_travis :
> Basically we provide a compatibility shim in buildbot that allows it to consume a .travis.yml file. > > buildbot_travis does however not support the full .travis.yml format.
https://github.com/python/core-workflow/issues
On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
> Hi > is there any interest (or anyone has done it before), building the > python interpreter using docker? > > The basic idea is building the toolchain (gcc) and on top of that > the python interpreter. On mac/linux it can build natively, but it can use > docker to target linux from mac/windows. > > Thanks > _______________________________________________ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-leave@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... > Code of Conduct: http://python.org/psf/codeofconduct/ >
Probably, but I'm still convinced a sdk would be nice.
Yes, I'm aware of manylinux. DEB/RP suffer from the fact that need to be "root" installed and manylinux needs docker to be used, so same issue.
Thanks
On Thu, 2 Apr 2020 at 20:55, Wes Turner wes.turner@gmail.com wrote:
No, I think you might be doing it wrong. FWIW, I haven't needed to compile CPython in 10 years (iot to build something other than CPython) because official builds of Conda, DEB, RPM, or Docker containers are generally sufficient.
Are you familiar with the manylinux specs? I should've thought to mention that you can build maximally-portable packages with the manylinux docker containers as your base image: https://github.com/pypa/manylinux
https://github.com/pypa/manylinux/blob/master/README.rst#docker-images
https://pypi.org/project/auditwheel/ :
auditwheel is a command line tool to facilitate the creation of Python
wheel packages for Linux (containing pre-compiled binary extensions) that are compatible with a wide variety of Linux distributions, consistent with the PEP 513 manylinux1, PEP 571 manylinux2010 and PEP 599 manylinux2014 platform tags.
On Thu, Apr 2, 2020, 8:48 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
To distribute python binaries (interpreter + extensions) on linux one need to compile it using the lowest common denominator (eg. some linux distro really old, like rhel6). Later when a user need to build an extension that might not be possible (because the running host might not have a compatible compiler).
So I think a good idea would be starting from the toolchain (eg. in my case I use gcc 9.2.x).
These days if I need to use java/node/etc. I won't go through the re-building the interpreter/jvm. Does that make sense?
On Thu, 2 Apr 2020 at 20:39, Wes Turner wes.turner@gmail.com wrote:
So you want to import from / call into what would be in $prefix/lib & $prefix/include?
How is the SDK archive use case different from the package archive use case?
On Thu, Apr 2, 2020, 8:32 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Yes, I'm aware of those.. providing an application wouldn't be what I have in mind.
The point would be providing a python sdk, similar to what node/java/.net provide. something it would possible to build upon. Those are distributed as stand alone "zip" to the general public.
On Thu, 2 Apr 2020 at 20:15, Wes Turner wes.turner@gmail.com wrote:
Would e.g. pyinstaller or constructor solve the problem?
https://pyinstaller.readthedocs.io/en/stable/
PyInstaller bundles a Python application and all its dependencies
into a single package. The user can run the packaged app without installing a Python interpreter or any modules. PyInstaller supports Python 2.7 and Python 3.5+, and correctly bundles the major Python packages such as numpy, PyQt, Django, wxPython, and others.
PyInstaller is tested against Windows, Mac OS X, and GNU/Linux.
However, it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD but testing against them is not part of our continuous integration tests.
https://github.com/conda/constructor :
Constructor is a tool which allows constructing an installer for a
collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython
On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Not quite, my hope is to have a python tarball similar to the "Windows x86 embeddable zip file" but for linux. Similar to miniconda but for plain python, or sort of python "sdk", if that makes sense.
Thanks
PS. I didn't know about the core workflow, thanks
On Thu, 2 Apr 2020 at 19:55, Wes Turner wes.turner@gmail.com wrote:
> https://devguide.python.org/buildbots/ > > These run in Docker containers: > - https://github.com/python/cpython/blob/master/.travis.yml > - > https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh > > These are all of the current builds; are you proposing another? > > https://github.com/python/buildmaster-config/blob/master/master/custom/build... > > https://github.com/python/buildmaster-config/blob/master/worker_example.Dock... > > https://github.com/buildbot/buildbot_travis : > > > Basically we provide a compatibility shim in buildbot that allows > it to consume a .travis.yml file. > > > > buildbot_travis does however not support the full .travis.yml > format. > > https://github.com/python/core-workflow/issues > > > On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo < > antonio.cavallo.71@gmail.com> wrote: > >> Hi >> is there any interest (or anyone has done it before), building the >> python interpreter using docker? >> >> The basic idea is building the toolchain (gcc) and on top of that >> the python interpreter. On mac/linux it can build natively, but it can use >> docker to target linux from mac/windows. >> >> Thanks >> _______________________________________________ >> Python-ideas mailing list -- python-ideas@python.org >> To unsubscribe send an email to python-ideas-leave@python.org >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >
It should be possible to do rootless manylinux builds in containers with podman/buildah: https://developers.redhat.com/blog/2019/02/21/podman-and-buildah-for-docker-...
Last I checked, rootless within rootless as you'd need for CI builds is problematic due to overlayfs within overlayfs not working.
Otherwise, You *can* specify a nonroot --prefix/--installroot to install (build-dep) packages to
But then you'd be building a chroot without seccomp, cgroups, namespaces
On Thu, Apr 2, 2020, 9:34 PM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
Probably, but I'm still convinced a sdk would be nice.
Yes, I'm aware of manylinux. DEB/RP suffer from the fact that need to be "root" installed and manylinux needs docker to be used, so same issue.
Thanks
On Thu, 2 Apr 2020 at 20:55, Wes Turner wes.turner@gmail.com wrote:
No, I think you might be doing it wrong. FWIW, I haven't needed to compile CPython in 10 years (iot to build something other than CPython) because official builds of Conda, DEB, RPM, or Docker containers are generally sufficient.
Are you familiar with the manylinux specs? I should've thought to mention that you can build maximally-portable packages with the manylinux docker containers as your base image: https://github.com/pypa/manylinux
https://github.com/pypa/manylinux/blob/master/README.rst#docker-images
https://pypi.org/project/auditwheel/ :
auditwheel is a command line tool to facilitate the creation of Python
wheel packages for Linux (containing pre-compiled binary extensions) that are compatible with a wide variety of Linux distributions, consistent with the PEP 513 manylinux1, PEP 571 manylinux2010 and PEP 599 manylinux2014 platform tags.
On Thu, Apr 2, 2020, 8:48 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
To distribute python binaries (interpreter + extensions) on linux one need to compile it using the lowest common denominator (eg. some linux distro really old, like rhel6). Later when a user need to build an extension that might not be possible (because the running host might not have a compatible compiler).
So I think a good idea would be starting from the toolchain (eg. in my case I use gcc 9.2.x).
These days if I need to use java/node/etc. I won't go through the re-building the interpreter/jvm. Does that make sense?
On Thu, 2 Apr 2020 at 20:39, Wes Turner wes.turner@gmail.com wrote:
So you want to import from / call into what would be in $prefix/lib & $prefix/include?
How is the SDK archive use case different from the package archive use case?
On Thu, Apr 2, 2020, 8:32 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
Yes, I'm aware of those.. providing an application wouldn't be what I have in mind.
The point would be providing a python sdk, similar to what node/java/.net provide. something it would possible to build upon. Those are distributed as stand alone "zip" to the general public.
On Thu, 2 Apr 2020 at 20:15, Wes Turner wes.turner@gmail.com wrote:
Would e.g. pyinstaller or constructor solve the problem?
https://pyinstaller.readthedocs.io/en/stable/
> PyInstaller bundles a Python application and all its dependencies into a single package. The user can run the packaged app without installing a Python interpreter or any modules. PyInstaller supports Python 2.7 and Python 3.5+, and correctly bundles the major Python packages such as numpy, PyQt, Django, wxPython, and others. > > PyInstaller is tested against Windows, Mac OS X, and GNU/Linux. However, it is not a cross-compiler: to make a Windows app you run PyInstaller in Windows; to make a GNU/Linux app you run it in GNU/Linux, etc. PyInstaller has been used successfully with AIX, Solaris, FreeBSD and OpenBSD but testing against them is not part of our continuous integration tests.
https://github.com/conda/constructor :
> Constructor is a tool which allows constructing an installer for a collection of conda packages. It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell installers, MacOS .pkg installers, and Windows .exe installers. Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.
One advantage of ~ 'dynamic linking' / not shipping the python binary is that you then don't need to sign and distribute new releases for every minor release of cpython
On Thu, Apr 2, 2020, 8:09 PM Antonio Cavallo < antonio.cavallo.71@gmail.com> wrote:
> Not quite, my hope is to have a python tarball similar to the > "Windows x86 embeddable zip file" but for linux. > Similar to miniconda but for plain python, or sort of python "sdk", > if that makes sense. > > Thanks > > PS. I didn't know about the core workflow, thanks > > > On Thu, 2 Apr 2020 at 19:55, Wes Turner wes.turner@gmail.com > wrote: > >> https://devguide.python.org/buildbots/ >> >> These run in Docker containers: >> - https://github.com/python/cpython/blob/master/.travis.yml >> - >> https://github.com/conda-forge/python-feedstock/blob/master/recipe/build.sh >> >> These are all of the current builds; are you proposing another? >> >> https://github.com/python/buildmaster-config/blob/master/master/custom/build... >> >> https://github.com/python/buildmaster-config/blob/master/worker_example.Dock... >> >> https://github.com/buildbot/buildbot_travis : >> >> > Basically we provide a compatibility shim in buildbot that allows >> it to consume a .travis.yml file. >> > >> > buildbot_travis does however not support the full .travis.yml >> format. >> >> https://github.com/python/core-workflow/issues >> >> >> On Thu, Apr 2, 2020, 6:36 PM Antonio Cavallo < >> antonio.cavallo.71@gmail.com> wrote: >> >>> Hi >>> is there any interest (or anyone has done it before), building the >>> python interpreter using docker? >>> >>> The basic idea is building the toolchain (gcc) and on top of that >>> the python interpreter. On mac/linux it can build natively, but it can use >>> docker to target linux from mac/windows. >>> >>> Thanks >>> _______________________________________________ >>> Python-ideas mailing list -- python-ideas@python.org >>> To unsubscribe send an email to python-ideas-leave@python.org >>> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >>> Message archived at >>> https://mail.python.org/archives/list/python-ideas@python.org/message/DX7WNX... >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>
Hi Antonio,
could you please spell out your requirements a bit more clearly ?
I have a hard time trying to understand what you're after. It seems to be about some sort of Python extension building environment for Linux, but I could be wrong.
You normally don't need to build Python itself just to build an extension.
If you're looking for a standardized Python build for Linux, PyRun might be what you want: http://pyrun.org/
Thanks,
Hi Marc-Andre,
yes pyrun is sort of what I had in mind :) That would be like the jre part of the jdk if you're familiar with java.
Ideally users will download the jre and use for running applications (eg. pyrun) but if there's a need for an extension build one need to have a compiler.. now given the plethora of linux distros that's a logistic nightmare.
So my little project does that (following the linux from scratch guidelines): it builds the toolchain (minus libc) the compiler and the python interpreter in one giant (at the moment) tarball. It should be easy to extract from it the runtime part (eg. the pyrun equivalent or the miniconda).
I hope this makes a bit more sense.
On Fri, 3 Apr 2020 at 04:46, M.-A. Lemburg mal@egenix.com wrote:
Hi Antonio,
could you please spell out your requirements a bit more clearly ?
I have a hard time trying to understand what you're after. It seems to be about some sort of Python extension building environment for Linux, but I could be wrong.
You normally don't need to build Python itself just to build an extension.
If you're looking for a standardized Python build for Linux, PyRun might be what you want: http://pyrun.org/
Thanks,
Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Apr 03 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Conda can supply (in principle ) anything. And it certainly supply compilers. And conda-forge has them, at least for some packages.
It seems that conda could provide your SDK. And if you want it, you could use constructor to build an installer with everything in one package.
-CHB
On Fri, Apr 3, 2020 at 6:22 AM Antonio Cavallo antonio.cavallo.71@gmail.com wrote:
Hi Marc-Andre,
yes pyrun is sort of what I had in mind :) That would be like the jre part of the jdk if you're familiar with java.
Ideally users will download the jre and use for running applications (eg. pyrun) but if there's a need for an extension build one need to have a compiler.. now given the plethora of linux distros that's a logistic nightmare.
So my little project does that (following the linux from scratch guidelines): it builds the toolchain (minus libc) the compiler and the python interpreter in one giant (at the moment) tarball. It should be easy to extract from it the runtime part (eg. the pyrun equivalent or the miniconda).
I hope this makes a bit more sense.
On Fri, 3 Apr 2020 at 04:46, M.-A. Lemburg mal@egenix.com wrote:
Hi Antonio,
could you please spell out your requirements a bit more clearly ?
I have a hard time trying to understand what you're after. It seems to be about some sort of Python extension building environment for Linux, but I could be wrong.
You normally don't need to build Python itself just to build an extension.
If you're looking for a standardized Python build for Linux, PyRun might be what you want: http://pyrun.org/
Thanks,
Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Apr 03 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 https://www.google.com/maps/search/Pastor-Loeh-Str.48+%0D%0A%C2%A0+%C2%A0+D-40764+Langenfeld,+Germany?entry=gmail&source=g D-40764 Langenfeld, Germany https://www.google.com/maps/search/Pastor-Loeh-Str.48+%0D%0A%C2%A0+%C2%A0+D-40764+Langenfeld,+Germany?entry=gmail&source=g. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/42VLPX... Code of Conduct: http://python.org/psf/codeofconduct/
Hi Antonio,
you may want to have a look at the Alpine images for Python and this optimized variant:
https://github.com/jfloff/alpine-python
They also come with dev tools installed. Still, they are overall rather clunky to ship around. On the plus side, you can layer images in docker, so that applications could reuse the base images, once downloaded.
Thanks,
On Tuesday, 7 April 2020 10:52:36 CEST M.-A. Lemburg wrote:
Hi Antonio,
you may want to have a look at the Alpine images for Python and this optimized variant:
https://github.com/jfloff/alpine-python
They also come with dev tools installed. Still, they are overall rather clunky to ship around. On the plus side, you can layer images in docker, so that applications could reuse the base images, once downloaded.
Thanks,
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 4/3/2020 3:18 PM, Antonio Cavallo wrote:
Hi Marc-Andre,
yes pyrun is sort of what I had in mind :) That would be like the jre part of the jdk if you're familiar with java.
Ideally users will download the jre and use for running applications (eg. pyrun) but if there's a need for an extension build one need to have a compiler.. now given the plethora of linux distros that's a logistic nightmare.
So my little project does that (following the linux from scratch guidelines): it builds the toolchain (minus libc) the compiler and the python interpreter in one giant (at the moment) tarball. It should be easy to extract from it the runtime part (eg. the pyrun equivalent or the miniconda).
I hope this makes a bit more sense.
On Fri, 3 Apr 2020 at 04:46, M.-A. Lemburg <mal@egenix.com
mailto:mal@egenix.com> wrote: Hi Antonio,
could you please spell out your requirements a bit more clearly ? I have a hard time trying to understand what you're after. It seems to be about some sort of Python extension building environment for Linux, but I could be wrong. You normally don't need to build Python itself just to build an extension. If you're looking for a standardized Python build for Linux, PyRun might be what you want: http://pyrun.org/ Thanks, Professional Python Services directly from the Experts (#1, Apr 03 2020) >>> Python Projects, Coaching and Support ... >>> https://www.egenix.com/ >>> Python Product Development ... >>> https://consulting.egenix.com/ ______________________________________________________________________ __ ::: We implement business ideas - efficiently in both time and costs
:::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/42V LPXWZKET6L3PIXSTIT2DN4E4M25VF/ Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/74UAJ 5YKJU3VQPFNVP2QUCPNT656Z7EE/ Code of Conduct: http://python.org/psf/codeofconduct/
It's a bad idea to use alpine with Python. See:
- https://github.com/iron-io/dockers/issues/42#issuecomment-290763088 - https://github.com/pypa/manylinux/issues/37 - https://superuser.com/questions/1219609/why-is-the-alpine-docker-image-over-...
Re: manylinux wheels in Alpine docker (musl (with glibc optionally also installed))
"Clarify that pip wheels are incompatible with alpine-based images" https://github.com/docker-library/docs/issues/904
"Install manylinux wheel although it's not officially supported by the host" https://github.com/pypa/pip/issues/3969#issuecomment-495991727 :
RUN echo 'manylinux1_compatible = True' >
/usr/local/lib/python3.7/site-packages/_manylinux.py
RUN python -c 'import sys; sys.path.append(r"/_manylinux.py")'
On Tue, Apr 7, 2020, 10:58 AM João Santos jmcs@jsantos.eu wrote:
On Tuesday, 7 April 2020 10:52:36 CEST M.-A. Lemburg wrote:
Hi Antonio,
you may want to have a look at the Alpine images for Python and this optimized variant:
https://github.com/jfloff/alpine-python
They also come with dev tools installed. Still, they are overall rather clunky to ship around. On the plus side, you can layer images in docker, so that applications could reuse the base images, once downloaded.
Thanks,
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 4/3/2020 3:18 PM, Antonio Cavallo wrote:
Hi Marc-Andre,
yes pyrun is sort of what I had in mind :) That would be like the jre part of the jdk if you're familiar with java.
Ideally users will download the jre and use for running applications (eg. pyrun) but if there's a need for an extension build one need to have a compiler.. now given the plethora of linux distros that's a logistic nightmare.
So my little project does that (following the linux from scratch guidelines): it builds the toolchain (minus libc) the compiler and the python interpreter in one giant (at the moment) tarball. It should be easy to extract from it the runtime part (eg. the pyrun equivalent or the miniconda).
I hope this makes a bit more sense.
On Fri, 3 Apr 2020 at 04:46, M.-A. Lemburg <mal@egenix.com
mailto:mal@egenix.com> wrote: Hi Antonio,
could you please spell out your requirements a bit more clearly ? I have a hard time trying to understand what you're after. It seems to be about some sort of Python extension building environment for Linux, but I could be wrong. You normally don't need to build Python itself just to build an extension. If you're looking for a standardized Python build for Linux, PyRun might be what you want: http://pyrun.org/ Thanks, Professional Python Services directly from the Experts (#1, Apr 03 2020) >>> Python Projects, Coaching and Support ... >>> https://www.egenix.com/ >>> Python Product Development ... >>> https://consulting.egenix.com/
__ ::: We implement business ideas - efficiently in both time andcosts :::
eGenix.com Software, Skills and Services GmbHPastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/42V
LPXWZKET6L3PIXSTIT2DN4E4M25VF/ Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/74UAJ
5YKJU3VQPFNVP2QUCPNT656Z7EE/ Code of Conduct: http://python.org/psf/codeofconduct/
It's a bad idea to use alpine with Python. See:
- https://github.com/iron-io/dockers/issues/42#issuecomment-290763088
- https://github.com/pypa/manylinux/issues/37
https://superuser.com/questions/1219609/why-is-the-alpine-docker-image-over-...
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QWIOLN... Code of Conduct: http://python.org/psf/codeofconduct/
participants (5)
-
Antonio Cavallo -
Christopher Barker -
João Santos -
M.-A. Lemburg -
Wes Turner