Distributing packages to offline machines
I had a question about distributing python packages to offline machines when the offline machine is running a different OS then a machine with an internet connection. The packages I am concerned with are third party upon which mine depend.
Based on what I have learned so far, there are three solutions.
(a) Use a CI to run a fleet of machines for each OS one needs to target to obtain the OS specific wheels.
(b) 'pip download <package_list> --no-binary :all:' -- the intention here is to grab the source distribution without any OS specific code included.
(c) use https://warehouse.pypa.io/api-reference/json to look for distributed wheels for the target OS and python version and download them directly. (This may make for a nice flag to add to pip somewhere...the ability to specify what wheel one wants when it isn’t for the machine pip is running on)
The issue I see with (a) is the shear amount work it would take to setup and maintain such a system.
The issue I see with (b) is that it is not 100% reliable as some packages are tricky to install and may not work well with 'pip download’.
I have not played around with (c) yet, so I do not know how well it will work, but it seems like a viable solution.
I was just wondering if anyone had any comments on this as I think though the ways to solve this problem.
Thank you.
Have you already tried `pip download --platform`?
https://pip.pypa.io/en/stable/reference/pip_download/#cmdoption-platform
It may be worth setting up devpi (maybe in a container) and caching the packages; particularly for CI:
https://packaging.python.org/guides/index-mirrors-and-caches/
AFAIU, there's no way to specify a limited set of packages and platforms to mirror with bandersnatch?
On Wednesday, April 4, 2018, Eric Gorr ericgorr@gmail.com wrote:
I had a question about distributing python packages to offline machines when the offline machine is running a different OS then a machine with an internet connection. The packages I am concerned with are third party upon which mine depend.
Based on what I have learned so far, there are three solutions.
(a) Use a CI to run a fleet of machines for each OS one needs to target to obtain the OS specific wheels.
(b) 'pip download <package_list> --no-binary :all:' -- the intention here is to grab the source distribution without any OS specific code included.
(c) use https://warehouse.pypa.io/api-reference/json to look for distributed wheels for the target OS and python version and download them directly. (This may make for a nice flag to add to pip somewhere...the ability to specify what wheel one wants when it isn’t for the machine pip is running on)
The issue I see with (a) is the shear amount work it would take to setup and maintain such a system.
The issue I see with (b) is that it is not 100% reliable as some packages are tricky to install and may not work well with 'pip download’.
I have not played around with (c) yet, so I do not know how well it will work, but it seems like a viable solution.
I was just wondering if anyone had any comments on this as I think though the ways to solve this problem.
Thank you.
On Thu, Apr 5, 2018, at 2:11 AM, Eric Gorr wrote:
(c) use https://warehouse.pypa.io/api-reference/json to look for distributed wheels for the target OS and python version and download them directly. (This may make for a nice flag to add to pip somewhere...the ability to specify what wheel one wants when it isn’t for the machine pip is running on)
If it's useful, Pynsist includes some code to find and download wheels for Windows, regardless of the platform it runs on: https://github.com/takluyver/pynsist/blob/master/nsist/pypi.py
Pynsist is a tool to make Windows installers. The tool can also run on Linux and Mac, but it always builds Windows installers. I didn't know about "pip download --platform" when I wrote this - it may be redundant.
On 5 April 2018 at 11:11, Eric Gorr ericgorr@gmail.com wrote:
I had a question about distributing python packages to offline machines when the offline machine is running a different OS then a machine with an internet connection. The packages I am concerned with are third party upon which mine depend.
Based on what I have learned so far, there are three solutions.
(a) Use a CI to run a fleet of machines for each OS one needs to target to obtain the OS specific wheels.
The issue I see with (a) is the shear amount work it would take to setup and maintain such a system.
Depending on just how many packages and how many operating system versions are involved, this option may be less work that you anticipate, thanks to the excellent wagon project from Cloudify: https://github.com/cloudify-cosmo/wagon
Keep a requirements.txt file or `Pipfile` in source control, then run CI jobs based on that repo to emit wagon files with all the required wheels embedded in them (either sourced from PyPI, or built locally in CI as needed).
Cheers, Nick.
P.S. See https://github.com/cloudify-cosmo/wagon/issues/30#issuecomment-374096806 for the technique of supplying a dummy setup.py file that gives wagon the metadata it wants, without actually needing to define a full top level "application" package.
P.P.S Depending on how strict the requirement is for "offline wheel files" vs "offline prebuilt binary dependencies", it may also be worth your while to explore https://www.paypal-engineering.com/2016/09/07/python-packaging-at-paypal/ (regarding the use of conda as a Python platform manager) and https://conda.io/docs/user-guide/tasks/create-custom-channels.html (regarding the creation of tar-friendly custom conda channels)
Nick Coghlan ncoghlan@gmail.com writes:
Keep a requirements.txt file or `Pipfile` in source control, then run CI jobs based on that repo […]
What is a “Pipfile”?
On Friday, April 6, 2018, Ben Finney ben+python@benfinney.id.au wrote:
Nick Coghlan ncoghlan@gmail.com writes:
Keep a requirements.txt file or `Pipfile` in source control, then run CI jobs based on that repo […]
What is a “Pipfile”?
https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock
-- \ “Software patents provide one more means of controlling access | `\ to information. They are the tool of choice for the internet | _o__) highwayman.” —Anthony Taylor | Ben Finney
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 7 April 2018 at 12:21, Wes Turner wes.turner@gmail.com wrote:
On Friday, April 6, 2018, Ben Finney ben+python@benfinney.id.au wrote:
Nick Coghlan ncoghlan@gmail.com writes:
Keep a requirements.txt file or `Pipfile` in source control, then run CI jobs based on that repo […]
What is a “Pipfile”?
https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock
And some additional info is available in PyPUG's application dependency management tutorial: https://packaging.python.org/tutorials/managing-dependencies/
Cheers, Nick.
participants (5)
-
Ben Finney -
Eric Gorr -
Nick Coghlan -
Thomas Kluyver -
Wes Turner