Thanks, Glyph! I haven't tried this yet, but I think it is exactly what I need. -- John Santos On 8/28/2023 7:13 PM, Glyph wrote:
On Aug 28, 2023, at 2:41 PM, John Santos <john@egh.com> wrote:
There is no "setup.py" How do I install from sources?
Pip is the only supported installer. If we want to support other installation mechanisms, we need to add them to CI first.
I can't use pip for several reasons:
Luckily it's totally possible to use pip and satisfy all these requirements!
1) Customer systems are not Internet-connected and can't use pypi to retrieve packages at install time.
Pip has an option, `--no-index`, which is explicitly for this sort of non-internet-connected system.
For what it's worth, `setup.py` can and will /also/ use setuptools to reach out to the internet to download stuff at install time; mostly, build dependencies. It's running arbitrary code and sometimes that code will attempt to shell out to easy_install.
2) Version control. I need to be sure the versions of all packages pulled in due to dependencies exactly match what I'm using on my test/development systems.
A great deal of pip's infrastructure is dedicated to allowing for transitive pinning of dependencies. You may want to look into pip-compile, specifically with the --generate-hashes option: https://pypi.org/project/pip-tools/ <https://pypi.org/project/pip-tools/>
3) I need to have actual source code, not an opaque blob, for documentation and accountability.
Twisted and all of its dependencies ship source distributions as well as pre-built wheels to PyPI because many users have this requirement, so you don't need to use the wheels.
The basic process for your type of build pipeline is two commands, although you can get a lot fancier if you want.
The first command, you run on your development systems to build a meta-package of all the source distributions that you use as input to your development process; this needs Internet access and given your requirements this is the step at which you'd unpack all the downloaded archives and have a look inside them to review changes to validate updates for security, or whatever other purposes you need the source code for:
1. pip download --no-binary :all: twisted hatchling flit_core wheel calver setuptools_scm hatch_vcs hatch-fancy-pypi-readme
This command is annoyingly long because I am not sure how to tell `pip download` to include all build dependencies to create a hermetic environment, but this is the current transitive list of all build-deps for Twisted's dependency tree, as far as I can tell. I think once everybody has adopted PEP 517 this will be simpler.
If you need twisted extras like [tls], [http2], etc, or you want to use this process for a different project, the process of discovering these extra build-time dependencies is basically just do this step, do the next step, look for any package names in the installation error if you get one, then repeat.
Note that "--no-binary :all:" will tell Pip to download source distributions only, for *all* packages. No prebuilt wheels, no binary components.
2. pip install --find-links . --no-index twisted
This will install Twisted entirely from the source distributions you just downloaded with the previous command, no communication with PyPI at all, no loading of blobs.
This is, of course, /much/ slower than using all the prebuilt stuff and caching infrastructure one would normally invoke, it takes 1-2 minutes to get through this full process as opposed to only a few seconds for a `pip install`, but it satisfies your requirements.
I would suggest adding a few extra steps though, since install-time for customers is quite different from verification-time for developers. Rather than doing a flat install directly off the source distributions on customer systems, I'd build your /own/ wheels from the source of dependencies you've validated in-house, which would be something like:
# on your build system pip wheel --find-links . --no-index twisted cp *.whl /.../installer-media
# on customer system, later pip install --no-index --find-links /.../installer-media/ twisted
Hope this helps!
-g
_______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-leave@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/CX6H355S5DD... Code of Conduct: https://twisted.org/conduct
-- John Santos Evans Griffiths & Hart, Inc. 781-861-0670 ext 539