On Saturday, November 5, 2016, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 4 November 2016 at 06:07, Nathaniel Smith <njs@pobox.com> wrote:
> I think we're drifting pretty far off topic here... IIRC the original
> discussion was about whether the travis-ci infrastructure could be suborned
> to provide an sdist->wheel autobuilding service for pypi. (Answer: maybe,
> though it would be pretty awkward, and no one seems to be jumping up to make
> it happen.)
The hard part of designing any such system isn't so much the building
process, it's the authentication, authorisation and trust management
for the release publication step. At the moment, that amounts to "Give
the service your PyPI password, so PyPI will 100% trust that they're
you" due to limitations on the PyPI side of things, and "Hello web
service developer, I grant you 100% authority to impersonate me to the
rest of the world however you like" is a questionable idea in any
circumstance, let alone when we're talking about publishing software.
Since we don't currently provide end-to-end package signing, PyPI
initiated builds would solve the trust problem by having PyPI trust
*itself*, and only require user credentials for the initial source
upload. This has the major downside that "safely" running arbitrary
code from unknown publishers is a Hard Problem, which is one of the
big reasons that Linux distros put so many hurdles in the way of
becoming a package maintainer (i.e. package maintainers get to run
arbitrary code not just inside the distro build system but also on end
user machines, usually with elevated privileges, so you want to
establish a pretty high level of trust before letting people do it).
If I understand correctly, conda-forge works on the same basic
principle - reviewing the publishers before granting them publication
access, rather than defending against arbitrarily malicious code at
build time.
A more promising long term path is trust federation, which many folks
will already be familiar with through granting other services access
to their GitHub repositories, or using Twitter/Facebook/Google/et al
to sign into various systems. That's not going to be a quick fix
though, as it's contingent on sorting out the Warehouse migration
challenges, and those are already significant enough without piling
additional proposed changes on top of the already pending work.
- [ ] Warehouse: ENH,SEC: A table, form, API for creating and revoking OAuth authz
- (project, key, UPLOAD_RELEASE)
- key renewal date
There are a few existing OAuth Server libraries for pyramid (Warehouse):
- [ ] CI Release utility secrets:
- VCS commit signature checking keyring
- Package signing key (GPG ASC)
- Package signature pubkey
I just found these:
- scripted GPG
- SHOULD have OOB keyring dist
- twine uploads *.asc if it exists
- SHOULD specify a limited keying-dir/
- [ ] howto create an .asc signature
- [ ] oauth key instead of pass
- upload release to GitHub
- [ ] Is it possible to maintain a simple index of GitHub-hosted releases and .asc signatures w/ gh-pages? (for backups)
- twine: Is uploading GitHub releases in scope?
However, something that could potentially be achieved in the near term
given folks interested enough in the idea to set about designing it
would be a default recommendation for a Travis CI + Appveyor + GitHub
Releases based setup that automated everything *except* the final
upload to PyPI, but then also offered a relatively simple way for
folks to pull their built artifacts from GitHub and push them to PyPI
(such that their login credentials never left their local system).
Folks that care enough about who hosts their source code to want to
avoid GitHub, or have complex enough build system needs that Travis CI
isn't sufficient, are likely to be technically sophisticated enough to
adapt service specific instructions to their particular circumstances,
and any such design work now would be a useful template for full
automation at some point in the future.
... also useful:
- GitLab CI
- .gitlab-ci.yml , config.toml
- Jenkins
- Buildbot
- GitFlow and HubFlow specify a release/ branch with actual release tags all on master/
Cheers,
Nick.
--
Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig