Re: [Distutils] continuous integration options (was Re: Travis-CI is not open source, except in fact it *is* open source)

On Saturday, November 5, 2016, Nick Coghlan wrote:

On 4 November 2016 at 06:07, Nathaniel Smith javascript:;> wrote:

...

I think we're drifting pretty far off topic here... IIRC the original discussion was about whether the travis-ci infrastructure could be suborned to provide an sdist->wheel autobuilding service for pypi. (Answer: maybe, though it would be pretty awkward, and no one seems to be jumping up to make it happen.)

The hard part of designing any such system isn't so much the building process, it's the authentication, authorisation and trust management for the release publication step. At the moment, that amounts to "Give the service your PyPI password, so PyPI will 100% trust that they're you" due to limitations on the PyPI side of things, and "Hello web service developer, I grant you 100% authority to impersonate me to the rest of the world however you like" is a questionable idea in any circumstance, let alone when we're talking about publishing software.

Since we don't currently provide end-to-end package signing, PyPI initiated builds would solve the trust problem by having PyPI trust *itself*, and only require user credentials for the initial source upload. This has the major downside that "safely" running arbitrary code from unknown publishers is a Hard Problem, which is one of the big reasons that Linux distros put so many hurdles in the way of becoming a package maintainer (i.e. package maintainers get to run arbitrary code not just inside the distro build system but also on end user machines, usually with elevated privileges, so you want to establish a pretty high level of trust before letting people do it). If I understand correctly, conda-forge works on the same basic principle - reviewing the publishers before granting them publication access, rather than defending against arbitrarily malicious code at build time.

- https://conda-forge.github.io - https://github.com/conda-forge - https://github.com/conda-forge/feedstocks - https://github.com/conda-forge/conda-smithy

A more promising long term path is trust federation, which many folks will already be familiar with through granting other services access to their GitHub repositories, or using Twitter/Facebook/Google/et al to sign into various systems. That's not going to be a quick fix though, as it's contingent on sorting out the Warehouse migration challenges, and those are already significant enough without piling additional proposed changes on top of the already pending work.

- [ ] Warehouse: ENH,SEC: A table, form, API for creating and revoking OAuth authz - (project, key, UPLOAD_RELEASE) - key renewal date There are a few existing OAuth Server libraries for pyramid (Warehouse): - https://github.com/sneridagh/osiris - https://github.com/tilgovi/pyramid-oauthlib - https://github.com/elliotpeele/pyramid_oauth2_provider - [ ] CI Release utility secrets: - VCS commit signature checking keyring - Package signing key (GPG ASC) - Package signature pubkey I just found these: - https://gist.github.com/audreyr/5990987 - https://github.com/michaeljoseph/changes - https://pypi.python.org/pypi/jarn.mkrelease - scripted GPG - https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ - SHOULD have OOB keyring dist - https://github.com/pypa/twine/issues/157 - twine uploads *.asc if it exists - http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution - http://pythonhosted.org/distlib/tutorial.html#verifying-signatures - SHOULD specify a limited keying-dir/ - https://packaging.python.org/distributing/ - [ ] howto create an .asc signature - https://docs.travis-ci.com/user/deployment/pypi/ - https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/pypi.rb - [x] https://github.com/travis-ci/dpl/issues/253 - [ ] oauth key instead of pass - https://docs.travis-ci.com/user/deployment/releases/ - upload release to GitHub - [ ] Is it possible to maintain a simple index of GitHub-hosted releases and .asc signatures w/ gh-pages? (for backups) - twine: Is uploading GitHub releases in scope? - https://pypi.org/search/?q=Github+release

However, something that could potentially be achieved in the near term given folks interested enough in the idea to set about designing it would be a default recommendation for a Travis CI + Appveyor + GitHub Releases based setup that automated everything *except* the final upload to PyPI, but then also offered a relatively simple way for folks to pull their built artifacts from GitHub and push them to PyPI (such that their login credentials never left their local system). Folks that care enough about who hosts their source code to want to avoid GitHub, or have complex enough build system needs that Travis CI isn't sufficient, are likely to be technically sophisticated enough to adapt service specific instructions to their particular circumstances, and any such design work now would be a useful template for full automation at some point in the future.

https://www.google.com/search&q=Travis+Appveyor+GitHub+pypi ... also useful: - GitLab CI - .gitlab-ci.yml , config.toml - https://docs.gitlab.com/ce/ci/docker/using_docker_images.html - Jenkins - https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin - https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin - https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin - https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin - Buildbot - http://docs.buildbot.net/latest/manual/cfg-workers-docker.html - GitFlow and HubFlow specify a release/ branch with actual release tags all on master/ - https://datasift.github.io/gitflow/IntroducingGitFlow.html

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com javascript:; | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org javascript:; https://mail.python.org/mailman/listinfo/distutils-sig