For automated deployment / continuous deployment / "continuous delivery":<div><br></div><div>- pip maintains a local cache</div><div>- devpi can be configured as a transparent proxy cache (in front of <a href="http://pypi.org">pypi.org</a>)</div><div><br></div><div>- GitLab CI can show a checkmark for a deploy pipeline stage<br><br>On Saturday, November 5, 2016, Wes Turner <<a href="mailto:wes.turner@gmail.com">wes.turner@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br><br>On Saturday, November 5, 2016, Nick Coghlan <<a href="javascript:_e(%7B%7D,'cvml','ncoghlan@gmail.com');" target="_blank">ncoghlan@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 4 November 2016 at 06:07, Nathaniel Smith <<a>njs@pobox.com</a>> wrote:<br>
> I think we're drifting pretty far off topic here... IIRC the original<br>
> discussion was about whether the travis-ci infrastructure could be suborned<br>
> to provide an sdist->wheel autobuilding service for pypi. (Answer: maybe,<br>
> though it would be pretty awkward, and no one seems to be jumping up to make<br>
> it happen.)<br>
<br>
The hard part of designing any such system isn't so much the building<br>
process, it's the authentication, authorisation and trust management<br>
for the release publication step. At the moment, that amounts to "Give<br>
the service your PyPI password, so PyPI will 100% trust that they're<br>
you" due to limitations on the PyPI side of things, and "Hello web<br>
service developer, I grant you 100% authority to impersonate me to the<br>
rest of the world however you like" is a questionable idea in any<br>
circumstance, let alone when we're talking about publishing software.<br>
<br>
Since we don't currently provide end-to-end package signing, PyPI<br>
initiated builds would solve the trust problem by having PyPI trust<br>
*itself*, and only require user credentials for the initial source<br>
upload. This has the major downside that "safely" running arbitrary<br>
code from unknown publishers is a Hard Problem, which is one of the<br>
big reasons that Linux distros put so many hurdles in the way of<br>
becoming a package maintainer (i.e. package maintainers get to run<br>
arbitrary code not just inside the distro build system but also on end<br>
user machines, usually with elevated privileges, so you want to<br>
establish a pretty high level of trust before letting people do it).<br>
If I understand correctly, conda-forge works on the same basic<br>
principle - reviewing the publishers before granting them publication<br>
access, rather than defending against arbitrarily malicious code at<br>
build time.</blockquote><div><br></div><div>- <a href="https://conda-forge.github.io" target="_blank">https://conda-forge.github.<wbr>io</a></div><div>- <a href="https://github.com/conda-forge" target="_blank">https://github.com/conda-<wbr>forge</a></div><div>- <a href="https://github.com/conda-forge/feedstocks" target="_blank">https://github.com/conda-<wbr>forge/feedstocks</a></div><div>- <a href="https://github.com/conda-forge/conda-smithy" target="_blank">https://github.com/conda-<wbr>forge/conda-smithy</a></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
A more promising long term path is trust federation, which many folks<br>
will already be familiar with through granting other services access<br>
to their GitHub repositories, or using Twitter/Facebook/Google/et al<br>
to sign into various systems. That's not going to be a quick fix<br>
though, as it's contingent on sorting out the Warehouse migration<br>
challenges, and those are already significant enough without piling<br>
additional proposed changes on top of the already pending work.</blockquote><div><br></div><div><div>- [ ] Warehouse: ENH,SEC: A table, form, API for creating and revoking OAuth authz</div><div>  - (project, key, UPLOAD_RELEASE)</div></div><div>  - key renewal date</div><div><br></div><div>There are a few existing OAuth Server libraries for pyramid (Warehouse):</div><div><br></div><div>- <a href="https://github.com/sneridagh/osiris" target="_blank">https://github.com/<wbr>sneridagh/osiris</a></div><div>- <a href="https://github.com/tilgovi/pyramid-oauthlib" target="_blank">https://github.com/tilgovi/<wbr>pyramid-oauthlib</a></div><div>- <a href="https://github.com/elliotpeele/pyramid_oauth2_provider" target="_blank">https://github.com/<wbr>elliotpeele/pyramid_oauth2_<wbr>provider</a></div><div><br></div><div><br></div><div><br></div><div>- [ ] CI Release utility secrets:</div><div>  - VCS commit signature checking keyring</div><div>  - Package signing key (GPG ASC)</div><div>  - Package signature pubkey</div><div><br></div><div>I just found these:</div><div><br></div><div>- <a href="https://gist.github.com/audreyr/5990987" target="_blank">https://gist.github.com/<wbr>audreyr/5990987</a></div><div>- <a href="https://github.com/michaeljoseph/changes" target="_blank">https://github.com/<wbr>michaeljoseph/changes</a></div><div><br></div><div>- <a href="https://pypi.python.org/pypi/jarn.mkrelease" target="_blank">https://pypi.python.org/<wbr>pypi/jarn.mkrelease</a></div><div>  - scripted GPG</div><div><br></div><div>- <a href="https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/" target="_blank">https://caremad.io/posts/<wbr>2013/07/packaging-signing-not-<wbr>holy-grail/</a></div><div>  - SHOULD have OOB keyring dist</div><div><br></div><div>- <a href="https://github.com/pypa/twine/issues/157" target="_blank">https://github.com/pypa/<wbr>twine/issues/157</a></div><div>  - twine uploads *.asc if it exists</div><div><br></div><div>- <a href="http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution" target="_blank">http://pythonhosted.org/<wbr>distlib/tutorial.html#signing-<wbr>a-distribution</a></div><div>- <a href="http://pythonhosted.org/distlib/tutorial.html#verifying-signatures" target="_blank">http://pythonhosted.org/<wbr>distlib/tutorial.html#<wbr>verifying-signatures</a></div><div>  - SHOULD specify a limited keying-dir/</div><div><br></div><div>- <a href="https://packaging.python.org/distributing/" target="_blank">https://packaging.python.<wbr>org/distributing/</a></div><div>  - [ ] howto create an .asc signature</div><div><br></div><div>- <a href="https://docs.travis-ci.com/user/deployment/pypi/" target="_blank">https://docs.travis-ci.com/<wbr>user/deployment/pypi/</a></div><div>  - <a href="https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/pypi.rb" target="_blank">https://github.com/travis-<wbr>ci/dpl/blob/master/lib/dpl/<wbr>provider/pypi.rb</a></div><div>  - [x] <a href="https://github.com/travis-ci/dpl/issues/253" target="_blank">https://github.com/travis-<wbr>ci/dpl/issues/253</a></div><div>  - [ ] oauth key instead of pass</div><div><br></div><div>- <a href="https://docs.travis-ci.com/user/deployment/releases/" target="_blank">https://docs.travis-ci.com/<wbr>user/deployment/releases/</a></div><div>  - upload release to GitHub</div><div><br></div><div>- [ ] Is it possible to maintain a simple index of GitHub-hosted releases and .asc signatures w/ gh-pages? (for backups)</div><div>  - twine: Is uploading GitHub releases in scope?</div><div>    - <a href="https://pypi.org/search/?q=Github+release" target="_blank">https://pypi.org/search/?q=<wbr>Github+release</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
However, something that could potentially be achieved in the near term<br>
given folks interested enough in the idea to set about designing it<br>
would be a default recommendation for a Travis CI + Appveyor + GitHub<br>
Releases based setup that automated everything *except* the final<br>
upload to PyPI, but then also offered a relatively simple way for<br>
folks to pull their built artifacts from GitHub and push them to PyPI<br>
(such that their login credentials never left their local system).<br>
Folks that care enough about who hosts their source code to want to<br>
avoid GitHub, or have complex enough build system needs that Travis CI<br>
isn't sufficient, are likely to be technically sophisticated enough to<br>
adapt service specific instructions to their particular circumstances,<br>
and any such design work now would be a useful template for full<br>
automation at some point in the future. </blockquote><div><br></div><div><a href="https://www.google.com/search&q=Travis+Appveyor+GitHub+pypi" target="_blank">https://www.google.com/search&<wbr>q=Travis+Appveyor+GitHub+pypi</a></div><div><br></div><div>... also useful:</div><div><br></div><div>- GitLab CI</div><div>  - .gitlab-ci.yml , config.toml</div><div>  - <a href="https://docs.gitlab.com/ce/ci/docker/using_docker_images.html" target="_blank">https://docs.gitlab.com/ce/<wbr>ci/docker/using_docker_images.<wbr>html</a></div><div> </div><div>- Jenkins</div><div>  - <a href="https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin" target="_blank">https://wiki.jenkins-ci.org/<wbr>display/JENKINS/Docker+Plugin</a></div><div>  - <a href="https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin" target="_blank">https://wiki.jenkins-ci.org/<wbr>display/JENKINS/ShiningPanda+<wbr>Plugin</a></div><div>  - <a href="https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin" target="_blank">https://wiki.jenkins-ci.org/<wbr>display/JENKINS/GitHub+Plugin</a></div><div>  - <a href="https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin" target="_blank">https://wiki.jenkins-ci.org/<wbr>display/JENKINS/Release+Plugin</a></div><div><br></div><div>- Buildbot</div><div>  - <a href="http://docs.buildbot.net/latest/manual/cfg-workers-docker.html" target="_blank">http://docs.buildbot.net/<wbr>latest/manual/cfg-workers-<wbr>docker.html</a></div><div><br></div><div>- GitFlow and HubFlow specify a release/ branch with actual release tags all on master/</div><div>  - <a href="https://datasift.github.io/gitflow/IntroducingGitFlow.html" target="_blank">https://datasift.github.io/<wbr>gitflow/IntroducingGitFlow.<wbr>html</a></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
Nick.<br>
<br>
--<br>
Nick Coghlan   |   <a>ncoghlan@gmail.com</a>   |   Brisbane, Australia<br>
______________________________<wbr>_________________<br>
Distutils-SIG maillist  -  <a>Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/distutils-sig</a><br>
</blockquote>
</blockquote></div>