<br><br>On Saturday, November 5, 2016, Wes Turner <<a href="mailto:wes.turner@gmail.com">wes.turner@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">For automated deployment / continuous deployment / "continuous delivery":<div><br></div><div>- pip maintains a local cache</div><div>- devpi can be configured as a transparent proxy cache (in front of <a href="http://pypi.org" target="_blank">pypi.org</a>)</div></blockquote><div><br></div><div><a href="http://doc.devpi.net/latest/quickstart-pypimirror.html">http://doc.devpi.net/latest/quickstart-pypimirror.html</a><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br></div><div>- GitLab CI can show a checkmark for a deploy pipeline stage<br><br>On Saturday, November 5, 2016, Wes Turner <<a href="javascript:_e(%7B%7D,'cvml','wes.turner@gmail.com');" target="_blank">wes.turner@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br><br>On Saturday, November 5, 2016, Nick Coghlan <<a>ncoghlan@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 4 November 2016 at 06:07, Nathaniel Smith <<a>njs@pobox.com</a>> wrote:<br>
> I think we're drifting pretty far off topic here... IIRC the original<br>
> discussion was about whether the travis-ci infrastructure could be suborned<br>
> to provide an sdist->wheel autobuilding service for pypi. (Answer: maybe,<br>
> though it would be pretty awkward, and no one seems to be jumping up to make<br>
> it happen.)<br>
<br>
The hard part of designing any such system isn't so much the building<br>
process, it's the authentication, authorisation and trust management<br>
for the release publication step. At the moment, that amounts to "Give<br>
the service your PyPI password, so PyPI will 100% trust that they're<br>
you" due to limitations on the PyPI side of things, and "Hello web<br>
service developer, I grant you 100% authority to impersonate me to the<br>
rest of the world however you like" is a questionable idea in any<br>
circumstance, let alone when we're talking about publishing software.<br>
<br>
Since we don't currently provide end-to-end package signing, PyPI<br>
initiated builds would solve the trust problem by having PyPI trust<br>
*itself*, and only require user credentials for the initial source<br>
upload. This has the major downside that "safely" running arbitrary<br>
code from unknown publishers is a Hard Problem, which is one of the<br>
big reasons that Linux distros put so many hurdles in the way of<br>
becoming a package maintainer (i.e. package maintainers get to run<br>
arbitrary code not just inside the distro build system but also on end<br>
user machines, usually with elevated privileges, so you want to<br>
establish a pretty high level of trust before letting people do it).<br>
If I understand correctly, conda-forge works on the same basic<br>
principle - reviewing the publishers before granting them publication<br>
access, rather than defending against arbitrarily malicious code at<br>
build time.</blockquote><div><br></div><div>- <a href="https://conda-forge.github.io" target="_blank">https://conda-forge.github.i<wbr>o</a></div><div>- <a href="https://github.com/conda-forge" target="_blank">https://github.com/conda-for<wbr>ge</a></div><div>- <a href="https://github.com/conda-forge/feedstocks" target="_blank">https://github.com/conda-for<wbr>ge/feedstocks</a></div><div>- <a href="https://github.com/conda-forge/conda-smithy" target="_blank">https://github.com/conda-for<wbr>ge/conda-smithy</a></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
A more promising long term path is trust federation, which many folks<br>
will already be familiar with through granting other services access<br>
to their GitHub repositories, or using Twitter/Facebook/Google/et al<br>
to sign into various systems. That's not going to be a quick fix<br>
though, as it's contingent on sorting out the Warehouse migration<br>
challenges, and those are already significant enough without piling<br>
additional proposed changes on top of the already pending work.</blockquote><div><br></div><div><div>- [ ] Warehouse: ENH,SEC: A table, form, API for creating and revoking OAuth authz</div><div>  - (project, key, UPLOAD_RELEASE)</div></div><div>  - key renewal date</div><div><br></div><div>There are a few existing OAuth Server libraries for pyramid (Warehouse):</div><div><br></div><div>- <a href="https://github.com/sneridagh/osiris" target="_blank">https://github.com/sneridagh<wbr>/osiris</a></div><div>- <a href="https://github.com/tilgovi/pyramid-oauthlib" target="_blank">https://github.com/tilgovi/p<wbr>yramid-oauthlib</a></div><div>- <a href="https://github.com/elliotpeele/pyramid_oauth2_provider" target="_blank">https://github.com/elliotpee<wbr>le/pyramid_oauth2_provider</a></div><div><br></div><div><br></div><div><br></div><div>- [ ] CI Release utility secrets:</div><div>  - VCS commit signature checking keyring</div><div>  - Package signing key (GPG ASC)</div><div>  - Package signature pubkey</div><div><br></div><div>I just found these:</div><div><br></div><div>- <a href="https://gist.github.com/audreyr/5990987" target="_blank">https://gist.github.com/audr<wbr>eyr/5990987</a></div><div>- <a href="https://github.com/michaeljoseph/changes" target="_blank">https://github.com/michaeljo<wbr>seph/changes</a></div><div><br></div><div>- <a href="https://pypi.python.org/pypi/jarn.mkrelease" target="_blank">https://pypi.python.org/pypi<wbr>/jarn.mkrelease</a></div><div>  - scripted GPG</div><div><br></div><div>- <a href="https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/" target="_blank">https://caremad.io/posts/201<wbr>3/07/packaging-signing-not-hol<wbr>y-grail/</a></div><div>  - SHOULD have OOB keyring dist</div><div><br></div><div>- <a href="https://github.com/pypa/twine/issues/157" target="_blank">https://github.com/pypa/twin<wbr>e/issues/157</a></div><div>  - twine uploads *.asc if it exists</div><div><br></div><div>- <a href="http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution" target="_blank">http://pythonhosted.org/dist<wbr>lib/tutorial.html#signing-a-<wbr>distribution</a></div><div>- <a href="http://pythonhosted.org/distlib/tutorial.html#verifying-signatures" target="_blank">http://pythonhosted.org/dist<wbr>lib/tutorial.html#verifying-<wbr>signatures</a></div><div>  - SHOULD specify a limited keying-dir/</div><div><br></div><div>- <a href="https://packaging.python.org/distributing/" target="_blank">https://packaging.python.org<wbr>/distributing/</a></div><div>  - [ ] howto create an .asc signature</div><div><br></div><div>- <a href="https://docs.travis-ci.com/user/deployment/pypi/" target="_blank">https://docs.travis-ci.com/u<wbr>ser/deployment/pypi/</a></div><div>  - <a href="https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/pypi.rb" target="_blank">https://github.com/travis-ci<wbr>/dpl/blob/master/lib/dpl/provi<wbr>der/pypi.rb</a></div><div>  - [x] <a href="https://github.com/travis-ci/dpl/issues/253" target="_blank">https://github.com/travis-<wbr>ci/dpl/issues/253</a></div><div>  - [ ] oauth key instead of pass</div><div><br></div><div>- <a href="https://docs.travis-ci.com/user/deployment/releases/" target="_blank">https://docs.travis-ci.com/u<wbr>ser/deployment/releases/</a></div><div>  - upload release to GitHub</div><div><br></div><div>- [ ] Is it possible to maintain a simple index of GitHub-hosted releases and .asc signatures w/ gh-pages? (for backups)</div><div>  - twine: Is uploading GitHub releases in scope?</div><div>    - <a href="https://pypi.org/search/?q=Github+release" target="_blank">https://pypi.org/search/?q=G<wbr>ithub+release</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
However, something that could potentially be achieved in the near term<br>
given folks interested enough in the idea to set about designing it<br>
would be a default recommendation for a Travis CI + Appveyor + GitHub<br>
Releases based setup that automated everything *except* the final<br>
upload to PyPI, but then also offered a relatively simple way for<br>
folks to pull their built artifacts from GitHub and push them to PyPI<br>
(such that their login credentials never left their local system).<br>
Folks that care enough about who hosts their source code to want to<br>
avoid GitHub, or have complex enough build system needs that Travis CI<br>
isn't sufficient, are likely to be technically sophisticated enough to<br>
adapt service specific instructions to their particular circumstances,<br>
and any such design work now would be a useful template for full<br>
automation at some point in the future. </blockquote><div><br></div><div><a href="https://www.google.com/search&q=Travis+Appveyor+GitHub+pypi" target="_blank">https://www.google.com/search&<wbr>q=Travis+Appveyor+GitHub+pypi</a></div><div><br></div><div>... also useful:</div><div><br></div><div>- GitLab CI</div><div>  - .gitlab-ci.yml , config.toml</div><div>  - <a href="https://docs.gitlab.com/ce/ci/docker/using_docker_images.html" target="_blank">https://docs.gitlab.com/ce/c<wbr>i/docker/using_docker_images.h<wbr>tml</a></div><div> </div><div>- Jenkins</div><div>  - <a href="https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin" target="_blank">https://wiki.jenkins-ci.org/<wbr>display/JENKINS/Docker+Plugin</a></div><div>  - <a href="https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin" target="_blank">https://wiki.jenkins-ci.org/<wbr>display/JENKINS/ShiningPanda+P<wbr>lugin</a></div><div>  - <a href="https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin" target="_blank">https://wiki.jenkins-ci.org/<wbr>display/JENKINS/GitHub+Plugin</a></div><div>  - <a href="https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin" target="_blank">https://wiki.jenkins-ci.org/<wbr>display/JENKINS/Release+Plugin</a></div><div><br></div><div>- Buildbot</div><div>  - <a href="http://docs.buildbot.net/latest/manual/cfg-workers-docker.html" target="_blank">http://docs.buildbot.net/lat<wbr>est/manual/cfg-workers-docker.<wbr>html</a></div><div><br></div><div>- GitFlow and HubFlow specify a release/ branch with actual release tags all on master/</div><div>  - <a href="https://datasift.github.io/gitflow/IntroducingGitFlow.html" target="_blank">https://datasift.github.io/g<wbr>itflow/IntroducingGitFlow.html</a></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
Nick.<br>
<br>
--<br>
Nick Coghlan   |   <a>ncoghlan@gmail.com</a>   |   Brisbane, Australia<br>
______________________________<wbr>_________________<br>
Distutils-SIG maillist  -  <a>Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" target="_blank">https://mail.python.org/mailma<wbr>n/listinfo/distutils-sig</a><br>
</blockquote>
</blockquote></div>
</blockquote>