[Distutils] continuous integration options (was Re: Travis-CI is not open source, except in fact it *is* open source)

[Wes Turner]

On Saturday, November 5, 2016, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 4 November 2016 at 06:07, Nathaniel Smith <njs at pobox.com <javascript:;>>
> wrote:
> > I think we're drifting pretty far off topic here... IIRC the original
> > discussion was about whether the travis-ci infrastructure could be
> suborned
> > to provide an sdist->wheel autobuilding service for pypi. (Answer: maybe,
> > though it would be pretty awkward, and no one seems to be jumping up to
> make
> > it happen.)
>
> The hard part of designing any such system isn't so much the building
> process, it's the authentication, authorisation and trust management
> for the release publication step. At the moment, that amounts to "Give
> the service your PyPI password, so PyPI will 100% trust that they're
> you" due to limitations on the PyPI side of things, and "Hello web
> service developer, I grant you 100% authority to impersonate me to the
> rest of the world however you like" is a questionable idea in any
> circumstance, let alone when we're talking about publishing software.
>
> Since we don't currently provide end-to-end package signing, PyPI
> initiated builds would solve the trust problem by having PyPI trust
> *itself*, and only require user credentials for the initial source
> upload. This has the major downside that "safely" running arbitrary
> code from unknown publishers is a Hard Problem, which is one of the
> big reasons that Linux distros put so many hurdles in the way of
> becoming a package maintainer (i.e. package maintainers get to run
> arbitrary code not just inside the distro build system but also on end
> user machines, usually with elevated privileges, so you want to
> establish a pretty high level of trust before letting people do it).
> If I understand correctly, conda-forge works on the same basic
> principle - reviewing the publishers before granting them publication
> access, rather than defending against arbitrarily malicious code at
> build time.


- https://conda-forge.github.io
- https://github.com/conda-forge
- https://github.com/conda-forge/feedstocks
- https://github.com/conda-forge/conda-smithy


>
> A more promising long term path is trust federation, which many folks
> will already be familiar with through granting other services access
> to their GitHub repositories, or using Twitter/Facebook/Google/et al
> to sign into various systems. That's not going to be a quick fix
> though, as it's contingent on sorting out the Warehouse migration
> challenges, and those are already significant enough without piling
> additional proposed changes on top of the already pending work.


- [ ] Warehouse: ENH,SEC: A table, form, API for creating and revoking
OAuth authz
  - (project, key, UPLOAD_RELEASE)
  - key renewal date

There are a few existing OAuth Server libraries for pyramid (Warehouse):

- https://github.com/sneridagh/osiris
- https://github.com/tilgovi/pyramid-oauthlib
- https://github.com/elliotpeele/pyramid_oauth2_provider



- [ ] CI Release utility secrets:
  - VCS commit signature checking keyring
  - Package signing key (GPG ASC)
  - Package signature pubkey

I just found these:

- https://gist.github.com/audreyr/5990987
- https://github.com/michaeljoseph/changes

- https://pypi.python.org/pypi/jarn.mkrelease
  - scripted GPG

- https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/
  - SHOULD have OOB keyring dist

- https://github.com/pypa/twine/issues/157
  - twine uploads *.asc if it exists

- http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution
- http://pythonhosted.org/distlib/tutorial.html#verifying-signatures
  - SHOULD specify a limited keying-dir/

- https://packaging.python.org/distributing/
  - [ ] howto create an .asc signature

- https://docs.travis-ci.com/user/deployment/pypi/
  - https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/pypi.rb
  - [x] https://github.com/travis-ci/dpl/issues/253
  - [ ] oauth key instead of pass

- https://docs.travis-ci.com/user/deployment/releases/
  - upload release to GitHub

- [ ] Is it possible to maintain a simple index of GitHub-hosted releases
and .asc signatures w/ gh-pages? (for backups)
  - twine: Is uploading GitHub releases in scope?
    - https://pypi.org/search/?q=Github+release


> However, something that could potentially be achieved in the near term
> given folks interested enough in the idea to set about designing it
> would be a default recommendation for a Travis CI + Appveyor + GitHub
> Releases based setup that automated everything *except* the final
> upload to PyPI, but then also offered a relatively simple way for
> folks to pull their built artifacts from GitHub and push them to PyPI
> (such that their login credentials never left their local system).
> Folks that care enough about who hosts their source code to want to
> avoid GitHub, or have complex enough build system needs that Travis CI
> isn't sufficient, are likely to be technically sophisticated enough to
> adapt service specific instructions to their particular circumstances,
> and any such design work now would be a useful template for full
> automation at some point in the future.


https://www.google.com/search&q=Travis+Appveyor+GitHub+pypi

... also useful:

- GitLab CI
  - .gitlab-ci.yml , config.toml
  - https://docs.gitlab.com/ce/ci/docker/using_docker_images.html

- Jenkins
  - https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
  - https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin
  - https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin
  - https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin

- Buildbot
  - http://docs.buildbot.net/latest/manual/cfg-workers-docker.html

- GitFlow and HubFlow specify a release/ branch with actual release tags
all on master/
  - https://datasift.github.io/gitflow/IntroducingGitFlow.html



> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com <javascript:;>   |   Brisbane,
> Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20161105/b66fa219/attachment-0001.html>