[Distutils] continuous integration options (was Re: Travis-CI is not open source, except in fact it *is* open source)

[Wes Turner]

For automated deployment / continuous deployment / "continuous delivery":

- pip maintains a local cache
- devpi can be configured as a transparent proxy cache (in front of pypi.org
)

- GitLab CI can show a checkmark for a deploy pipeline stage

On Saturday, November 5, 2016, Wes Turner <wes.turner at gmail.com> wrote:

>
>
> On Saturday, November 5, 2016, Nick Coghlan <ncoghlan at gmail.com
> <javascript:_e(%7B%7D,'cvml','ncoghlan at gmail.com');>> wrote:
>
>> On 4 November 2016 at 06:07, Nathaniel Smith <njs at pobox.com> wrote:
>> > I think we're drifting pretty far off topic here... IIRC the original
>> > discussion was about whether the travis-ci infrastructure could be
>> suborned
>> > to provide an sdist->wheel autobuilding service for pypi. (Answer:
>> maybe,
>> > though it would be pretty awkward, and no one seems to be jumping up to
>> make
>> > it happen.)
>>
>> The hard part of designing any such system isn't so much the building
>> process, it's the authentication, authorisation and trust management
>> for the release publication step. At the moment, that amounts to "Give
>> the service your PyPI password, so PyPI will 100% trust that they're
>> you" due to limitations on the PyPI side of things, and "Hello web
>> service developer, I grant you 100% authority to impersonate me to the
>> rest of the world however you like" is a questionable idea in any
>> circumstance, let alone when we're talking about publishing software.
>>
>> Since we don't currently provide end-to-end package signing, PyPI
>> initiated builds would solve the trust problem by having PyPI trust
>> *itself*, and only require user credentials for the initial source
>> upload. This has the major downside that "safely" running arbitrary
>> code from unknown publishers is a Hard Problem, which is one of the
>> big reasons that Linux distros put so many hurdles in the way of
>> becoming a package maintainer (i.e. package maintainers get to run
>> arbitrary code not just inside the distro build system but also on end
>> user machines, usually with elevated privileges, so you want to
>> establish a pretty high level of trust before letting people do it).
>> If I understand correctly, conda-forge works on the same basic
>> principle - reviewing the publishers before granting them publication
>> access, rather than defending against arbitrarily malicious code at
>> build time.
>
>
> - https://conda-forge.github.io
> - https://github.com/conda-forge
> - https://github.com/conda-forge/feedstocks
> - https://github.com/conda-forge/conda-smithy
>
>
>>
>> A more promising long term path is trust federation, which many folks
>> will already be familiar with through granting other services access
>> to their GitHub repositories, or using Twitter/Facebook/Google/et al
>> to sign into various systems. That's not going to be a quick fix
>> though, as it's contingent on sorting out the Warehouse migration
>> challenges, and those are already significant enough without piling
>> additional proposed changes on top of the already pending work.
>
>
> - [ ] Warehouse: ENH,SEC: A table, form, API for creating and revoking
> OAuth authz
>   - (project, key, UPLOAD_RELEASE)
>   - key renewal date
>
> There are a few existing OAuth Server libraries for pyramid (Warehouse):
>
> - https://github.com/sneridagh/osiris
> - https://github.com/tilgovi/pyramid-oauthlib
> - https://github.com/elliotpeele/pyramid_oauth2_provider
>
>
>
> - [ ] CI Release utility secrets:
>   - VCS commit signature checking keyring
>   - Package signing key (GPG ASC)
>   - Package signature pubkey
>
> I just found these:
>
> - https://gist.github.com/audreyr/5990987
> - https://github.com/michaeljoseph/changes
>
> - https://pypi.python.org/pypi/jarn.mkrelease
>   - scripted GPG
>
> - https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/
>   - SHOULD have OOB keyring dist
>
> - https://github.com/pypa/twine/issues/157
>   - twine uploads *.asc if it exists
>
> - http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution
> - http://pythonhosted.org/distlib/tutorial.html#verifying-signatures
>   - SHOULD specify a limited keying-dir/
>
> - https://packaging.python.org/distributing/
>   - [ ] howto create an .asc signature
>
> - https://docs.travis-ci.com/user/deployment/pypi/
>   - https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/pypi.rb
>   - [x] https://github.com/travis-ci/dpl/issues/253
>   - [ ] oauth key instead of pass
>
> - https://docs.travis-ci.com/user/deployment/releases/
>   - upload release to GitHub
>
> - [ ] Is it possible to maintain a simple index of GitHub-hosted releases
> and .asc signatures w/ gh-pages? (for backups)
>   - twine: Is uploading GitHub releases in scope?
>     - https://pypi.org/search/?q=Github+release
>
>
>> However, something that could potentially be achieved in the near term
>> given folks interested enough in the idea to set about designing it
>> would be a default recommendation for a Travis CI + Appveyor + GitHub
>> Releases based setup that automated everything *except* the final
>> upload to PyPI, but then also offered a relatively simple way for
>> folks to pull their built artifacts from GitHub and push them to PyPI
>> (such that their login credentials never left their local system).
>> Folks that care enough about who hosts their source code to want to
>> avoid GitHub, or have complex enough build system needs that Travis CI
>> isn't sufficient, are likely to be technically sophisticated enough to
>> adapt service specific instructions to their particular circumstances,
>> and any such design work now would be a useful template for full
>> automation at some point in the future.
>
>
> https://www.google.com/search&q=Travis+Appveyor+GitHub+pypi
>
> ... also useful:
>
> - GitLab CI
>   - .gitlab-ci.yml , config.toml
>   - https://docs.gitlab.com/ce/ci/docker/using_docker_images.html
>
> - Jenkins
>   - https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
>   - https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin
>   - https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin
>   - https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin
>
> - Buildbot
>   - http://docs.buildbot.net/latest/manual/cfg-workers-docker.html
>
> - GitFlow and HubFlow specify a release/ branch with actual release tags
> all on master/
>   - https://datasift.github.io/gitflow/IntroducingGitFlow.html
>
>
>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20161105/04dd33af/attachment.html>