[Distutils] continuous integration options (was Re: Travis-CI is not open source, except in fact it *is* open source)

[Wes Turner]

On Saturday, November 5, 2016, Wes Turner <wes.turner at gmail.com> wrote:

> For automated deployment / continuous deployment / "continuous delivery":
>
> - pip maintains a local cache
> - devpi can be configured as a transparent proxy cache (in front of
> pypi.org)
>

http://doc.devpi.net/latest/quickstart-pypimirror.html


>
> - GitLab CI can show a checkmark for a deploy pipeline stage
>
> On Saturday, November 5, 2016, Wes Turner <wes.turner at gmail.com
> <javascript:_e(%7B%7D,'cvml','wes.turner at gmail.com');>> wrote:
>
>>
>>
>> On Saturday, November 5, 2016, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>> On 4 November 2016 at 06:07, Nathaniel Smith <njs at pobox.com> wrote:
>>> > I think we're drifting pretty far off topic here... IIRC the original
>>> > discussion was about whether the travis-ci infrastructure could be
>>> suborned
>>> > to provide an sdist->wheel autobuilding service for pypi. (Answer:
>>> maybe,
>>> > though it would be pretty awkward, and no one seems to be jumping up
>>> to make
>>> > it happen.)
>>>
>>> The hard part of designing any such system isn't so much the building
>>> process, it's the authentication, authorisation and trust management
>>> for the release publication step. At the moment, that amounts to "Give
>>> the service your PyPI password, so PyPI will 100% trust that they're
>>> you" due to limitations on the PyPI side of things, and "Hello web
>>> service developer, I grant you 100% authority to impersonate me to the
>>> rest of the world however you like" is a questionable idea in any
>>> circumstance, let alone when we're talking about publishing software.
>>>
>>> Since we don't currently provide end-to-end package signing, PyPI
>>> initiated builds would solve the trust problem by having PyPI trust
>>> *itself*, and only require user credentials for the initial source
>>> upload. This has the major downside that "safely" running arbitrary
>>> code from unknown publishers is a Hard Problem, which is one of the
>>> big reasons that Linux distros put so many hurdles in the way of
>>> becoming a package maintainer (i.e. package maintainers get to run
>>> arbitrary code not just inside the distro build system but also on end
>>> user machines, usually with elevated privileges, so you want to
>>> establish a pretty high level of trust before letting people do it).
>>> If I understand correctly, conda-forge works on the same basic
>>> principle - reviewing the publishers before granting them publication
>>> access, rather than defending against arbitrarily malicious code at
>>> build time.
>>
>>
>> - https://conda-forge.github.io
>> - https://github.com/conda-forge
>> - https://github.com/conda-forge/feedstocks
>> - https://github.com/conda-forge/conda-smithy
>>
>>
>>>
>>> A more promising long term path is trust federation, which many folks
>>> will already be familiar with through granting other services access
>>> to their GitHub repositories, or using Twitter/Facebook/Google/et al
>>> to sign into various systems. That's not going to be a quick fix
>>> though, as it's contingent on sorting out the Warehouse migration
>>> challenges, and those are already significant enough without piling
>>> additional proposed changes on top of the already pending work.
>>
>>
>> - [ ] Warehouse: ENH,SEC: A table, form, API for creating and revoking
>> OAuth authz
>>   - (project, key, UPLOAD_RELEASE)
>>   - key renewal date
>>
>> There are a few existing OAuth Server libraries for pyramid (Warehouse):
>>
>> - https://github.com/sneridagh/osiris
>> - https://github.com/tilgovi/pyramid-oauthlib
>> - https://github.com/elliotpeele/pyramid_oauth2_provider
>>
>>
>>
>> - [ ] CI Release utility secrets:
>>   - VCS commit signature checking keyring
>>   - Package signing key (GPG ASC)
>>   - Package signature pubkey
>>
>> I just found these:
>>
>> - https://gist.github.com/audreyr/5990987
>> - https://github.com/michaeljoseph/changes
>>
>> - https://pypi.python.org/pypi/jarn.mkrelease
>>   - scripted GPG
>>
>> - https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/
>>   - SHOULD have OOB keyring dist
>>
>> - https://github.com/pypa/twine/issues/157
>>   - twine uploads *.asc if it exists
>>
>> - http://pythonhosted.org/distlib/tutorial.html#signing-a-distribution
>> - http://pythonhosted.org/distlib/tutorial.html#verifying-signatures
>>   - SHOULD specify a limited keying-dir/
>>
>> - https://packaging.python.org/distributing/
>>   - [ ] howto create an .asc signature
>>
>> - https://docs.travis-ci.com/user/deployment/pypi/
>>   - https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/pypi.rb
>>   - [x] https://github.com/travis-ci/dpl/issues/253
>>   - [ ] oauth key instead of pass
>>
>> - https://docs.travis-ci.com/user/deployment/releases/
>>   - upload release to GitHub
>>
>> - [ ] Is it possible to maintain a simple index of GitHub-hosted releases
>> and .asc signatures w/ gh-pages? (for backups)
>>   - twine: Is uploading GitHub releases in scope?
>>     - https://pypi.org/search/?q=Github+release
>>
>>
>>> However, something that could potentially be achieved in the near term
>>> given folks interested enough in the idea to set about designing it
>>> would be a default recommendation for a Travis CI + Appveyor + GitHub
>>> Releases based setup that automated everything *except* the final
>>> upload to PyPI, but then also offered a relatively simple way for
>>> folks to pull their built artifacts from GitHub and push them to PyPI
>>> (such that their login credentials never left their local system).
>>> Folks that care enough about who hosts their source code to want to
>>> avoid GitHub, or have complex enough build system needs that Travis CI
>>> isn't sufficient, are likely to be technically sophisticated enough to
>>> adapt service specific instructions to their particular circumstances,
>>> and any such design work now would be a useful template for full
>>> automation at some point in the future.
>>
>>
>> https://www.google.com/search&q=Travis+Appveyor+GitHub+pypi
>>
>> ... also useful:
>>
>> - GitLab CI
>>   - .gitlab-ci.yml , config.toml
>>   - https://docs.gitlab.com/ce/ci/docker/using_docker_images.html
>>
>> - Jenkins
>>   - https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
>>   - https://wiki.jenkins-ci.org/display/JENKINS/ShiningPanda+Plugin
>>   - https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin
>>   - https://wiki.jenkins-ci.org/display/JENKINS/Release+Plugin
>>
>> - Buildbot
>>   - http://docs.buildbot.net/latest/manual/cfg-workers-docker.html
>>
>> - GitFlow and HubFlow specify a release/ branch with actual release tags
>> all on master/
>>   - https://datasift.github.io/gitflow/IntroducingGitFlow.html
>>
>>
>>
>>> Cheers,
>>> Nick.
>>>
>>> --
>>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>>> _______________________________________________
>>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20161105/752672f0/attachment-0001.html>