Re: Packaging Advice for EFF's Certbot
Wow! Thanks for all the feedback everyone.
On Jul 23, 2018, at 6:52 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
1) Can you list in the document the non-Python dependencies and what they're used for to give people an idea? Done! I added this to the “Certbot Background” section at the top of the document. 2) I don't know how Certbot is architected, but would it be possible to put the "meat" of Certbot inside a Docker container (ideally including most of the non-Python dependencies), and then have a much lighter-weight Python application (with fewer dependencies) running outside of Docker, and that calls the application inside the Docker container when it needs to do stuff? Maybe. Currently each method of performing the domain validation required to get a certificate and the support for installing the certificate in different servers is implemented by a plugin to Certbot. These plugins implement a class with a certain interface which it registers as a setuptools entry_point. At runtime, Certbot picks the class type desired by the user, creates an instance of it, and calls its methods as necessary to obtain a certificate, install it, etc. Most of these plugins make use of code in Certbot’s modules that implement common functionality to simplify plugin development.
This architecture seems like the opposite of what you’re proposing as Certbot is calling the plugin as needed rather than the other way around, but we’re certainly open to the possibility of changing things to make it easier to distribute Certbot. Are you aware of another significantly sized project that is structured in this way?
On Jul 23, 2018, at 7:04 PM, Eli Ribble via Distutils-SIG <distutils-sig@python.org> wrote:
I think I'd separate out certbot installed by the package manager which is just a simple bootstrapper and the certbot installed in the virtualenv which does real work. This is an interesting idea. By making our own .deb and .rpm packages, we can handle things like installing non-Python dependencies and adding cron jobs or systemd timers in the conventional, system specific way rather than trying to use our single certbot-auto script that tries to work everywhere. It also gives us the option to distribute this Certbot bootstrapper separately on less popular OSes with the caveat that those users will have to install dependencies and set up automating certificate renewal themselves. Thanks for the suggestion!
On Jul 23, 2018, at 7:54 PM, Michael Sarahan <msarahan@anaconda.com> wrote:
How will separately distributed plugins work?
Having that layout makes plugins easy, because the paths are both standardized and relocatable (assume relative paths most of the time). Do you know if our approach to using setuptools entry_points to find plugins will work? This is described at https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discover... <https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins>.
I work for Anaconda, and am also a core member of Conda-forge. If you have any questions or just want to talk about how conda/conda-forge may or may not meet your needs, please feel free to reach out. Thanks a lot on this offer. I may take you up on that as I continue to dig into our options and experiment.
On Jul 23, 2018, at 11:10 PM, James Bennett <ubernostrum@gmail.com> wrote:
On Mon, Jul 23, 2018 at 8:17 PM, Alex Walters <tritium-list@sdamon.com <mailto:tritium-list@sdamon.com>> wrote: As a user of certbot, docker, conda, nix, and guix are non-starters. I'm not depending on those tools for my production server (and while docker may be a dependency for some people, that is hardly universal). Adding heavyweight technical dependencies are problematic if your goal is to get everyone using your software. You're better off with cx_freeze or pyinstaller binaries downloaded from a website or a PPA-like-system to add to system package managers, which are not perfect solutions either.
I would emphasize this point. Not wanting to install a lot of extra software to use Certbot is certainly fair and we’d obviously prefer our packaging solution to be as lightweight as possible. Thanks for bringing this up as a consideration.
With that said, our current approaches aren’t working for us. We’re a small development team and continuing to maintain our custom certbot-auto installer which installs dependencies through your OS package manager and creates a virtual environment tucked away in /opt is a significant drain on our resources. If there’s existing tools or projects which can make this easier for us, we’d like to consider them so we can focus our efforts on Certbot and its features rather than packaging and distribution. Our concerns with cx_freeze/pyinstaller and maintaining our own native packaging repos are listed in the Google Doc. If you have information which may help here, please share! Barring someone suggesting something we haven’t considered, maintaining our own native package repos is one of the three options we’re still considering and we haven’t heard many complaints about certbot-auto (when it works). If we go with something that requires significantly more resources though, we’ll definitely take the fact it does this into consideration and would likely offer a lighter weight (but more manual) installation process for sysadmins who prefer it.
The real question here is what use case you're targeting.
Our main use case has been the long tail of individuals or small teams of sysadmins who maintain servers and need or want help and automation around maintaining a secure TLS configuration. Making Certbot available to anyone else who wants to use it is just icing on the cake.
On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding. This is definitely our preferred approach to building native packages right now. To be honest, no one on my team has any experience building .debs and .rpms and we’re happy to learn what we need to if we go with this approach, but the more reliable automation around the process we can use the better. Regardless, given the design constraints and the target audience, it's unlikely to be possible to avoid offering a two-level solution, where the lower platform dependent layer sets up any services and cron jobs required, as well as establishing a Python virtual environment in /opt, and then the bulk of certbot, as well as any installed plugins, run in that venv (plugins would still be managed with pip, just installed into the certbot venv rather than globally). Outside of FPM’s support for packaging a virtual environment which Freddy Rietdijk mentioned, we really hadn’t considered this approach. I think it’s an interesting idea and could work well for us. I gave a couple more thoughts on this approach in my reply to Eli Ribble who suggested the same thing. For actually building and publishing those platform dependent pieces, openSUSE's Open Build Service is the most complete offering I'm aware of: https://openbuildservice.org/ (Fedora's COPR is good for targeting RPM based distros, and Ubuntu offer their PPA system, but OBS allows creation of multiple respository formats). Thanks a lot for this and the other pointers in your post Nick. It’s very helpful!
On Tue, Jul 24, 2018 at 5:48 PM, Brad Warren <bmw@eff.org> wrote:
On Jul 23, 2018, at 6:52 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
2) I don't know how Certbot is architected, but would it be possible to put the "meat" of Certbot inside a Docker container (ideally including most of the non-Python dependencies), and then have a much lighter-weight Python application (with fewer dependencies) running outside of Docker, and that calls the application inside the Docker container when it needs to do stuff?
Maybe. Currently each method of performing the domain validation required to get a certificate and the support for installing the certificate in different servers is implemented by a plugin to Certbot. These plugins implement a class with a certain interface which it registers as a setuptools entry_point. At runtime, Certbot picks the class type desired by the user, creates an instance of it, and calls its methods as necessary to obtain a certificate, install it, etc. Most of these plugins make use of code in Certbot’s modules that implement common functionality to simplify plugin development.
This architecture seems like the opposite of what you’re proposing as Certbot is calling the plugin as needed rather than the other way around, but we’re certainly open to the possibility of changing things to make it easier to distribute Certbot. Are you aware of another significantly sized project that is structured in this way?
Well, I think I'm just suggesting the idea of exposing the "meat" of Certbot as some kind of service, with the service running inside a Docker container. In that sense, any project using multiple Docker containers can be thought of as using an architecture like this (where each container is providing some piece of the functionality). Depending on your needs, the "service" could potentially be as simple as exposing it only via a command-line API. For example, in the case of Certbot, Augeas could be installed inside the Docker container (responsible for parsing and transforming Apache configuration files), and the master client / orchestration piece could be running natively on the machine. When the master client needs to transform an Apache configuration file, it could pass it to the service running inside the Docker container and get back the output. With your plug-in system, it might / probably need to be the case that installing a plug-in would require installing pieces in both the client and service sides (e.g. to get the advantage of keeping complex dependencies inside Docker), with the division that I've described. --Chris
On Tue, Jul 24, 2018 at 5:48 PM, Brad Warren <bmw@eff.org> wrote:
Do you know if our approach to using setuptools entry_points to find plugins will work [with conda]? This is described at https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discover....
Yes, installing Python inside a conda environment gives you a regular Python environment, and things like entry_points work fine.
On Jul 23, 2018, at 11:10 PM, James Bennett <ubernostrum@gmail.com> wrote:
On Mon, Jul 23, 2018 at 8:17 PM, Alex Walters <tritium-list@sdamon.com> wrote:
As a user of certbot, docker, conda, nix, and guix are non-starters. I'm not depending on those tools for my production server (and while docker may be a dependency for some people, that is hardly universal). Adding heavyweight technical dependencies are problematic if your goal is to get everyone using your software. You're better off with cx_freeze or pyinstaller binaries downloaded from a website or a PPA-like-system to add to system package managers, which are not perfect solutions either.
I would emphasize this point.
Not wanting to install a lot of extra software to use Certbot is certainly fair and we’d obviously prefer our packaging solution to be as lightweight as possible. Thanks for bringing this up as a consideration.
I feel like these responses may be lumping things together more than is helpful... using docker is quite heavyweight in the sense that it makes major demands on the host system: you need a docker daemon running, it has to be root, maybe it needs some special kernel modules to handle overlay filesystems, and so forth. That's fine if it's there already, but if it's not then certbot shouldn't be trying to bootstrap a docker installation from scratch. Conda OTOH is nothing like that -- it's just tools and conventions for working with regular files on a regular filesystem. One of its core design goals was that non-technical grad students should be able to drop it into a random directory on some HPC cluster where they don't have root and that's running some weird old distro like Scientific Linux 6, run one command, and have everything just work. Which sounds pretty similar to your use case. I don't know where nix and guix fall on this spectrum.
On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding.
But it sounds like they also want to be able to install/remove/upgrade *parts* of the Python project, for their plugin support. And maybe upgrade the Python interpreter as well. Do any of these tools allow that? That's the thing that really made me think about conda. -n -- Nathaniel J. Smith -- https://vorpus.org
On Tue, Jul 24, 2018 at 7:39 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Jul 24, 2018 at 5:48 PM, Brad Warren <bmw@eff.org> wrote:
Not wanting to install a lot of extra software to use Certbot is certainly fair and we’d obviously prefer our packaging solution to be as lightweight as possible. Thanks for bringing this up as a consideration.
I feel like these responses may be lumping things together more than is helpful... using docker is quite heavyweight in the sense that it makes major demands on the host system: you need a docker daemon running, it has to be root, maybe it needs some special kernel modules to handle overlay filesystems, and so forth. That's fine if it's there already, but if it's not then certbot shouldn't be trying to bootstrap a docker installation from scratch.
Just to be clear, I wasn't meaning to promote or recommend the Docker option I described. I was just throwing it out there as an additional option to consider because I didn't see it listed in the Google doc in the form I had in mind. It could very well be too heavyweight -- I understand that downside. I don't have a good enough sense of what the deployments look like to evaluate the trade-offs one way or the other. --Chris
On 25 July 2018 at 12:39, Nathaniel Smith <njs@pobox.com> wrote:
On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding.
But it sounds like they also want to be able to install/remove/upgrade *parts* of the Python project, for their plugin support. And maybe upgrade the Python interpreter as well. Do any of these tools allow that? That's the thing that really made me think about conda.
Right, that's why my suggestion was for a two layer solution (native packaging of a base platform integration layer via fpm, combined with pip for plugin management within that base environment), akin to the way Linux distro packages of Firefox and Chromium still leave the browser to do its own plugin management. That way the fpm-built native package can depend on any required system packages, as well as lay out the base virtual environment in /opt. In many ways, it's the same thing that certbot-auto is already doing, it's just replacing the current directly downloaded shell script with native Linux packages built with fpm. You can certainly do the same thing with conda instead (as per [1]), but given that the target audience for certbot includes professional Linux sysadmins, being able to integrate with the native system package manager seems to be an actively desired feature rather than an unwanted hassle. So while I'd agree conda is well worth a look as a potential helper for environment management within the /opt directory, in this particular case I don't think it's going to be a suitable replacement for offering native packages as the core update mechanism for the base platform. Cheers, Nick. [1] http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.ht... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I agree that you are probably best off integrating with the system packaging system in this case. But if you do want to deploy and app with all its dependencies in a controlled environment, conda constructor May make that easy: https://github.com/conda/constructor -CHB Sent from my iPhone On Jul 26, 2018, at 4:20 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: On 25 July 2018 at 12:39, Nathaniel Smith <njs@pobox.com> wrote: On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding. But it sounds like they also want to be able to install/remove/upgrade *parts* of the Python project, for their plugin support. And maybe upgrade the Python interpreter as well. Do any of these tools allow that? That's the thing that really made me think about conda. Right, that's why my suggestion was for a two layer solution (native packaging of a base platform integration layer via fpm, combined with pip for plugin management within that base environment), akin to the way Linux distro packages of Firefox and Chromium still leave the browser to do its own plugin management. That way the fpm-built native package can depend on any required system packages, as well as lay out the base virtual environment in /opt. In many ways, it's the same thing that certbot-auto is already doing, it's just replacing the current directly downloaded shell script with native Linux packages built with fpm. You can certainly do the same thing with conda instead (as per [1]), but given that the target audience for certbot includes professional Linux sysadmins, being able to integrate with the native system package manager seems to be an actively desired feature rather than an unwanted hassle. So while I'd agree conda is well worth a look as a potential helper for environment management within the /opt directory, in this particular case I don't think it's going to be a suitable replacement for offering native packages as the core update mechanism for the base platform. Cheers, Nick. [1] http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.ht... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/K...
As a heavy user of certbot myself across numerous systems I would echo Nick and others’ suggestions about native packaging. Third party tools and docker containers are basically non starters. Dan Ryan // pipenv maintainer gh: @techalchemy
On Jul 26, 2018, at 11:30 AM, Chris Barker - NOAA Federal via Distutils-SIG <distutils-sig@python.org> wrote:
I agree that you are probably best off integrating with the system packaging system in this case.
But if you do want to deploy and app with all its dependencies in a controlled environment, conda constructor May make that easy:
https://github.com/conda/constructor
-CHB
Sent from my iPhone
On Jul 26, 2018, at 4:20 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 25 July 2018 at 12:39, Nathaniel Smith <njs@pobox.com> wrote:
On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding.
But it sounds like they also want to be able to install/remove/upgrade *parts* of the Python project, for their plugin support. And maybe upgrade the Python interpreter as well. Do any of these tools allow that? That's the thing that really made me think about conda.
Right, that's why my suggestion was for a two layer solution (native packaging of a base platform integration layer via fpm, combined with pip for plugin management within that base environment), akin to the way Linux distro packages of Firefox and Chromium still leave the browser to do its own plugin management.
That way the fpm-built native package can depend on any required system packages, as well as lay out the base virtual environment in /opt. In many ways, it's the same thing that certbot-auto is already doing, it's just replacing the current directly downloaded shell script with native Linux packages built with fpm.
You can certainly do the same thing with conda instead (as per [1]), but given that the target audience for certbot includes professional Linux sysadmins, being able to integrate with the native system package manager seems to be an actively desired feature rather than an unwanted hassle. So while I'd agree conda is well worth a look as a potential helper for environment management within the /opt directory, in this particular case I don't think it's going to be a suitable replacement for offering native packages as the core update mechanism for the base platform.
Cheers, Nick.
[1] http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.ht...
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/K... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/V...
Brad Warren <bmw@eff.org> writes:
Wow! Thanks for all the feedback everyone.
Thank you for seeking it, and for the work on making Certbot viable.
On Jul 23, 2018, at 11:10 PM, James Bennett <ubernostrum@gmail.com> wrote:
On Mon, Jul 23, 2018 at 8:17 PM, Alex Walters <tritium-list@sdamon.com> wrote: As a user of certbot, docker, conda, nix, and guix are non-starters. […] You're better off with cx_freeze or pyinstaller binaries downloaded from a website or a PPA-like-system to add to system package managers, which are not perfect solutions either.
I would emphasize this point.
Not wanting to install a lot of extra software to use Certbot is certainly fair and we’d obviously prefer our packaging solution to be as lightweight as possible. Thanks for bringing this up as a consideration.
With that said, our current approaches aren’t working for us. We’re a small development team and continuing to maintain our custom certbot-auto installer which installs dependencies through your OS package manager and creates a virtual environment tucked away in /opt is a significant drain on our resources.
Another approach is to get a good handle on the third-party dependency libraries needed for Certbot, and encourage those dependencies to be part of the OS package repositories. That way, you don't take on the huge maintenance burden of trying to keep the third-party library code *and* Certbot in good shape. Just focus on Certbot, and cheer from the sidelines as the OS distributions do the work of third party packages. Yes, that's a different set of problems (for example, keeping Certbot compatible with those versions of the libraries the OS repositories provide). But it is likely to be more manageable, for a team that wants to focus on Certbot's code base primarily.
If there’s existing tools or projects which can make this easier for us, we’d like to consider them so we can focus our efforts on Certbot and its features rather than packaging and distribution.
The OS distributions – the free-software ones, anyway – have communities that make it their business to do that job well. Engage with them and ask what it'll take to have Library Foo kept up to date in the OS, so you can have Certbot just declare a dependency on that. In other words, I advise: Don't look for tools to do this automatically, engage with the people who already know how to do this in each OS community.
Our main use case has been the long tail of individuals or small teams of sysadmins who maintain servers and need or want help and automation around maintaining a secure TLS configuration.
For what it's worth, I certainly concur that most people in the group you describe will thank you to not have a bundle of custom dependencies from outside the OS repository, and instead make Certbot work with (and/or be advocates to introduce) the dependencies as OS-provided library packages.
This is definitely our preferred approach to building native packages right now. To be honest, no one on my team has any experience building .debs and .rpms and we’re happy to learn what we need to if we go with this approach, but the more reliable automation around the process we can use the better.
For Debian, instead of becoming packaging experts yourselves, instead we ask that you make Certbot easier for *us* to package and maintain in Debian. See <URL:https://wiki.debian.org/UpstreamGuide>; other OS projects likely have similar documentation. -- \ “There's a fine line between fishing and standing on the shore | `\ looking like an idiot.” —Steven Wright | _o__) | Ben Finney
On Thu, Jul 26, 2018, 05:48 Ben Finney via Distutils-SIG < distutils-sig@python.org> wrote:
Brad Warren <bmw@eff.org> writes:
Our main use case has been the long tail of individuals or small teams of sysadmins who maintain servers and need or want help and automation around maintaining a secure TLS configuration.
For what it's worth, I certainly concur that most people in the group you describe will thank you to not have a bundle of custom dependencies from outside the OS repository, and instead make Certbot work with (and/or be advocates to introduce) the dependencies as OS-provided library packages.
I wonder if they have any user survey or anything that would speak to this. FWIW my impression is that the kinds of sysadmins who know enough to have opinions about packaging hygiene also know enough to set up one of certbot's many simpler, less magical competitors. Certbot's target audience has always been folks who didn't really understand any of this stuff and wanted to just hit a button and have things somehow work out by whatever means necessary. Which is great, those people deserve secure communications. But the point is that you can't make everything your number one priority, and when they have to choose, certbot has chosen to prioritize "make it work" over "fit nicely into a traditional distro", and I think that's the right decision for what they are.
This is definitely our preferred approach to building native packages
right now. To be honest, no one on my team has any experience building .debs and .rpms and we’re happy to learn what we need to if we go with this approach, but the more reliable automation around the process we can use the better.
For Debian, instead of becoming packaging experts yourselves, instead we ask that you make Certbot easier for *us* to package and maintain in Debian. See <URL:https://wiki.debian.org/UpstreamGuide>; other OS projects likely have similar documentation.
That's a nice idea in principle and all, but see also https://wiki.debian.org/LetsEncrypt which cheerily notes that the version of certbot shipped in stretch is so old that it doesn't actually work, and as a workaround it recommends running a command that will take down the user's website without even warning that that's what it does. I love Debian. I've been a loyal user for twenty years. But Debian is really bad at coping with software that needs to react quickly to changing external conditions, or that depends on a tight feedback loop between users and developers. (Indeed, Debian's whole value-add is to insert themselves as a buffer between users and developers, which is great in some situations but terrible in others.) Maybe if certbot worked hard enough they could arrange to get special treatment like Firefox or clamav or something, but in their position I would see this as a total non-starter. Even if they did somehow manage to navigate Debian's politics, they'd still have to go and repeat the process for Redhat, SuSE, Ubuntu, Gentoo, ... -n
If we didn’t want to trust any binaries built by someone else or proprietary code, how much work would that be?
- Docker Notary (The Update Framework) - PEP 458, PEP 480 (TUF) - Host GPG .asc(s) for things you just found ## To build the whole toolchain yourself? Build, Package, Install, Upgrade/Replace_with_new_container - Makefile - Conda-forge recipes with CI (meta.yaml, build.sh) - [x] platform / architecture compilation optimizations - [x] support for Windows (build.bat) - [x] support for Mac & Linux (build.sh) - conda constructor - build an installer - tar up the whole virtualenvwrapper virtualenv and unpack that into the exact same path in a container - and check the .ASC ## To sign trusted releases: - See: Warehouse & GPG, Wheel & signatures - *The Update Framework* - There are plans to merge support for TUF (which is designed to withstand package repo compromise) into Warehouse at some point, AFAIU. TUF: The Update Framework ======================== | Wikipedia: https://en.wikipedia.org/wiki/The_Update_Framework_(TUF) | Homepage: https://theupdateframework.github.io/ | Src: https://github.com/theupdateframework | Spec: https://github.com/theupdateframework/specification/blob/master/tuf-spec.md - Docker Notary supports TUF - TUF is mostly written in Python - Python PEP458: - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf - "PEP 458 -- Surviving a Compromise of PyPI" https://www.python.org/dev/peps/pep-0458/" - "PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model" https://www.python.org/dev/peps/pep-0480/ - https://github.com/pypa/pip/issues/1035 "Implement "hook" support for package signature verification." ## To build native packages for your package and the whole toolchain? - fpm - release keys (note that any key in the ring can sign for any package) - docker/kubernetes - conda-forge (Appveyor, CircleCI, Travis CI) - OSX instances fpm ===== | Src: https://github.com/jordansissel/fpm | Docs: https://fpm.readthedocs.io/en/latest/ | Docs: https://fpm.readthedocs.io/en/latest/source/virtualenv.html - { deb, rpm, osxpkg, tar, } ## To host package repositories yourselves: Pulp ===== | Homepage: https://pulpproject.org/ | Src: https://github.com/pulp/ ## To sign releases uploaded to e.g. GitHub (which are CDN'd, AFAIU): - Wheel had a native signing mechanism that's since been removed - [ ] Attach a GPG .asc signature to the GitHub release - https://wiki.debian.org/Creating%20signed%20GitHub%20releases ## To sign commits and merges: - Configure a per-project GPG agent and keyring config - Trust local datetimes - Remember that `git push -f` can occur. ## Include SELinux filesystem labels with the package: - This isn't possible with Python packaging (without extending setup.py with a custom command with a build dependency or calling out to a shell script that may be locatable with distutils.spawn.find_executable) On Thu, Jul 26, 2018 at 1:41 PM Nathaniel Smith <njs@pobox.com> wrote:
On Thu, Jul 26, 2018, 05:48 Ben Finney via Distutils-SIG < distutils-sig@python.org> wrote:
Brad Warren <bmw@eff.org> writes:
Our main use case has been the long tail of individuals or small teams of sysadmins who maintain servers and need or want help and automation around maintaining a secure TLS configuration.
For what it's worth, I certainly concur that most people in the group you describe will thank you to not have a bundle of custom dependencies from outside the OS repository, and instead make Certbot work with (and/or be advocates to introduce) the dependencies as OS-provided library packages.
I wonder if they have any user survey or anything that would speak to this.
FWIW my impression is that the kinds of sysadmins who know enough to have opinions about packaging hygiene also know enough to set up one of certbot's many simpler, less magical competitors. Certbot's target audience has always been folks who didn't really understand any of this stuff and wanted to just hit a button and have things somehow work out by whatever means necessary. Which is great, those people deserve secure communications. But the point is that you can't make everything your number one priority, and when they have to choose, certbot has chosen to prioritize "make it work" over "fit nicely into a traditional distro", and I think that's the right decision for what they are.
This is definitely our preferred approach to building native packages
right now. To be honest, no one on my team has any experience building .debs and .rpms and we’re happy to learn what we need to if we go with this approach, but the more reliable automation around the process we can use the better.
For Debian, instead of becoming packaging experts yourselves, instead we ask that you make Certbot easier for *us* to package and maintain in Debian. See <URL:https://wiki.debian.org/UpstreamGuide>; other OS projects likely have similar documentation.
That's a nice idea in principle and all, but see also https://wiki.debian.org/LetsEncrypt which cheerily notes that the version of certbot shipped in stretch is so old that it doesn't actually work, and as a workaround it recommends running a command that will take down the user's website without even warning that that's what it does.
I love Debian. I've been a loyal user for twenty years. But Debian is really bad at coping with software that needs to react quickly to changing external conditions, or that depends on a tight feedback loop between users and developers. (Indeed, Debian's whole value-add is to insert themselves as a buffer between users and developers, which is great in some situations but terrible in others.)
Maybe if certbot worked hard enough they could arrange to get special treatment like Firefox or clamav or something, but in their position I would see this as a total non-starter. Even if they did somehow manage to navigate Debian's politics, they'd still have to go and repeat the process for Redhat, SuSE, Ubuntu, Gentoo, ...
-n
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/E...
Took a minute more to read the gdoc link. The CI build should produce every {OS,} package as a build artifact (along with coverage and static analysis reports). #goeswithoutsaying Someone can probably point to a good GitLab CI config that produces all of the requisite artifacts and uploads them to the appropriate repos (with an optional manual approval step). ## a new package updater tool - IIRC, cloudmatrix/esky has transactional updates but is no longer updated. - what are other good ways way to do auto-updating transactional updates? - It makes sense to *start with TUF* (which is derived from the Tor updater) - Polling for updates costs time and money. ## systemd/cron config Official repos with e.g. Puppet, Salt, Ansible configs would be advantageous because: - certbot should be run on an appropriately secured baseline (with filesystem labels) - otherwise there are like 10 not even forks of unofficial configs and no consensus These also need to be part of the CI build. On Thursday, July 26, 2018, Wes Turner <wes.turner@gmail.com> wrote:
If we didn’t want to trust any binaries built by someone else or proprietary code, how much work would that be?
- Docker Notary (The Update Framework) - PEP 458, PEP 480 (TUF) - Host GPG .asc(s) for things you just found
## To build the whole toolchain yourself? Build, Package, Install, Upgrade/Replace_with_new_container
- Makefile - Conda-forge recipes with CI (meta.yaml, build.sh) - [x] platform / architecture compilation optimizations - [x] support for Windows (build.bat) - [x] support for Mac & Linux (build.sh)
- conda constructor - build an installer - tar up the whole virtualenvwrapper virtualenv and unpack that into the exact same path in a container - and check the .ASC
## To sign trusted releases: - See: Warehouse & GPG, Wheel & signatures - *The Update Framework* - There are plans to merge support for TUF (which is designed to withstand package repo compromise) into Warehouse at some point, AFAIU.
TUF: The Update Framework ======================== | Wikipedia: https://en.wikipedia.org/wiki/The_Update_Framework_(TUF) | Homepage: https://theupdateframework.github.io/ | Src: https://github.com/theupdateframework | Spec: https://github.com/theupdateframework/ specification/blob/master/tuf-spec.md
- Docker Notary supports TUF - TUF is mostly written in Python - Python PEP458: - https://www.pypa.io/en/latest/roadmap/#pypi-integrate-tuf
- "PEP 458 -- Surviving a Compromise of PyPI" https://www.python.org/dev/ peps/pep-0458/"
- "PEP 480 -- Surviving a Compromise of PyPI: The Maximum Security Model" https://www.python.org/dev/peps/pep-0480/
- https://github.com/pypa/pip/issues/1035 "Implement "hook" support for package signature verification."
## To build native packages for your package and the whole toolchain? - fpm - release keys (note that any key in the ring can sign for any package) - docker/kubernetes - conda-forge (Appveyor, CircleCI, Travis CI) - OSX instances
fpm ===== | Src: https://github.com/jordansissel/fpm | Docs: https://fpm.readthedocs.io/en/latest/ | Docs: https://fpm.readthedocs.io/en/latest/source/virtualenv.html
- { deb, rpm, osxpkg, tar, }
## To host package repositories yourselves:
Pulp ===== | Homepage: https://pulpproject.org/ | Src: https://github.com/pulp/
## To sign releases uploaded to e.g. GitHub (which are CDN'd, AFAIU): - Wheel had a native signing mechanism that's since been removed - [ ] Attach a GPG .asc signature to the GitHub release - https://wiki.debian.org/Creating%20signed%20GitHub%20releases
## To sign commits and merges: - Configure a per-project GPG agent and keyring config - Trust local datetimes - Remember that `git push -f` can occur.
## Include SELinux filesystem labels with the package: - This isn't possible with Python packaging (without extending setup.py with a custom command with a build dependency or calling out to a shell script that may be locatable with distutils.spawn.find_executable)
On Thu, Jul 26, 2018 at 1:41 PM Nathaniel Smith <njs@pobox.com> wrote:
On Thu, Jul 26, 2018, 05:48 Ben Finney via Distutils-SIG < distutils-sig@python.org> wrote:
Brad Warren <bmw@eff.org> writes:
Our main use case has been the long tail of individuals or small teams of sysadmins who maintain servers and need or want help and automation around maintaining a secure TLS configuration.
For what it's worth, I certainly concur that most people in the group you describe will thank you to not have a bundle of custom dependencies from outside the OS repository, and instead make Certbot work with (and/or be advocates to introduce) the dependencies as OS-provided library packages.
I wonder if they have any user survey or anything that would speak to this.
FWIW my impression is that the kinds of sysadmins who know enough to have opinions about packaging hygiene also know enough to set up one of certbot's many simpler, less magical competitors. Certbot's target audience has always been folks who didn't really understand any of this stuff and wanted to just hit a button and have things somehow work out by whatever means necessary. Which is great, those people deserve secure communications. But the point is that you can't make everything your number one priority, and when they have to choose, certbot has chosen to prioritize "make it work" over "fit nicely into a traditional distro", and I think that's the right decision for what they are.
This is definitely our preferred approach to building native packages
right now. To be honest, no one on my team has any experience building .debs and .rpms and we’re happy to learn what we need to if we go with this approach, but the more reliable automation around the process we can use the better.
For Debian, instead of becoming packaging experts yourselves, instead we ask that you make Certbot easier for *us* to package and maintain in Debian. See <URL:https://wiki.debian.org/UpstreamGuide>; other OS projects likely have similar documentation.
That's a nice idea in principle and all, but see also https://wiki.debian.org/LetsEncrypt which cheerily notes that the version of certbot shipped in stretch is so old that it doesn't actually work, and as a workaround it recommends running a command that will take down the user's website without even warning that that's what it does.
I love Debian. I've been a loyal user for twenty years. But Debian is really bad at coping with software that needs to react quickly to changing external conditions, or that depends on a tight feedback loop between users and developers. (Indeed, Debian's whole value-add is to insert themselves as a buffer between users and developers, which is great in some situations but terrible in others.)
Maybe if certbot worked hard enough they could arrange to get special treatment like Firefox or clamav or something, but in their position I would see this as a total non-starter. Even if they did somehow manage to navigate Debian's politics, they'd still have to go and repeat the process for Redhat, SuSE, Ubuntu, Gentoo, ...
-n
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ archives/list/distutils-sig@python.org/message/ EDCELE3LBN7LMUXF6FX4EYERX4I33ELL/
Nathaniel Smith wrote:
I love Debian. I've been a loyal user for twenty years. But Debian is really bad at coping with software that needs to react quickly to changing external conditions, or that depends on a tight feedback loop between users and developers. (Indeed, Debian's whole value-add is to insert themselves as a buffer between users and developers, which is great in some situations but terrible in others.)
Maybe if certbot worked hard enough they could arrange to get special treatment like Firefox or clamav or something, but in their position I would see this as a total non-starter. Even if they did somehow manage to navigate Debian's politics, they'd still have to go and repeat the process for Redhat, SuSE, Ubuntu, Gentoo, ...
I wouldn't be too put off by the idea of Debian politics. Certbot should be a good fit for stable-updates: https://wiki.debian.org/StableUpdates under the "Packages that need to be current to be useful" criterion listed at https://www.debian.org/News/2011/20110215 -M-
It's generally easiest to find a maintainer who has an established track record to adopt the project. On Sat, Jul 28, 2018 at 11:53 AM, <matthew@woodcraft.me.uk> wrote:
I love Debian. I've been a loyal user for twenty years. But Debian is really bad at coping with software that needs to react quickly to changing external conditions, or that depends on a tight feedback loop between users and developers. (Indeed, Debian's whole value-add is to insert themselves as a buffer between users and developers, which is great in some situations but terrible in others.)
Maybe if certbot worked hard enough they could arrange to get special treatment like Firefox or clamav or something, but in their position I would see this as a total non-starter. Even if they did somehow manage to navigate Debian's politics, they'd still have to go and repeat the
Nathaniel Smith wrote: process
for Redhat, SuSE, Ubuntu, Gentoo, ...
I wouldn't be too put off by the idea of Debian politics. Certbot should be a good fit for stable-updates: https://wiki.debian.org/StableUpdates under the "Packages that need to be current to be useful" criterion listed at https://www.debian.org/News/2011/20110215
-M- -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ archives/list/distutils-sig@python.org/message/ EGMGR2HFKJOPANXPORLSLNDRTYVB3FOM/
participants (10)
-
Ben Finney -
Brad Warren -
Chris Barker - NOAA Federal -
Chris Jerdonek -
Dan Ryan -
Matt Joyce -
matthew＠woodcraft.me.uk -
Nathaniel Smith -
Nick Coghlan -
Wes Turner