Re: Packaging Advice for EFF's Certbot
On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding.
But it sounds like they also want to be able to install/remove/upgrade *parts* of the Python project, for their plugin support. That’s correct. We currently have 18 official plugins for Certbot with plans to add more and a few dozen third-party plugins. Do any of these tools allow that? This is a good question. If we went with something like dh-virtualenv or packaged virtualenvs with fpm, would we be able to have separate packages for our plugins that installed things in the same virtualenv? I haven’t looked into this yet, but I wouldn’t expect this to work. That's the thing that really made me think about conda. My biggest concern with conda right now is I believe we (or our users) would be on their own for setting up a cron job or systemd timer for renewal. Is this correct?
On Jul 24, 2018, at 11:20 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Just to be clear, I wasn't meaning to promote or recommend the Docker option I described. Sure! After reading your 2nd email describing how this would work in more detail, I think this would require a pretty major rewrite to how Certbot currently works. Given the other downsides, I’m not sure this is the best approach for us, but I appreciate you throwing out the idea regardless just in case it was!
On Jul 26, 2018, at 3:20 AM, Ben Finney via Distutils-SIG <distutils-sig@python.org> wrote: Just focus on Certbot, and cheer from the sidelines as the OS distributions do the work of third party packages.
Yes, that's a different set of problems (for example, keeping Certbot compatible with those versions of the libraries the OS repositories provide). We’ve done the work to maintain compatibility with the older versions of our dependencies available in the official OS repos where we are packaged. The source of our problems with official OS packaging is described below and in the Google Doc linked in my initial email.
On Jul 28, 2018, at 8:53 AM, matthew@woodcraft.me.uk wrote:
I wouldn't be too put off by the idea of Debian politics. Certbot should be a good fit for stable-updates:
We thought so too and getting updates like this was our main packaging plan when we launched in 2015. Unfortunately, it hasn’t gone well and is the main reason we’re seeking our own solution. Perhaps we did something incorrectly, but as Nathaniel pointed out, Certbot is broken in Debian Stretch and has been since January. The same and many other problems affect the packages in Ubuntu Xenial. We’ve also struggled to find people to help maintain our PPA for older, non-EOL’d versions of Ubuntu. If anyone reading this would like to help us solve these problems or know someone who would, please reach out off-list. While these packages exist in OS repos, some users will continue to use them regardless of the alternative packaging approach we take. Unless the current issues are resolved and we’re confident new ones in the future will be fixed quickly as well, I think we need to offer alternative packaging so we can provide our users some means of getting a working version of Certbot.
On Jul 28, 2018, at 12:00 PM, Wes Turner <wes.turner@gmail.com> wrote:
Took a minute more to read the gdoc link.
Wes Turner, while I don’t have any specific questions or comments right now, thank you very much for all the ideas and links.
Yw! * Pytest, tox, and devpi handle plugins with pluggy; which works with OS and Python package installs AFAICT: https://pluggy.readthedocs.io/en/latest/ * IDK what's needed to remove a package from a repo because it's dangerously out of date. * Arrow recently gained lots of build/test/packaging scripts (that run on TravisCI and AppVeyor): - https://github.com/apache/arrow/blob/master/.travis.yml - TIL about 'after_success:' - https://github.com/apache/arrow/tree/master/ci * A few certbot links that you're probably already aware of: - https://github.com/certbot/certbot/blob/master/.travis.yml - https://github.com/certbot/certbot/blob/master/tox.ini * And the TravisCI deployment docs: - https://docs.travis-ci.com/user/deployment/ - https://docs.travis-ci.com/user/deployment/releases/ (GitHub releases) - https://docs.travis-ci.com/user/deployment/pypi/ - https://docs.travis-ci.com/user/deployment/launchpad/ - [ ] PPA - [ ] EPEL - [ ] Copr (Fedora RPM builds for yum/dnf) - [ ] Conda-forge On Wednesday, August 1, 2018, Brad Warren <bmw@eff.org> wrote:
On Jul 24, 2018, at 4:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding.
But it sounds like they also want to be able to install/remove/upgrade *parts* of the Python project, for their plugin support. That’s correct. We currently have 18 official plugins for Certbot with plans to add more and a few dozen third-party plugins. Do any of these tools allow that? This is a good question. If we went with something like dh-virtualenv or packaged virtualenvs with fpm, would we be able to have separate packages for our plugins that installed things in the same virtualenv? I haven’t looked into this yet, but I wouldn’t expect this to work. That's the thing that really made me think about conda. My biggest concern with conda right now is I believe we (or our users) would be on their own for setting up a cron job or systemd timer for renewal. Is this correct?
On Jul 24, 2018, at 11:20 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Just to be clear, I wasn't meaning to promote or recommend the Docker option I described. Sure! After reading your 2nd email describing how this would work in more detail, I think this would require a pretty major rewrite to how Certbot currently works. Given the other downsides, I’m not sure this is the best approach for us, but I appreciate you throwing out the idea regardless just in case it was!
On Jul 26, 2018, at 3:20 AM, Ben Finney via Distutils-SIG < distutils-sig@python.org> wrote: Just focus on Certbot, and cheer from the sidelines as the OS distributions do the work of third party packages.
Yes, that's a different set of problems (for example, keeping Certbot compatible with those versions of the libraries the OS repositories provide). We’ve done the work to maintain compatibility with the older versions of our dependencies available in the official OS repos where we are packaged. The source of our problems with official OS packaging is described below and in the Google Doc linked in my initial email.
On Jul 28, 2018, at 8:53 AM, matthew@woodcraft.me.uk wrote:
I wouldn't be too put off by the idea of Debian politics. Certbot should be a good fit for stable-updates:
We thought so too and getting updates like this was our main packaging plan when we launched in 2015. Unfortunately, it hasn’t gone well and is the main reason we’re seeking our own solution. Perhaps we did something incorrectly, but as Nathaniel pointed out, Certbot is broken in Debian Stretch and has been since January. The same and many other problems affect the packages in Ubuntu Xenial. We’ve also struggled to find people to help maintain our PPA for older, non-EOL’d versions of Ubuntu.
If anyone reading this would like to help us solve these problems or know someone who would, please reach out off-list. While these packages exist in OS repos, some users will continue to use them regardless of the alternative packaging approach we take. Unless the current issues are resolved and we’re confident new ones in the future will be fixed quickly as well, I think we need to offer alternative packaging so we can provide our users some means of getting a working version of Certbot.
On Jul 28, 2018, at 12:00 PM, Wes Turner <wes.turner@gmail.com> wrote:
Took a minute more to read the gdoc link.
Wes Turner, while I don’t have any specific questions or comments right now, thank you very much for all the ideas and links.
[Nick wrote:]
However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding.
[Nathaniel wrote:]
But it sounds like they also want to be able to install/remove/upgrade *parts* of the Python project, for their plugin support.
[Brad wrote:]
That’s correct. We currently have 18 official plugins for Certbot with plans to add more and a few dozen third-party plugins.
Do any of these tools allow that? This is a good question. If we went with something like dh-virtualenv or packaged virtualenvs with fpm, would we be able to have separate packages for our plugins that installed things in the same virtualenv? I haven’t looked into this yet, but I wouldn’t expect this to work.
As long as the base package puts the virtualenv in a pre-defined location, then other packages can add to it (at least in the world of RPM - I'm assuming similar things would be possible with Debian control files). The trick is that the base package then becomes a build dependency in addition to being a runtime dependency, as you'll need it to set up the correct filesystem layout in the build root. I don't think there are any predefined helpers in fpm for that kind of model though, so you'd likely need some additional automation around setting up auto-generation of certbot-plugin packages (whereas certbot itself could likely use the existing virtualenv helpers). However, the other main option is to handle this in a way similar to what Mozilla does for Firefox plugins: use the system package manager to keep the base application up to date, but use application provided tooling to keep the plugins up to date. If you did that, then the plugins wouldn't get native system packages at all - they'd all remain solely as Python packages, and then the system level certbot package would install a privileged cron job to handle venv updates. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Aug 4, 2018, at 5:36 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I don't think there are any predefined helpers in fpm for that kind of model though, so you'd likely need some additional automation around setting up auto-generation of certbot-plugin packages (whereas certbot itself could likely use the existing virtualenv helpers).
However, the other main option is to handle this in a way similar to what Mozilla does for Firefox plugins: use the system package manager to keep the base application up to date, but use application provided tooling to keep the plugins up to date. There are other benefits to having our application handle plugin installation too (such as helping our discover new plugins).
My main motivation for considering multiple packages from dh-virtualenv/fpm modifying the same virtualenv is to avoid writing our own custom packaging/installation code. I seems we’ll be unable to avoid that entirely though if we’re trying to get all the properties we wanted that I listed in my initial Google Doc.
participants (3)
-
Brad Warren -
Nick Coghlan -
Wes Turner