Thanks Nathaniel for the mention of conda.

How will separately distributed plugins work?

How should the user invoke Certbot (and maybe conda) if we don’t want to

Conda is based on a unix-like filesystem layout. There's a notion of a "prefix", which on linux systems is something like /usr or /usr/local. This can exist anywhere with conda, but there's a lib, bin, and include subfolder, for example. Windows is special, but much the same. Having that layout makes plugins easy, because the paths are both standardized and relocatable (assume relative paths most of the time). put another Python in the user’s PATH to avoid breaking other Python code on their system? Conda has this "activate" functionality. It can stay off of PATH until you need it, modify it temporarily, and also set environment variables as part of activation if you need that. There are ways to make shortcuts that run executables with a conda environment activated, too.

What should we do for systems not using 32-bit or 64-bit x86?

If we didn’t want to trust any binaries built by someone else or

Cross compiling is quite possible, though still pretty young. We have pseudo cross compilers based on crosstool-ng right now, and extending the concept to true cross compilers would be reasonably easy (but still would require elbow grease on your end.) proprietary code, how much work would that be? That depends very much on how much of the stack you feel like you need to rebuild. Just python? That's maybe a day's work at most, though more platforms means more time, and if you need to cross compile, that may mean more troubleshooting. The recipes are out there, you'd just need to either run them to build your own packages, or tweak them to your heart's content. Conda-forge is considered the authoritative source, and Anaconda forks their recipes and modifies them to implement different functionality, more optimization, or newer conda packaging features. The eventual goal is for Conda-forge and Anaconda to be in sync. Obviously forking further from Conda-forge or Anaconda implies more work for you. I work for Anaconda, and am also a core member of Conda-forge. If you have any questions or just want to talk about how conda/conda-forge may or may not meet your needs, please feel free to reach out. HTH, Michael On Mon, Jul 23, 2018 at 9:33 PM Nathaniel Smith wrote:

On Mon, Jul 23, 2018 at 4:31 PM, Brad Warren wrote:

...

Hi!

I work at the Electronic Frontier Foundation on Certbot which is the most popular end user application for obtaining and installing SSL/TLS certificates from Let’s Encrypt. Over the past few years, distributing Certbot has been one of our development team's biggest challenges and we’re currently rethinking how we do so.

It was suggested to me that I post to this list to see if anyone was interested in offering advice for how we should approach this. Of course, Certbot is written entirely in Python.

If you’re interested, I wrote up a bit of background and what we’re currently thinking at

https://docs.google.com/document/d/1y2tc65yWnGuYsun9wsXu7ZRLCVT9eih9eu0pj7Ad... .

...

Feel free to reach out to me on or off list or on IRC at bmw on Freenode.

Reading the problem description at the top of your document, my first thought was that this seemed like exactly what conda is designed for: a "real" package manager designed to be portable across platforms and work in isolation from the system package manager.

You should also look at Nix and Guix, which are the other systems I see people mention in this space.

I'm not an expert in conda at all -- if you want to go down this path you should probably have a chat with Anaconda and also conda-forge (which is a very impressive community-run packaging and build effort). I have some idea about some of the questions you raised though :-):

...

How will separately distributed plugins work?

Conda has a system they call "channels" to let third-parties distribute extra conda packages, and existing systems for using/hosting/maintaining them. (Sort of similar to Ubuntu PPAs, if you know those.)

...

How should the user invoke Certbot (and maybe conda) if we don’t want to put another Python in the user’s PATH to avoid breaking other Python code on their system?

A little shell script to set the PATH and then exec the right binary should work. Or just setting up the #! line in your main script properly.

...

What should we do for systems not using 32-bit or 64-bit x86?

I know the conda folks have some stuff for ARM, though I don't know the details.

...

If we didn’t want to trust any binaries built by someone else or proprietary code, how much work would that be?

This is where you want to talk to conda-forge – one of the original motivations was to make a community alternative to Anaconda Inc's official packages (which were not originally open-source, and do still contain proprietary code). Nowadays everyone's on better terms, but having once rebuilt the whole distro from the ground up means they can probably share some experience with you here.

-n

-- Nathaniel J. Smith -- https://vorpus.org -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/S...

On Mon, Jul 23, 2018 at 8:17 PM, Alex Walters wrote:

As a user of certbot, docker, conda, nix, and guix are non-starters. I'm not depending on those tools for my production server (and while docker may be a dependency for some people, that is hardly universal). Adding heavyweight technical dependencies are problematic if your goal is to get everyone using your software. You're better off with cx_freeze or pyinstaller binaries downloaded from a website or a PPA-like-system to add to system package managers, which are not perfect solutions either.

I would emphasize this point. The real question here is what use case you're targeting. Many sysadmin and devops type folks will simply never *ever* use a language-specific package manager; they'll tell you that for them, your tool or library doesn't exist unless it's accessible via the operating-system package manager (RPM, .deb, etc.). Language-specific package managers are at best for the users of the systems those people maintain, to handle application dependencies (and the sysadmin/devops people try, in their ideal world, to push for those to be replaced by custom system-package-manager repositories). So it's likely that there's not much utility in having this be installable via the language-specific Python packaging toolchain, because your target users will refuse to take advantage of that toolchain.

On 24 July 2018 at 16:10, James Bennett wrote:

On Mon, Jul 23, 2018 at 8:17 PM, Alex Walters wrote:

...

As a user of certbot, docker, conda, nix, and guix are non-starters. I'm not depending on those tools for my production server (and while docker may be a dependency for some people, that is hardly universal). Adding heavyweight technical dependencies are problematic if your goal is to get everyone using your software. You're better off with cx_freeze or pyinstaller binaries downloaded from a website or a PPA-like-system to add to system package managers, which are not perfect solutions either.

I would emphasize this point.

The real question here is what use case you're targeting. Many sysadmin and devops type folks will simply never *ever* use a language-specific package manager; they'll tell you that for them, your tool or library doesn't exist unless it's accessible via the operating-system package manager (RPM, .deb, etc.). Language-specific package managers are at best for the users of the systems those people maintain, to handle application dependencies (and the sysadmin/devops people try, in their ideal world, to push for those to be replaced by custom system-package-manager repositories).

So it's likely that there's not much utility in having this be installable via the language-specific Python packaging toolchain, because your target users will refuse to take advantage of that toolchain.

However, there *are* folks that have been working on allowing applications to be defined primarily as Python projects, and then have the creation of wrapper native installers be a pushbutton exercise, rather than requiring careful human handholding. In addition to FPM, which Freddy Rietdijk mentioned in another post, there are also projects like pyp2rpm, which backs the PyPI -> RPM feature in Fedora's COPR service: https://docs.pagure.org/copr.copr/user_documentation.html#pypi Regardless, given the design constraints and the target audience, it's unlikely to be possible to avoid offering a two-level solution, where the lower platform dependent layer sets up any services and cron jobs required, as well as establishing a Python virtual environment in /opt, and then the bulk of certbot, as well as any installed plugins, run in that venv (plugins would still be managed with pip, just installed into the certbot venv rather than globally). For actually building and publishing those platform dependent pieces, openSUSE's Open Build Service is the most complete offering I'm aware of: https://openbuildservice.org/ (Fedora's COPR is good for targeting RPM based distros, and Ubuntu offer their PPA system, but OBS allows creation of multiple respository formats). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia