I've read through your document. The question is interesting. You have exactly the same bootstrapping problem that pip has. If I were in your shoes here's how I would architect a solution:
1 - If you can shift your dependencies to be pure python, do it. If you can't, stop distributing certbot-auto outside of a package manager so you can avoid having to build a badly implemented package manager inside certbot-auto.
2 - Move to requiring Python 3.4+ so you can rely on the presence of venv and pip.
3 - Continue using virtualenvs to bootstrap to the latest version of certbot
4 - Use pip to install plugins for certbot.
You've already identified why things like Docker or Snaps won't work for your use case. You could rearchitect to use system package managers and download files at runtime to update your logic without overwriting what the system package manager installs. This would be a lot of work for little gain. Therefore you've got to use your own update mechanism. Since you're written in python you might as well use pip and virtualenv to solve some of your problems. Especially since that's what you're using now.
End goal would be something like this:
sudo apt-get install certbot
- now you have the non-python dependencies and an (old) implementation of certbot that can update itself
sudo certbot <make some certificates>
- certbot creates a virtualenv
- certbot pip installs itself inside that virtualenv
- certbot calls certbot inside virtualenv to do real work
- certbot notices the system is using nginx, asks user if they want the plugin for nginx, user says 'yes'
- certbot uses pip to install certbot-nginx plugin
- certbot reloads itself to enable plugin
I think I'd separate out certbot installed by the package manager which is just a simple bootstrapper and the certbot installed in the virtualenv which does real work. The bootstrapper updates rarely and does so via package managers. The virtualenv gets checked for updates and updated every time the bootstrapper runs, including in cron jobs.
Take all that with a huge helping of salt, I've got zero reputation around here.
-Eli Ribble