[Python-Dev] Better support for consuming vendored packages

[Gregory Szorc]

On 3/22/2018 10:48 AM, Oleg Broytman wrote:
> Hi!
> 
> On Thu, Mar 22, 2018 at 09:58:07AM -0700, Gregory Szorc <gregory.szorc at gmail.com> wrote:
>> Not all consumers of Python packages wish to consume Python packages in the
>> common `pip install <package>`
> 
> IMO `pip` is for developers. To package and distribute end-user
> applications there are rpm, dpkg/deb, PyInstaller, cx_Freeze, py2exe
> (+ installer like NSIS or InnoSetup), py2app, etc...
> 
> Most of them pack a copy of Python interpreter and necessary parts
> of stdlib, so there is no problem with `sys.path` and wrong imports.

Yes, there are tools to create standalone packages. Some even bundle a
Python install so the execution environment is more deterministic. These
are great ways to distribute Python applications!

However, if you are a Python application that is maintained by your
distribution's package manager, you pretty much must use a Python
installed by the system package manager. And that leaves us with the
original problem of an undefined execution environment. So packaging
tools for standalone Python applications only work if you control the
package distribution channel. If you are a successful Python application
that is packaged by your distro, you lose the ability to control your
own destiny and must confront these problems for users who installed
your application through their distro's package manager. i.e. the cost
of success for your Python application is a lot of pain inflicted by
policies of downstream packagers.

Also, not vendoring dependencies puts the onus on downstream packagers
to deal with those dependencies. There can be package version conflicts
between various packaged Python applications ("dependency hell").
Vendoring dependencies under application-local package names removes the
potential for version conflicts.

It's worth noting that some downstream packagers do insist on unbundling
dependencies. So they may get stuck with work regardless. But if you
vendor dependencies, a downstream packager is at least capable of
packaging a Python application without having to deal with "dependency
hell."

Maybe what I'm asking for here is import machinery where an application
can forcefully limit or influence import mechanisms for modules in a
certain package. But this seems difficult to achieve given the
constraint of a single, global modules namespace (`sys.modules`) per
interpreter.