[Distutils] Using Wheel with zipimport

[Vinay Sajip]

Donald Stufft <donald <at> stufft.io> writes:

> 1. That unpacking into the home directory is problematic because users
>     that run services often don’t have home directories. This you waved
>     away by saying that the home directory isn’t a required place, to
>     which Jim responded that unpacking them anywhere else was likely to
>     have access control / security issues and “generally cause pain”.

$HOME is just a sensible default location for caches, but as long as
other locations can be used in situations where there's no HOME
directory, I don't see how that's "waving away". If a service has no
access to writeable disk locations at all, I can see how that might pose a
problem, but that hardly seems like a use case which should dictate
policies for every other scenario.

OTOH "anywhere else was likely to have access control / security issues
and 'generally cause pain'" is light on details.

>    wholly agree with, intact Mitre has assigned CVE’s to applications that
>    did switch their Egg Cache to directories other than HOME.

Are you referring to the situation where users of setuptools set 
PYTHON_EGG_CACHE to a world-writeable location? If so, the problem would 
appear to be with the using code, not setuptools itself. AFAICT it was felt
that while CVEs could be assigned against individual applications that did
this, and setuptools itself was not assigned a CVE.

If a HOME-less application uses an alternative cache location which isn't
world-writeable, does a security issue arise?

> 2. Zipped Eggs are more difficult to work with for debugging purposes, 
which
>     you responded saying that Wheels are a “deployment format” (which I
>     don’t believe is supported by the PEP at all) and Jim responded with 
his
>     experience in developing against things that got installed as zipped 
Eggs
>     and the pain that has personally caused him.
> 
> 3. Zipped imports (In Jim’s experience) are slower than unpacked imports,
>     which you replied (quite literally) "Caveat emptor”. Jim’s response 
was that
>     this had been tried with Egg and had been found to be more pain than 
it
>     was worth. In his exact words:
> 
>     "It's been tried with eggs. This is not new ground. Encouraging people 
to
>      do this is going to cause pain and resentment.
> 
>      I think one of the reasons there's so much (IMO mostly irrational) 
hate for
>      eggs is that people think you can only used zipped eggs, and zipped 
eggs
>      cause pain and agony.”
> 
> He closes up with,
> 
>     "Importing from zipped eggs has proved itself to be an anti pattern.
> 
>      Buildout (as of buildout 2) always unzips eggs.”
> 
> So you have pip that doesn’t use zipped Eggs, buildout which made sure
> in buildout 2 (presumably drawing from the lessons of buildout 1) to 
always
> unzip eggs, and easy_install which does support zipped eggs (sometimes,
> if it thinks that particular Egg will support it).
> 
> Why do we insist upon repeating the mistakes of the past?

Because there are scenarios where that functionality is desirable, and the 
downsides (such as greater difficulty in debugging, no caching of byte code, 
lower performance of imports) are tolerable. Just because something can 
cause problems in some scenarios is no reason for a wholesale ban. C makes 
segfaulting easy, but that doesn't mean that C extensions are right out ;-)

Python is generally a "consenting adults" kind of system. An important
difference from the setuptools/easy_install/zipped eggs experience is that
nothing happens automatically - a specific action has to be taken to add
wheels to sys.path, whereas in the zipped eggs case, sys.path manipulation
happens under the hood. Since you don't know it's happening, failures are
bound to seem weird - "it's nothing I've done, I'm sure!"

Making something possible != recommending its use wholesale.

> 
> Zipped Eggs have a long long history of causing very weird failures 
conditions,
> you say your Wheel.mount checks for compatibility, does it also check that
> the library to be installed isn’t using __file__ shenanigans instead of
> pkgutil.get_data()? easy_install tried to do this with varying degrees of 
success,
> I’ve never seen much except pain from zipped Eggs.

Weird failure conditions are only weird because they haven't been
investigated to find the exact cause. There's a lot of software around that
can't run from zips because of __file__ shenanigans - that doesn't negate
the very real usefulness of zipimport for other software. If I want to
write software that runs from zip, I could take care to use pkgutil rather
than __file__. Presumably, others can do the same.

Regards,

Vinay Sajip