[Distutils] Using Wheel with zipimport

[Vinay Sajip]

Paul Moore <p.f.moore <at> gmail.com> writes:

> 1. It does *not* just use the fact that wheels are importable. It goes
> beyond that and *extracts* C extensions to make them importable, too.
> That is a workaround for a known and accepted limitation of zipimport,
> and as a workaround, it has issues. If C extensions in zipfiles could
> work reliably, this should go into zipimport itself, and *not* into
> 3rd party code. Then everyone would benefit.

(a) The mount method does not do the extraction of C extensions
    *unconditionally* - only when there is suitable indication in the
    metadata.

(b) What are these issues to which you refer? As I said in my other
    response, I don't see the zipped-egg problem as the same, because that
    sys.path manipulation is under the hood/not under the developer's
    control.

(c) Why can't third-party code break new ground, which may or may not
    prove fertile? If considered beneficial, it could always be added
    to zipimport at a later date. That's how a lot of functionality has
    entered the stdlib, after all.

> 2. It makes what should be a rare use case, to be used only when the
> code in the wheel has been carefully checked to work from a zipfile,
> seem like a common and straightforward operation. (The "attractive
> nuisance" argument). I believe that people using this API will
> typically *not* check the code, and will blame the wheel format, or
> distlib, when their application does not work as expected.

Doesn't the same argument apply to zipimport? Any code which is zipimported
shouldn't use __file__ manipulation to access package resources, for
example. Are you saying that you also feel wheels should never be
importable, since one can't guarantee whether any code in general will
work correctly from a zip? And yet, we have PEP 441, which is intended to
encourage use of zipimport. Do we say, "no point - there's bound to be
people out there who'll use __file__ instead of pkgutil"?

> 3. It is no easier than sys.path.insert(0, wheelname). All it adds
> over that is compatibility checking and the ability to import C
> extensions (see above on why I think that's a bad thing). As for
> compatibility checking, I'd prefer a distlib.wheel.check_compatibility
> API that people could call *before* manually inserting the wheel onto
> sys.path. That's a better separation of concerns, in my view.

There's no problem with providing a compatibility check API, but the
other POV is that people might forget to call that API, so it seems to
make sense to call it from the mount method too. And I don't see why you
say "all it adds" - surely the check is important to prevent a certain
class of "weird things" happening.

> I can't give specific examples of "failure modes" because I don't use
> the wheel mount functions, nor do I typically add wheels to sys.path.
> When I did (in virtualenv) I hit a number of issues, but all of these
> were ones I fixed in user code (either in virtualenv itself, or in
> pip, which was the wheel I was importing). So you could reasonably

When I looked into running pip from a zip, it was clear that the issues
were related to pip's use of __file__ and such, and nothing to do
with shortcomings in the wheel format or zipimport.

> dismiss these as "not related to the mount API" to which all I can say
> is that if I'd been able to do wheel.mount(wheelname) I would likely
> have put less thought into whether what I was doing was a good idea -
> and *that's* what I think is the bad aspect of the API.

Again, this feels like saying "zipimport is bad because people use
__file__ instead of pkgutil". Having any API available doesn't absolve
people from the responsibility of thinking about what they're doing and
what the implications of using that API are.

> Ultimately, I'll just never use the distlib mount functionality, and
> I'll recommend not using it when (if) people ask. But I'd rather it
> were not there to prompt the question.

You don't want it to be there, even if it might be useful to others,
just because it isn't useful to you? It's not as if distlib is forcing
that functionality on anyone.

Regards,

Vinay Sajip