[Distutils] Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script)

[Vinay Sajip]

Paul Moore <p.f.moore <at> gmail.com> writes:

> I'm not 100% sure what your proposal is here - I'm confused about the precise
> roles of setup.py/setuptools as "builder" vs distil as "builder" vs distlib
> as "wheel builder" vs distlib as "wheel installer". I'll try to get some time
> in the next day or so to review the code and make sure we're not talking at
> cross purposes here. Bear with me.

I'm not sure I've got a concrete proposal yet - I'm still thinking things
out, too :-)

Currently, the wheel module in distlib doesn't do anything too clever - just
the mechanical stuff of archiving and unarchiving. Because the exports stuff
wasn't in the PEP, distil wrote the generated scripts and .exe wrappers to
the work area, and so they ended up in the wheel. (This is the case for all
released versions of distil).

> I understood that distlib simply built wheels from whatever is in the
> directories supplied. I'm not clear where it would get the "exports" metadata
> to store in the wheel. Assuming that's sorted, though, then whether or not

Right now it's getting the exports courtesy of distil when you use that to
build the wheel - distlib can't rely on that information being available
because it's not part of the PEP.

If the PEP is updated to include the exports, they should be in the wheel no
matter which tool builds it. Then in theory distlib could generate the
scripts during installation, but there are a lot of options to consider -
did setuptools put them in there already? Do we want native launchers? etc.
which is perfectly doable in distlib, but I'm not sure that's the best place
for it because I think wheel processing should be uncomplicated.
Wheel.install already has quite a few kwargs:

dry_run=False: Don't actually do anything
executable=None:Custom executable for shebang lines (e.g. to support path
searching)
warner=None:Used to defer warning behaviour to calling application
lib_only=False:Process site-packages contents only (you suggested this)

I'd like to not have to add any more, unless it's unavoidable :-)

Nevertheless, I will probably try implementing it in distlib as an
experiment, too see how it looks.

> distlib processes the exports on install is not an issue to me (I think it
> always should). What is more of an issue is what "the thing that puts stuff
> into the directories" does. If that's a setuptools-based build, it will
> process the exports data *itself* and put wrappers into the scripts
> directory. That is what I think we should be trapping and suppressing. But
> we have to be careful, as if setuptools is used to install directly, *not*
> going via a wheel, it has to generate wrappers itself somehow.

Exactly why I'm so leery of putting this logic in distlib, until we think it
through and add it to the PEP. At the moment distil does it the same way as
setuptools/pip only to remain compatible, not for any other reason.

> Using distil to do the build is a whole other route.

Right. I'm aiming for distil to be able to do just about everything pip can
(functionally, the code is pretty much there barring installs from DVCS
URLs), but backward compatibility is always a concern and a challenge :-)

Another complication for distlib is that I expect it to work in 2.6+, where
you can't always rely on the py launcher being present - hence the wrappers
in distlib, with flags to disable writing them out.

Another area to consider for scripts is which of foo, fooX and foo-X.Y to
write to the scripts folder. This is particularly important in user
site-packages, where scripts for different Python versions will coexist in
the same folder, and the possibility exists of overwriting, which sometimes
leads to unexpected behaviour (e.g. if I install foo-dist using 2.x which
installs foo and foo-2.x to ~/.local/bin, then install it using 3.x, it
would write foo and foo-3.x to ~/.local/bin. Quite apart from headaches with
native launchers, it could be that the foo script installed with 3.x doesn't
work (e.g. if it tries to process Python code which is 2.x compatible but
not 3.x compatible).

Fun and games!

Regards,

Vinay Sajip