[Distutils] setup.py install using pip

[Erik Bray]

On Mon, Dec 7, 2015 at 11:45 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 7 December 2015 at 16:21, Erik Bray <erik.m.bray at gmail.com> wrote:
>> Exactly--as both a library developer / maintainer and system
>> integrator I would find such a flag very useful (especially since I
>> can just set it in a config file and forget it).  It would be right
>> for me.  But wouldn't break anything for anyone else.
>>
>> Ironically, the default behavior of `setup.py install`, on projects
>> that use setuptools, is to install an egg directory which is
>> *definitely* not for everybody, especially not anymore.  That's why
>> --single-version-externally-managed exists.  A --pip flag would be
>> very much like --single-version-externally-managed (sort of a
>> specialized extension of it) that also includes "do everything pip
>> does" which includes installing dependencies and copying
>> .egg-info/.dist-info to the appropriate location, which is what I want
>> to replace all instances of `setup.py install` with.  That includes
>> users running `setup.py install`, who have a hard enough time as it is
>> keeping up with Python build/installation "best practices" as it is.
>
> One thing that bothers me about this proposal. If someone does "pip
> install --no-binary" for your package, and you have the "--pip" flag
> in your setup.cfg, pip will use "setup.py install" to do the install.
> Which, if I understand this proposal correctly, will attempt to "fall
> back" to pip because "--pip" is in setup.cfg. Which results in an
> infinite loop of pip and setup.py invoking each other.

I wasn't able to produce this problem.  Even with --no-binary
specified pip installs (by default) with
--single-version-externally-managed.  My prototype implicitly disables
the --pip flag if --single-version-externally-managed was specified
(true to the purpose of that flag).

What *is* a problem is if --pip is in setup.cfg, and one invokes `pip
install --egg .`.  I wasn't quite able to make that go into an
infinite loop, but it did invoke pip.main recursively, and stuff broke
on the second invocation for reasons not clear to me.

This is easily worked around, however, by detecting, from the install
command, if we're already using pip.  As a quick hack I added to
finalize_options:

if 'pip' in sys.modules:
    self.pip = False

This did the trick.  pip ran fine and installed the package as an egg
using the standard setup.py install.  I don't think a more robust
solution would be hard.  pip could set an environment variable or even
a variable in the pip module itself to indicate that pip is already
being invoked to install this package.  There may even be something
like that already in pip that I'm not aware of.

> I'm not sure how pip could detect a situation like this, so there's a
> risk of some *very* obscure corner cases, which I'm sure people will
> end up hitting.

As mentioned above I don't think it should be pip's job to detect this
situation.  But if the setuptools install command can detect that
we're already in pip then the job is done.

> As a user level command line flag, "setup.py install --pip" isn't much
> better than "pip install ."
> As a project config, we get the issue noted above, and the user has to
> edit the project code to fix it.
> As a per-user or global config, we get the issue above, but we could
> reasonably say it's the user's mistake and the user has the means to
> fix it. But it's still not a great UX.

I don't think so either.  But it's also not great UX that we've told
users for decades that the way to install a Python package is to run
`python setup.py install`, but now the default behavior of that (for
packages using setuptools) which is to install a .egg, is old and bad.
I get confusion about this from users *frequently*.  It's only worse
that egg installs and flat installs are incompatible with each other
with respect to namespace packages.

If it would help, `setup.py install --pip` could also display a
warning that users should run `pip install .` instead.  To cut down on
noise I might only do this if the --pip option came from setup.cfg,
rather than when it's explicitly asked for via the command line.  This
would serve to inform users who don't know any better.

> It's quite possible I'm missing something here (at a minimum, I'm
> making huge assumptions about how the feature would be implemented)
> but I think the behaviour needs to be thought through in a bit more
> detail.

Completely agree!  I know there are corner cases I haven't thought of.
I'm also undecided on whether --pip should invoke pip.main() within
the same process, or run pip from a subprocess.  The former seems
sufficient but I don't know all the cases.

Thanks,
Erik