[Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Nathaniel Smith]

On Sat, Nov 7, 2015 at 4:02 PM, Donald Stufft <donald at stufft.io> wrote:
> On November 7, 2015 at 6:43:50 PM, Nathaniel Smith (njs at pobox.com) wrote:
>> On Sat, Nov 7, 2015 at 3:16 PM, Donald Stufft wrote:
>> [...]
>> > The outcome of all of these types of installs are subtly different and we’ve had a number
>> of users regularly get confused when they act differently over the years. I do not think
>> it’s possible to make (5) act like anything else because it is inherently different,
>> however I think we can get to the point that 1-4 all act the exact same way. and I think the
>> way to do it is to change these so instead it is like:
>> >
>> > 1. I have a wheel and pip installs it.
>> > 2. I have an sdist and pip turns it into a wheel and then pip installs it.
>> > 3. I have a directory and pip turns it into a sdist and then pip turns that sdist into a wheel
>> and then pip installs it.
>> > 4. I have a directory and pip installs it in editable mode.
>>
>> I wrote some more detailed comments on this idea in the reply I just
>> posted to Paul's message, but briefly, the alternative way to approach
>> this would be:
>>
>> 1. I have a wheel and pip installs it
>> 2. I have an sdist and pip unpacks it into a directory and builds a
>> wheel from that directory and then pip installs it.
>> 3. I have a directory and pip builds a wheel from that directory and
>> then pip installs it.
>> 4. I have a directory and pip installs it in editable mode.
>>
>> This is actually simpler, because we've eliminated the "create an
>> sdist" operation and replaced it with the far-more-trivial "unpack an
>> sdist". And it isn't even a replacement, because your 2 and my 2 are
>> actually identical when you look at what it means to turn an sdist
>> into a wheel :-).
>>
>
> The problem is that an sdist and a directory are not the same things even though they may trivially appear to be. A very common problem people run into right now is that they don’t adjust their MANIFEST.in so that some new file they’ve added gets included in the sdist. In the current system and your proposed system if someone types ``pip install .`` that just silently works. Then they go “Ok great, my package works” and they create a sdist and send that off… except the sdist is broken because it’s missing that file they needed.
>
> Since we’ve disabled the ability to delete + reupload files to PyPI I get probably once or twice a week someone contacting me asking if I can let them re-upload a file because they created an sdist that was missing a file. A decent number of those told me that they had “tested” it by running ``pip install .`` or ``setup.py install`` into a fresh virtual environment and that it had worked.
>
> It’s true that the MANIFEST.in system exacerbates this problem by being a pretty crummy and error prone system to begin with, however the same thing is going to exist for any system where you have a path that builds a sdist (and may or may not include a file in that sdist) and a path that goes direct to wheel.
>
> It might not only be files that didn’t get to be included because of a mistake either. Some files might not get generated until sdist build time, something like LXML generates .c files from Cython sources at sdist creation time and then they build a Wheel from those .c files. They do this to prevent people from needing to have Cython available on a machine other than a development machine. In your proposed work flow their “build wheel” command needs to be able to deal with the fact that the .c files may or may not be available (and will need to figure out a way to indicate that Cython is a build dependency if they are not). In my proposed workflow their wheel build command gets to be simpler, it only needs to deal with .c files and their sdist command gets used to create the .c files while the sdist is being generated.

I'm not sure how to respond, because I sympathize and agree with all
of these points, but I just think that the trade-offs are such that
pip is the wrong place to try and fix this.

Even if pip always copies the source tree to a temp dir, or even
builds an sdist and unpacks it to a temp dir, then this doesn't
actually guarantee that the final distribution will work, because of
the reasons I mentioned in my other email -- you can still forget to
check things in, have random detritus in your working directory
(orphaned .pyc files create all kinds of fun, since python will
happily import them even if the corresponding .py file has been
deleted), etc.

Which isn't to say that it's hopeless to try and improve matters, but
I don't think we should do so at the expense of adding otherwise
unneeded complexity to the pip <-> project-build-system interface.
("Otherwise unneeded" because nothing else in pip cares about
generating sdists.)

And just in general, your plan to improve matters has an ocean-boiling
feel to it to me, because you're pushing against a huge weight of
history and conventions that expect builds to happen inside the source
tree and to re-use partial build results etc. Convincing people to
lengthen their edit/compile/test cycle is almost always a losing
proposition, no matter how good your reason is. If people just refuse
to use pip in favor of setup.py install, then what have we really
gained? Paving cow paths, remember...

So I think that on balance, the right place to tackle this problem is
within the build system itself. Heck, bdist_wheel could probably today
be modified to call sdist, unpack the resulting sdist, and then
perform the resulting build, right? It'd be slow, but it'd work. And
it would leave our options open for later, without enshrining this
into the Standard Build System Interface that can't be changed without
PEPs and elaborate transition plans.

And there are all sorts of strategies that a new build system could
use to guarantee reliable sdists (e.g. using the same source-of-truth
for locating files to build as it uses for locating files to include
in the sdist!) without giving up on incremental rebuilds, so long as
pip doesn't just rule out incremental builds entirely. LXML's build
system can use the sdist->wheel strategy if they think it makes sense
to them -- they don't need anything from pip to do that.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org