[Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Donald Stufft]

On November 7, 2015 at 6:43:50 PM, Nathaniel Smith (njs at pobox.com) wrote:
> On Sat, Nov 7, 2015 at 3:16 PM, Donald Stufft wrote:
> [...]
> > The outcome of all of these types of installs are subtly different and we’ve had a number  
> of users regularly get confused when they act differently over the years. I do not think  
> it’s possible to make (5) act like anything else because it is inherently different,  
> however I think we can get to the point that 1-4 all act the exact same way. and I think the  
> way to do it is to change these so instead it is like:
> >
> > 1. I have a wheel and pip installs it.
> > 2. I have an sdist and pip turns it into a wheel and then pip installs it.
> > 3. I have a directory and pip turns it into a sdist and then pip turns that sdist into a wheel  
> and then pip installs it.
> > 4. I have a directory and pip installs it in editable mode.
>  
> I wrote some more detailed comments on this idea in the reply I just
> posted to Paul's message, but briefly, the alternative way to approach
> this would be:
>  
> 1. I have a wheel and pip installs it
> 2. I have an sdist and pip unpacks it into a directory and builds a
> wheel from that directory and then pip installs it.
> 3. I have a directory and pip builds a wheel from that directory and
> then pip installs it.
> 4. I have a directory and pip installs it in editable mode.
>  
> This is actually simpler, because we've eliminated the "create an
> sdist" operation and replaced it with the far-more-trivial "unpack an
> sdist". And it isn't even a replacement, because your 2 and my 2 are
> actually identical when you look at what it means to turn an sdist
> into a wheel :-).
>  

The problem is that an sdist and a directory are not the same things even though they may trivially appear to be. A very common problem people run into right now is that they don’t adjust their MANIFEST.in so that some new file they’ve added gets included in the sdist. In the current system and your proposed system if someone types ``pip install .`` that just silently works. Then they go “Ok great, my package works” and they create a sdist and send that off… except the sdist is broken because it’s missing that file they needed.

Since we’ve disabled the ability to delete + reupload files to PyPI I get probably once or twice a week someone contacting me asking if I can let them re-upload a file because they created an sdist that was missing a file. A decent number of those told me that they had “tested” it by running ``pip install .`` or ``setup.py install`` into a fresh virtual environment and that it had worked.

It’s true that the MANIFEST.in system exacerbates this problem by being a pretty crummy and error prone system to begin with, however the same thing is going to exist for any system where you have a path that builds a sdist (and may or may not include a file in that sdist) and a path that goes direct to wheel.

It might not only be files that didn’t get to be included because of a mistake either. Some files might not get generated until sdist build time, something like LXML generates .c files from Cython sources at sdist creation time and then they build a Wheel from those .c files. They do this to prevent people from needing to have Cython available on a machine other than a development machine. In your proposed work flow their “build wheel” command needs to be able to deal with the fact that the .c files may or may not be available (and will need to figure out a way to indicate that Cython is a build dependency if they are not). In my proposed workflow their wheel build command gets to be simpler, it only needs to deal with .c files and their sdist command gets used to create the .c files while the sdist is being generated.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA