[Distutils] latest setuptools appears to require six in a breaking way

[Donald Stufft]

> On Jan 27, 2017, at 2:05 PM, Eric Brunson <brunson at brunson.com> wrote:
> 
> On Wed, Jan 25, 2017 at 5:52 PM Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> 
>> On Jan 25, 2017, at 12:27 PM, Eric Brunson <brunson at brunson.com <mailto:brunson at brunson.com>> wrote:
>> 
>> It wasn't until recently the I realized how quickly releases to setuptools and pip are being made, starting back in mid Dec when much of our dependency resolution started failing.  There were three releases in the past two days.  Four major and 22 minor releases in the past two months.  While I applaud the speed of development and the improvement in these tools, don't you feel that breaking changes should be advertised better before release or perhaps we should slow down the cadence for release?
>> 
>> I think an expectation that every setuptools user in the community start their day by checking to see if there was a release in the past 24 hours is a little unreasonable.  I've spent a dozen hours since 32.0.0 resolving breakage in my own projects and assisting other developers in my org with their setuptools issues, all the while pushing setuptools as the best practice to do development and distribution.  Is this period of breaking changes a short term thing that we just have to tough out for a few more weeks?
>> 
>> Thanks,
>> e.
>> 
> 
> 
> I don’t believe that pip is really releasing that quickly. We generally make 1-2 “major” versions a year that include breaking changes, 2-4 “minor” releases a year that add new features, and 6-10 patch releases that fix bugs. To me that feels like a pretty decent pace of balancing not breaking people and getting new changes into people’s hands and getting rid of broken or less optimal parts of the code.
> 
> Now, setuptools is releasing faster than pip is and whether that’s a good thing or not I don’t know. That’s a question for Jason largely :)
> 
> Hey Donald,
> 
> Thanks for the reply.
> 
> Doesn't pip rely heavily on setuptools?  I understand they have different origins, but I thought that since pip was moved under the purview of PYPA a lot of work was being done to converge the projects. When I run a pip -e one of the last message I see is "running setuptools.py develop", which isn't really a dependency, but can certainly cause people to infer that the problem is with pip and not know setuptools. Even the release notes Matthias references mentions pip as though it might be affected.
> 
> If pip doesn't rely on setuptools, does that mean we have two separate and possibly different dependency resolution algorithms?
> 
> Sincerely,
> e.


The answer here is… complex.

Pip itself does not depend on setuptools though we _do_ depend on pkg_resources, but like all of our dependencies we bundle a copy internally and use that. Our uses of pkg_resources is primarily to inspect the currently installed packages to see what versions of what have been installed.

If you’re installing from a Wheel file with pip, then you do not need to have setuptools installed at all (although if the wheel depends on setuptools we will obviously install it then).

If you’re installing from a sdist, *then* you need setuptools installed because we’re going to execute the ``setup.py`` that is inside of that sdist and it will use setuptools (and we do some gross hacks to ensure it *always* uses setuptools, even if the project imports distutils).


We are careful to pass in to setuptools the correct arguments that disables setuptools own handling of dependencies and we implement our own resolver. In the cases setuptools is being used (besides our bundled pkg_resources) it is primarily being used as a build tool. That being said, setuptools currently handles setup_requires on it’s own so the setuptools resolver handles that (though PEP 518 provides a way that in the future pip will use so that pip can handle build requirements instead of needing setuptools to do it). There is also a little known ``—egg`` parameter to pip that tells pip to stop preventing setuptools from doing it’s own resolving… but that is deprecated and is either removed or will be removed in the next version.

So yes, they do have two separate and different dependency resolution algorithms (for instance, pip won’t install pre-releases by default unless it’s all that is available while setuptools will) but when you’re using ``pip install`` the setuptools one is carefully disabled *except* for setup_requires.

—
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170127/5b7c529c/attachment.html>