[Distutils] wacky idea about reifying extras

[Donald Stufft]

On October 26, 2015 at 3:36:47 AM, Nathaniel Smith (njs at pobox.com) wrote:
> > TL;DR
> -----
>  
> If we:
>  
> - implement a real resolver, and
> - add a notion of a per-project namespace of distribution names,  
> that
> are collected under the same PyPI registration and come from  
> the same
> sdist, and
> - add Conflicts:, and Provides:,
>  
> then we can elegantly solve a collection of important and difficult  
> problems, and we can retroactively pun the old extras system  
> onto the
> new system in a way that preserves 100% compatibility with all  
> existing packages.
>  
> I think?
>  
> What do you think?

My initial reaction when I started reading your idea was that I didn't see a
point in having something like foo[bar] be a "real" package when you could just
as easily have foo-bar. However, as I continued to read through the idea it
started to grow on me. I think I need to let it percolate in my brain a little
bit, but there may be a non-crazy (or at least, crazy in a good way) idea here
that could push things forward in a nice way.

Some random thoughts:

* Reusing the extra syntax is nice because it doesn't require end users to
  learn any new concepts, however we shouldn't take a new syntax off the table
  either if it makes the feature easier to implement with regards to backwards
  compatability. Something like numpy{mkl,some-other-thing} could work just as
  well too. We'll need to make sure that whatever symbols we choose can be
  represented on all the major FS we care about and that they are ideally non
  ugly in an URL too. Of course, the filename and user interface symbols don't
  *need* to match. It could just as easily example numpy[mkl] out to numpy#mkl
  or whatever which should make it easier to come up with a nice scheme.

* Provides is a bit of an odd duck, I think in my head I've mostly come to
  terms with allowing unrestricted Provides when you've already installed the
  package doing the Providing but completely ignoring the field when pulling
  data from a repository. Our threat model assumes that once you've selected to
  install something then it's generally safe to trust (though we still do try
  to limit that). The problem with Provides mostly comes into play when you
  will respect the Provides: field for any random package on PyPI (or any other
  repo).

* The upgrade mess around extras as they stand today could also be solved just
  by recording what extras (if any) were selected to be installed so that we
  keep a consistent view of the world. Your proposal is essentially doing that,
  just by (ab)using the fact that by installing a package we essentially get
  that aspect of it for "free".

* Would this help at all with differentiating between SSE2 and SSE3 builds and
  things like that? Or does that need something more automatic to be really
  usable?

* PEP 426 (I think it was?) has some extra syntax for extras which could
  probably be really nice here, things like numpy[*] to get *all* of the extras
  (though if they are real packages, what even is "all"?). It also included
  (though this might have been only in my head) default to installed packages
  which meant you could do something like split numpy into numpy[abi2] and
  numpy[abi3] packages and have the different ABIs actually contained within
  those other packages. Then you could have your top level package default to
  installing abi3 and abi2 so that ``pip install numpy`` is equivilant to
  ``pip install numpy[abi2,abi3]``. The real power there, is that people can
  trim down their install a bit by then doing ``pip install numpy[-abi2]`` if
  they don't want to have that on-by-default feature.

I'm going to roll through around in my head more, and hopefully more people
with other interesting problems to solve can chime in and say whether they
think this would solve their problems or not. My less-than-immediate reaction
now that I'm through the entire thing is that it seems like it could be good
and I'm struggling to think of a major downside.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA