[Distutils] Breaking up the stdlib (Was: [Python-Dev] release cadence)

[Petr Viktorin]

On 07/04/2016 12:19 AM, Steve Dower wrote:
> My thinking on this issue was that some/most packages from the stdlib
> would move into site-packages. Certainly I'd expect asyncio to be in
> this category, and probably typing. Even going as far as email and
> urllib would potentially be beneficial (to those packages, is my thinking).
> 
> Obviously not every single module can do this, but there are plenty that
> aren't low-level dependencies for other modules that could. Depending on
> particular versions of these then becomes a case of adding normal
> package version constraints - we could even bundle version information
> for non-updateable packages so that installs fail on incompatible Python
> versions.
> 
> The "Uber repository" could be a requirements.txt that pulls down wheels
> for the selected stable versions of each package so that we still
> distribute all the same code with the same stability, but users have
> much more ability to patch their own stdlib after install.
> 
> (FWIW, we use a system similar to this at Microsoft for building Visual
> Studio, so I can vouch that it works on much more complicated software
> than Python.)

While we're on the subject, I'd like to offer another point for
consideration: not all implementations of Python can provide the full
stdlib, and not everyone wants the full stdlib.

For MicroPython, most of Python's batteries are too heavy. Tkinter on
Android is probably not useful enough for people to port it. Weakref
can't be emulated nicely in Javascript.
If packages had a way to opt-out of needing the whole standard library,
and instead specify the stdlib subset they need, answering questions
like "will this run on my phone?" and "what piece of the stdlib do we
want to port next?" would be easier.

Both Debian and Fedora package some parts of the stdlib separately
(tkinter, venv, tests), and have opt-in subsets of the stdlib for
minimal systems (python-minimal, system-python). Tools like pyinstaller
run magic heuristics to determine what parts of stdlib can be left out.
It would help these projects if the "not all of stdlib is installed"
case was handled more systematically at the CPython or distutils level.

As I said on the Language summit, this is just talk; I don't currently
have the resources to drive this effort. But if anyone is thinking of
splitting the stdlib, please keep these points in mind as well. I think
that, at least, if "pip install -U asyncio" becomes possible, "pip
uninstall --yes-i-know-what-im-doing asyncio" should be possible as well.



> From: Paul Moore <mailto:p.f.moore at gmail.com>
> Sent: ‎7/‎3/‎2016 14:23
> To: Brett Cannon <mailto:brett at python.org>
> Cc: Guido van Rossum <mailto:guido at python.org>; Nick Coghlan
> <mailto:ncoghlan at gmail.com>; Python-Dev <mailto:python-dev at python.org>;
> Steve Dower <mailto:steve.dower at python.org>
> Subject: Re: [Python-Dev] release cadence (was: Request for CPython
> 3.5.3 release)
> 
> On 3 July 2016 at 22:04, Brett Cannon <brett at python.org> wrote:
>> This last bit is what I would advocate if we broke the stdlib out
> unless an
>> emergency patch release is warranted for a specific module (e.g. like
>> asyncio that started this discussion). Obviously backporting is its own
>> thing.
> 
> It's also worth noting that pip has no mechanism for installing an
> updated stdlib module, as everything goes into site-packages, and the
> stdlib takes precedence over site-packages unless you get into
> sys.path hacking abominations like setuptools uses (or at least used
> to use, I don't know if it still does). So as things stand,
> independent patch releases of stdlib modules would need to be manually
> copied into place.
> 
> Allowing users to override the stdlib opens up a different can of
> worms - not necessarily one that we couldn't resolve, but IIRC, it was
> always a deliberate policy that overriding the stdlib wasn't possible
> (that's why backports have names like unittest2...)
>