[Python-Dev] Making stdlib modules optional for distributions (Was: Breaking up the stdlib (Was: release cadence))

[Steve Dower]

I consider making stdlib modules "optional" like this to be completely separate from making them individually versioned - can't quite tell whether you guys do as well?

To everyone: please don't conflate these two discussions. The other is about CPython workflow and this one is about community/user expectations (I have not been proposing to remove stdlib modules at any point).

Cheers,
Steve

Top-posted from my Windows Phone

-----Original Message-----
From: "Petr Viktorin" <encukou at gmail.com>
Sent: ‎7/‎6/‎2016 2:04
To: "Steven D'Aprano" <steve at pearwood.info>; "Python-Dev" <python-dev at python.org>
Subject: Re: [Python-Dev] Breaking up the stdlib (Was: release cadence)

On 07/06/2016 05:11 AM, Steven D'Aprano wrote:
> On Tue, Jul 05, 2016 at 08:01:43PM +0200, Petr Viktorin wrote:
> 
>> In the tkinter case, compiling for source is easy on a developer's
>> computer, but doing that on a headless server brings in devel files for
>> the entire graphical environment.
>> Are you saying Python on servers should have a way to do turtle
>> graphics, otherwise it's not Python?
> 
> That's a really good question.
> 
> I don't think we have an exact answer to "What counts as Python?". It's 
> not like EMCAScript (Javascript) or C where there's a standard that 
> defines the language and standard modules. We just have some de facto 
> guidelines:
> 
> - CPython is definitely Python;
> - Jython is surely Python, even if it lacks the byte-code of CPython and 
>   some things behave slightly differently;
> - MicroPython is probably Python, because nobody expects to be able to 
>   run Tkinter GUI apps on an embedded device with 256K or RAM;
> 
> but it's hard to make that judgement except on a case-by-case basis.
> 
> I think though that even if there's no documented line, most people 
> recognise that there are "core" and "non-core" standard modules. dis and 
> tkinter are non-core: if µPython leaves out tkinter, nobody will be 
> surprised; if Jython leaves out dis, nobody will hold it against them; 
> but if they leave out math or htmllib that's another story.

For MicroPython, I would definitely expect htmllib to be an optional
add-on – it's not useful for reading data off a thermometer saving it to
an SD card. But I guess that's getting too deep into specifics.

> So a headless server can probably leave out tkinter; but a desktop 
> shouldn't.

Up till recently this wasn't possible to express in terms of RPM
dependencies. Now, it's on the ever-growing TODO list...

Another problem here is that you don't explicitly "install Python" on
Fedora: when you install the system, you get a minimal set of packages
to make everything work, and most of Python is part of that – but
tkinter is not. This is in contrast to python.org releases, where you
explicitly ask for (all of) Python.
Technically it would now be possible to have to install Python to use
it, but we run into another "batteries included" problem: Python (or,
"most-of-Python") is a pretty good battery for an OS.

Maybe a good short-term solution would be to make "import tkinter" raise
ImportError("Run `dnf install tkinter` to install the tkinter module")
if not found. This would prevent confusion while keeping the status quo.
I'll look into that.


> [...]
>>> The other extreme is Javascript/Node.js, where the "just use pip" (or 
>>> npm in this case) philosophy has been taken to such extremes that one 
>>> developer practically brought down the entire Node.js ecosystem by 
>>> withdrawing an eleven line module, left-pad, in a fit of pique.
>>>
>>> Being open source, the damage was routed around quite quickly, but 
>>> still, I think it's a good cautionary example of how a technological 
>>> advance can transform a programming culture to the worse.
>>
>> I don't understand the analogy. Should the eleven-line module have been
>> in Node's stdlib? Outside of stdlib, people are doing this.
> 
> The point is that Javascript/Node.js is so lacking in batteries that the 
> community culture has gravitated to an extreme version of "just use 
> pip". I'm not suggesting that you, or anyone else, has proposed that 
> Python do the same, only that there's a balance to be found between the 
> extremes of "everything in the Python ecosystem should be part of the 
> standard installation" and "next to nothing should be part of the 
> standard installation".
> 
> The hard part is deciding where that balance should be :-)

I think the balance is where it needs to be for CPython, and it's also
where it needs to be for Fedora. The real hard part is acknowledging
that it needs to be in different places for different use cases, and
making sure work to support the different use cases is coordinated.

So, I guess I'm starting to form a concrete proposal:

1) Document what should happen when a stdlib module is not available.
This should be an ImportError informative error message, something along
the lines of 'This build of Python does not include SQLite support.' or
'MicroPython does not support turtle' or 'Use `sudo
your-package-manager` install tkinter` to install this module'.

2) Document leaf modules (or "clusters") that can be removed from the
stdlib, and their dependencies. Make no guarantees about cross-version
compatibility of this metadata.

3) Standardize a way to query which stdlib modules are present (without
side effects, i.e. trying an import doesn't count)

4) Adjust pip to ignore installed stdlib modules that are present, so
distributions can depend on "statistics" and not "statistics if
python_ver<3.4". (statistics is just an example, obviously this would
only work for modules added after the PEP). For missing stdlib modules,
pip should fail with the message from 1). (Unless the "pip upgrade
asynciio" proposal goes through, in which case install the module if
it's upgradable).

5) Reserve all stdlib module names on PyPI for backports or
non-installable placeholders.

6) To ease smoke-testing behavior on Pythons without all of the stdlib,
allow pip to remove leaf stdlib modules from a venv.


Looks like it's time for a PEP.


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160706/23bf734e/attachment.html>