[Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nick Coghlan]

On 1 November 2017 at 00:53, Guido van Rossum <guido at python.org> wrote:

> On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 31 October 2017 at 02:29, Guido van Rossum <guido at python.org> wrote:
>>
>>> What's your proposed process to arrive at the list of recommended
>>> packages?
>>>
>>
>> I'm thinking it makes the most sense to treat inclusion in the
>> recommended packages list as a possible outcome of proposals for standard
>> library inclusion, rather than being something we'd provide a way to
>> propose specifically.
>>
>
> I don't think that gets you off the hook for a process proposal. We need
> some criteria to explain why a module should be on the recommended list --
> not just a ruling as to why it shouldn't be in the stdlib.
>

The developer guide already has couple of sections on this aspect:

* https://devguide.python.org/stdlibchanges/#acceptable-types-of-modules
* https://devguide.python.org/stdlibchanges/#requirements

I don't think either of those sections is actually quite right (since we've
approved new modules that wouldn't meet them), but they're not terrible as
a starting point in general, and they're accurate for the recommended
packages use case.


> We'd only use it in cases where a proposal would otherwise meet the
>> criteria for stdlib inclusion, but the logistics of actually doing so don't
>> work for some reason.
>>
>
> But that would exclude most of the modules you mention below, since one of
> the criteria is that their development speed be matched with Python's
> release cycle. I think there must be some form of "popularity" combined
> with "best of breed". In particular I'd like to have a rule that explains
> why flask and Django would never make the list. (I don't know what that
> rule is, or I would tell you -- my gut tells me it's something to do with
> having their own community *and* competing for the same spot.)
>

The developer guide words this as "The module needs to be considered
best-of-breed.". In some categories (like WSGI frameworks), there are
inherent trade-offs that mean there will *never* be a single best-of-breed
solution, since projects like Django, Flask, and Pyramid occupy
deliberately different points in the space of available design decisions.

Running the initial 5 proposals through that filter:
>>
>> * six: a cross-version compatibility layer clearly needs to be outside
>> the standard library
>>
>
> Hm... Does six still change regularly? If not I think it *would* be a
> candidate for actual stdlib inclusion. Just like we added u"..." literals
> to Python 3.4.
>

It still changes as folks porting new projects discover additional
discrepancies between the 2.x and 3.x standard library layouts and
behaviour (e.g. we found recently that the 3.x subprocess module's
emulation of the old commands module APIs actually bit shifts the status
codes relative to the 2.7 versions). The rate of change has definitely
slowed down a lot since the early days, but it isn't zero.

In addition, the only folks that need it are those that already care about
older versions of Python - if you can start with whatever the latest
version of Python is, and don't have any reason to support users still
running older version, you can happily pretend those older versions don't
exist, and hence don't need a compatibility library like six. As a separate
library, it can just gracefully fade away as folks stop depending on it as
they drop Python 2.7 support. By contrast, if we were to bring it into the
3.x standard library, then we'd eventually have to figure out when we could
deprecate and remove it again.


> * setuptools: we want to update this in line with the PyPA interop specs,
>> not the Python language version
>>
>
> But does that exclude stdlib inclusion? Why would those specs change, and
> why couldn't they wait for a new Python release?
>

The specs mainly change when we want to offer publishers new capabilities
while still maintaining compatibility with older installation clients (and
vice-versa: we want folks still running Python 2.7 to be able to publish
wheel files and use recently added metadata fields like
Description-Content-Type).

The reason we can't wait for new Python releases is because when we add
such things, we need them to work on *all* supported Python releases
(including 2.7 and security-release-only 3.x versions).

There are also other drivers for setuptools updates, which include:

- operating system build toolchain changes (e.g. finding new versions of
Visual Studio or XCode)
- changes to PyPI's operations (e.g. the legacy upload API getting turned
off due to persistent service stability problems, switching to HTTPS only
access)

With setuptools as a separate project, a whole lot of package publication
problems can be solved via "pip install --upgrade setuptools wheel" in a
virtual environment, which is a luxury we don't have with plain distutils.


>  * cffi: updates may be needed for PyPA interop specs, Python
> implementation updates or C language definition updates
>
> Hm, again, I don't recall that this was debated -- I think it's a failure
> that it's not in the stdlib.
>

A couple of years ago I would have agreed with you, but I've spent enough
time on packaging problems now to realise that cffi actually qualifies as a
build tool due to the way it generates extension module wrappers when used
in "out-of-line" mode.

Being outside the standard library means that cffi still has significant
flexibility to evolve how it separates its buildtime functionality from its
runtime functionality, and may eventually be adjusted so that only
"_cffi_backend" needs to be installed at runtime for the out-of-line
compilation mode, without the full C header parsing and inline extension
module compilation capabilities of CFFI itself (see
https://cffi.readthedocs.io/en/latest/cdef.html#preparing-and-distributing-modules
for the details). Being separate also means that cffi can be updated to
generate more efficient code even for existing Python versions.


> * requests: updates are more likely to be driven by changes in network
>> protocols and client platform APIs than Python language changes
>>
>
> Here I agree. There's no alternative (except aiohttp, but that's
> asyncio-based) and it can't be in the stdlib because it's actively being
> developed.
>
>
>> * regex: we don't want two regex engines in the stdlib, transparently
>> replacing _sre would be difficult, and _sre is still good enough for most
>> purposes
>>
>
> I think this needn't be recommended at all. For 99.9% of regular
> expression uses, re is just fine. Let's just work on a strategy for
> introducing regex into the stdlib.
>

Given your informational PEP suggestion below, I'd probably still include
it, but in a separate section from the others (e.g. the others might be
listed as "Misaligned Feature Release Cycles", which is an inherent
logistical problem that no amount of coding can fix, while regex would
instead be categorised as "Technical Challenges").


> Of the 5, I'd suggest that regex is the only one that could potentially
>> still make its way into the standard library some day - it would just
>> require someone with both the time and inclination to create a CPython
>> variant that used _regex instead of _sre as the default regex engine, and
>> then gathered evidence to show that it was "compatible enough" with _sre to
>> serve as the default engine for CPython.
>>
>> For the first four, there are compelling arguments that their drivers for
>> new feature additions are such that their release cycles shouldn't ever be
>> tied to the rate at which we update the Python language definition.
>>
>
> As you can tell from my arguing, the reasons need to be written up in more
> detail.
>
>
>> And is it really just going to be a list of names, or is there going to
>>> be some documentation (about the vetting, not about the contents of the
>>> packages) for each name?
>>>
>>
>> I'm thinking a new subsection in https://docs.python.org/devgui
>> de/stdlibchanges.html for "Recommended Third Party Packages" would make
>> sense, covering what I wrote above.
>>
>
> That's too well hidden for my taste.
>
>
>> It also occurred to me that since the recommendations are independent of
>> the Python version, they don't really belong in the version specific
>> documentation.
>>
>
> But that doesn't mean they can't (also) be listed there. (And each
> probably has its version dependencies.)
>
>
>> While the Developer's Guide isn't really the right place for the list
>> either (except as an easier way to answer "Why isn't <X> in the standard
>> library?" questions), it could be a good interim option until I get around
>> to actually writing a first draft of https://github.com/python/redi
>> stributor-guide/ (which I was talking to Barry about at the dev sprint,
>> but didn't end up actually creating any content for since I went down a
>> signal handling rabbit hole instead).
>>
>
> Hm, let's not put more arbitrary check boxes in the way of progress. Maybe
> it can be an informational PEP that's occasionally updated?
>

If I'm correctly reading that as "Could the list of Recommended Third Party
Packages be an informational PEP?", then I agree that's probably a good way
to tackle it, since it will cover both the developer-centric "Why isn't
this in the standard library yet?" aspect *and* the "Redistributors should
probably provide this" aspect.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20171101/9731b938/attachment-0001.html>