[Python-Dev] Standard library vs Standard distribution?

[Paul Moore]

On Thu, 29 Nov 2018 at 21:33, Nathaniel Smith <njs at pobox.com> wrote:
>
> On Thu, Nov 29, 2018 at 10:22 AM Antoine Pitrou <antoine at python.org> wrote:
> >
> > Le 29/11/2018 à 19:07, Steve Dower a écrit :
> > > On 29Nov2018 0923, Antoine Pitrou wrote:
> > >> I think the whole argument amounts to hand waving anyway.  You are
> > >> inventing an extended distribution which doesn't exist (except as
> > >> Anaconda) to justify that we shouldn't accept more modules in the
> > >> stdlib.  But obviously maintaining an extended distribution is a lot
> > >> more work than accepting a single module in the stdlib, and that's why
> > >> you don't see anyone doing it, even though people have been floating the
> > >> idea for years.
> > >
> > > https://anaconda.com/
> > > https://www.activestate.com/products/activepython/
> > > http://winpython.github.io/
> > > http://python-xy.github.io/
> > > https://www.enthought.com/product/canopy/
> > > https://software.intel.com/en-us/distribution-for-python
> > > http://every-linux-distro-ever.example.com
> > >
> > > Do I need to keep going?
> >
> > I'm sure you could.  So what?  The point is that it's a lot of work to
> > maintain if you want to do it seriously and with quality standards that
> > would actually _satisfy_ the people for whom PyPI is not an option.
>
> Yeah, I draw two conclusions from the list above:
>
> - Paul expressed uncertainty about how many people are in his position
> of needing a single download with all the batteries included, but
> obviously he's not alone. So many people want a
> single-box-of-batteries that whole businesses are being built on
> fulfilling that need.
>
> - Currently, our single-box-of-batteries is doing such a lousy job of
> solving Paul's problem, that people are building whole businesses on
> our failure.

Ouch. Congratulations on neatly using my own arguments to reverse my
position :-)

Those are both very good points. However...

> If Python core wants to be in the business of providing a
> single-box-of-batteries that solves Paul's problem, then we need to
> rethink how the stdlib works. Or, we could decide we want to leave
> that to the distros that are better at it, and focus on our core
> strengths like the language and interpreter. But if the stdlib isn't a
> single-box-of-batteries, then what is it?

IMO, the CPython stdlib is the *reference* library. It's what you can
rely on if you want to publish code that "just works" *everywhere*. By
"publish" here, I don't particularly mean "distribute code as .py
files". I'm thinking much more of StackOverflow answers to "how do I
do X in Python" questions, books and tutorials teaching Python, etc
etc. It solves my problem, even if other distributions also do - and
it has the advantage of being the solution that every other solution
is a superset of, so I can say "this works on Python", and know that
my statement encompasses "this works on Anaconda", "this works on
ActiveState Python", "this works on your distribution's Python", ... -
whereas the converse is *not* true.

In the environment I work in, various groups and environments may get
management buy in (or more often, just management who are willing to
not object) for using Python. But they don't always end up using the
same version (the data science people use Anaconda, the automation
people use the distro Python, the integration guys like me use the
python.org Python, ...), so having that common denominator means we
can still share code.[1]

Steve Dower's explanation of how he sees "splitting up the stdlib"
working strikes me as potentially a good way of removing some of the
maintenance cost of the stdlib *without* losing the "this is stuff you
can expect to be available in every version of Python" aspect of the
stdlib. But I'd want to see a more concrete proposal on how it would
work, and how we could ensure that (for example) we retain the ability
for *every* Python implementation to get data from a URL, even if it's
clunky old urllib and the cool distributions with requests only supply
it "for compatibility", before I'd be completely sold on the idea.

> It's really hard to tell whether specific packages would be good or
> bad additions to the stdlib, when we don't even know what the stdlib
> is supposed to be.

Agreed. But I think you're making things sound worse than they are. We
(collectively) probably *do* know what the stdlib is, even if it's not
easy to articulate. It's what we confidently expect to be present in
any Python environment we sit down at. Much like you'd expect every
Linux distribution to include grep, even though newer tools like ag or
rg may be much better and what you'd prefer to use in practice. And
just like Linux isn't much if all you have is the kernel, so Python is
more than just the core language and builtins.

Paul

[1] Actually, things are currently not that far advanced - the various
groups don't interact at all, yet. So focusing on the stdlib for me is
a way of preparing for the time when we *do* start sharing, and making
that transition relatively effortless and hence an argument in favour
of Python rather than demonstrating that "these special-purpose
languages just create silos".