At this point, I would punt to distutils-sig about curated packages, and pip tooling to support that, but they are bogged down as it stands with just getting warehouse up and running. I don’t support putting specialized tooling in python itself to install curated packages, because that curation would be on the same release schedule as python itself. Pip is an important special case, but it’s a special case to avoid further special cases. If there was excess manpower on the packaging side, that’s where it should be developed. From: Python-ideas [mailto:python-ideas-bounces+tritium-list=sdamon.com@python.org] On Behalf Of Guido van Rossum Sent: Sunday, October 29, 2017 1:17 AM To: Nick Coghlan Cc: python-ideas@python.org Subject: Re: [Python-ideas] Defining an easily installable "Recommended baseline package set" Why? What's wrong with pip install? Why complicate things? Your motivation is really weak here. "beneficial"? "difficult cases"? On Sat, Oct 28, 2017 at 8:57 PM, Nick Coghlan mailto:ncoghlan@gmail.com > wrote: Over on python-dev, the question of recommending MRAB's "regex" module over the standard library's "re" module for more advanced regular expressions recently came up again. Because of various logistical issues and backwards compatibility risks, it's highly unlikely that we'll ever be able to swap out the current _sre based re module implementation in the standard library for an implementation based on the regex module. At the same time, it would be beneficial to have a way to offer an even stronger recommendation to redistributors that we think full-featured general purpose Python scripting environments should offer the regex module as an opt-in alternative to the baseline re feature set, since that would also help with other currently difficult cases like the requests module. What I'm thinking is that we could make some relatively simple additions to the `ensurepip` and `venv` modules to help with this: 1. Add a ensurepip.RECOMMENDED_PACKAGES mapping keyed by standard library module names containing dependency specifiers for recommended third party packages for particular tasks (e.g. "regex" as an enhanced alternative to "re", "requests" as an enhanced HTTPS-centric alternative to "urllib.request") 2. Add a new `install_recommended` boolean flag to ensurepip.bootstrap 3. Add a corresponding `--install-recommended flag to the `python -m ensurepip` CLI 4. Add a corresponding `--install-recommended flag to the `python -m venv` CLI (when combined with `--without-pip`, this would run pip directly from the bundled wheel file to do the installations) We'd also need either a new informational PEP or else a section in the developer guide to say that the contents of `ensurepip.RECOMMENDED_PACKAGES` are up to the individual module maintainers (hence keying the mapping by standard library module name, rather than having a single flat list for the entire standard library). For redistributors with weak dependency support, these reference interpreter level recommendations could become redistributor level recommendations. Redistributors without weak dependency support could still make a distinction between "default" installations (which would include them) and "minimal" installations (which would exclude them). Folks writing scripts and example code for independent distribution (i.e. no explicitly declared dependencies) could then choose between relying on just the standard library (as now), or on the standard library plus independently versioned recommended packages. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com mailto:ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas@python.org mailto:Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido http://python.org/~guido )

On 29 October 2017 at 15:16, Guido van Rossum wrote:

Why? What's wrong with pip install?

At a technical level, this would just be a really thin wrapper around 'pip install' (even thinner than ensurepip in general, since these libraries *wouldn't* be bundled for offline installation, only listed by name).

Why complicate things? Your motivation is really weak here. "beneficial"? "difficult cases"?

The main recurring problems with "pip install" are a lack of discoverability and a potential lack of availability (depending on the environment). This then causes a couple of key undesirable outcomes: - folks using Python as a teaching language have to choose between teaching with just the standard library APIs, requiring that learners restrict themselves to a particular preconfigured learning environment, or make a detour into package management tools in order to ensure learners have access to the APIs they actually want to use (this isn't hypothetical - I was a technical reviewer for a book that justified teaching XML-RPC over HTTPS+JSON on the basis that xmlrpc was in the standard library, and requests wasn't) - folks using Python purely as a scripting language (i.e without app level dependency management) may end up having to restrict themselves to the standard library API, even when there's a well-established frequently preferred alternative for what they're doing (e.g. requests for API management, regex for enhanced regular expressions) The underlying problem is that our reasons for omitting these particular libraries from the standard library relate mainly to publisher side concerns like the logistics of ongoing bug fixing and support, *not* end user concerns like software reliability or API usability. This means that if educators aren't teaching them, or redistributors aren't providing them, then they're actively doing their users a disservice (as opposed to other cases like web frameworks and similar, where there are multiple competing options, you're only going to want one of them in any given application, and the relevant trade-offs between the available options depend greatly on exactly what you're doing) Now, the Python-for-data-science community have taken a particular direction around handling this, and there's an additional library set beyond the standard library that's pretty much taken for granted in a data science context. While conda has been the focal point for those efforts more recently, it started a long time ago with initiatives like Python(x, y) and the Enthought Python Distribution. Similarly, initiatives like Raspberry Pi are able to assume a particular learning environment (Raspbian in the Pi's case), rather than coping with arbitrary starting points. Curated lists like the "awesome-python" one that Stephan linked don't really help that much with the discoverability problem, since they become just another thing for people to learn: How do they find out such lists exist in the first place? Given such a list, how do they determine if the recommendations it offers are actually relevant to their needs? Since assessing a published package API against your needs as a user is a skill that has to be learned like any other, it can be a lot easier to get started in a more prescriptive environment that says "This is what you have to work with for now, we'll explain more about your options for branching out later". The proposal in this thread thus stems from asking the question "Who is going to be best positioned to offer authoritative advice on which third party modules may be preferable to their standard library counterparts for end users of Python?" and answering it with "The standard library module maintainers that are already responsible for deciding whether or not to place appropriate See Also links in the module documentation". All the proposal does is to suggest taking those existing recommendations from the documentation and converting them into a more readibly executable form. I'm not particularly wedded to any particular approach to making the recommendations available in a more machine-friendly form, though - it's just the "offer something more machine friendly than scraping the docs for recommendation links" aspect that I'm interested in. For example, we could skip touching ensurepip or venv at all, and instead limit this to a documentation proposal to collect these recommendations from the documentation, and publish them within the `venv` module docs as a "recommended-libraries.txt" file (using pip's requirements.txt format). That would be sufficient to allow straightforward 3rd party automation, without necessarily committing to providing such automation ourselves. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Perhaps slightly off-topic, but I have sometimes wondered if pip could not be made somewhat friendlier for the absolute newbie and the classroom context. Some concrete proposals. 1. Add a function `pip` to the interactive interpreter (similar to how `help` is available). def pip(args): import sys import subprocess subprocess.check_call([sys.executable, "-m", "pip"] + args.split()) This allows people to install something using pip as long as they have a Python prompt open, and avoids instructors to have to deal with platform-specific instructions for various shells. Also avoids confusion when multiple Python interpreters are available (it operates in the context of the current interpreter.) 2. Add to Pypi package webpages a line like: To install, execute this line in your Python interpreter: pip("install my-package --user") Make this copyable with a button (like Github allows you to copy the repo name). 3. Add the notion of a "channel" to which one can "subscribe". This is actually nothing new, just a Pypi package with no code and just a bunch of dependencies. But it allows me to say: Just enter: pip("install stephans-awesome-stuff --user") in your Python and you get a bunch of useful stuff. Stephan 2017-10-29 8:54 GMT+01:00 Nick Coghlan :

On 29 October 2017 at 15:16, Guido van Rossum wrote:

...

Why? What's wrong with pip install?

At a technical level, this would just be a really thin wrapper around 'pip install' (even thinner than ensurepip in general, since these libraries *wouldn't* be bundled for offline installation, only listed by name).

...

Why complicate things? Your motivation is really weak here. "beneficial"? "difficult cases"?

The main recurring problems with "pip install" are a lack of discoverability and a potential lack of availability (depending on the environment).

This then causes a couple of key undesirable outcomes:

- folks using Python as a teaching language have to choose between teaching with just the standard library APIs, requiring that learners restrict themselves to a particular preconfigured learning environment, or make a detour into package management tools in order to ensure learners have access to the APIs they actually want to use (this isn't hypothetical - I was a technical reviewer for a book that justified teaching XML-RPC over HTTPS+JSON on the basis that xmlrpc was in the standard library, and requests wasn't) - folks using Python purely as a scripting language (i.e without app level dependency management) may end up having to restrict themselves to the standard library API, even when there's a well-established frequently preferred alternative for what they're doing (e.g. requests for API management, regex for enhanced regular expressions)

The underlying problem is that our reasons for omitting these particular libraries from the standard library relate mainly to publisher side concerns like the logistics of ongoing bug fixing and support, *not* end user concerns like software reliability or API usability. This means that if educators aren't teaching them, or redistributors aren't providing them, then they're actively doing their users a disservice (as opposed to other cases like web frameworks and similar, where there are multiple competing options, you're only going to want one of them in any given application, and the relevant trade-offs between the available options depend greatly on exactly what you're doing)

Now, the Python-for-data-science community have taken a particular direction around handling this, and there's an additional library set beyond the standard library that's pretty much taken for granted in a data science context. While conda has been the focal point for those efforts more recently, it started a long time ago with initiatives like Python(x, y) and the Enthought Python Distribution.

Similarly, initiatives like Raspberry Pi are able to assume a particular learning environment (Raspbian in the Pi's case), rather than coping with arbitrary starting points.

Curated lists like the "awesome-python" one that Stephan linked don't really help that much with the discoverability problem, since they become just another thing for people to learn: How do they find out such lists exist in the first place? Given such a list, how do they determine if the recommendations it offers are actually relevant to their needs? Since assessing a published package API against your needs as a user is a skill that has to be learned like any other, it can be a lot easier to get started in a more prescriptive environment that says "This is what you have to work with for now, we'll explain more about your options for branching out later".

The proposal in this thread thus stems from asking the question "Who is going to be best positioned to offer authoritative advice on which third party modules may be preferable to their standard library counterparts for end users of Python?" and answering it with "The standard library module maintainers that are already responsible for deciding whether or not to place appropriate See Also links in the module documentation".

All the proposal does is to suggest taking those existing recommendations from the documentation and converting them into a more readibly executable form.

I'm not particularly wedded to any particular approach to making the recommendations available in a more machine-friendly form, though - it's just the "offer something more machine friendly than scraping the docs for recommendation links" aspect that I'm interested in. For example, we could skip touching ensurepip or venv at all, and instead limit this to a documentation proposal to collect these recommendations from the documentation, and publish them within the `venv` module docs as a "recommended-libraries.txt" file (using pip's requirements.txt format). That would be sufficient to allow straightforward 3rd party automation, without necessarily committing to providing such automation ourselves.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 29 October 2017 at 09:51, Antoine Pitrou wrote:

On Sun, 29 Oct 2017 17:54:22 +1000 Nick Coghlan wrote:

...

This means that if educators aren't teaching them, or redistributors aren't providing them, then they're actively doing their users a disservice

Which redistributors do not provide the requests library, for example? regex is probably not as popular (mostly because re is good enough for most purposes), but it still appears to be available from Ubuntu and Anaconda.

I know it's not what you meant, but "the python.org installers" is the obvious answer here. On Windows, if you say to someone "install Python", they get the python.org distribution. Explicitly directing them to Anaconda is an option, but that gives them a distinctly different experience than "standard Python plus some best of breed packages like requests" does.

...

All the proposal does is to suggest taking those existing recommendations from the documentation and converting them into a more readibly executable form.

I'm curious what such a list looks like :-)

I am also. I'd put requests on it immediately, but that's the only thing I consider obvious. regex is what triggered this, but I'm not sure it adds *that* much - it's a trade off between people who need the extra features and people confused over why we have two regex libraries available. After that, you're almost immediately into domain-specific answers, and it becomes tricky fast. Paul

On Sunday, October 29, 2017, Paul Moore wrote:

On 29 October 2017 at 10:40, Stephan Houben javascript:;> wrote:

...

Perhaps slightly off-topic, but I have sometimes wondered if pip could not be made somewhat friendlier for the absolute newbie and the classroom context.

Some concrete proposals.

1. Add a function `pip` to the interactive interpreter (similar to how `help` is available).

def pip(args): import sys import subprocess subprocess.check_call([sys.executable, "-m", "pip"] + args.split())

This allows people to install something using pip as long as they have a Python prompt open, and avoids instructors to have to deal with platform-specific instructions for various shells. Also avoids confusion when multiple Python interpreters are available (it operates in the context of the current interpreter.)

There are subtle issues around whether newly installed/upgraded packages are visible in a running Python interpreter.

If the package is already imported, reload() isn't sufficient; but deepreload from IPython may be. IPython also supports shell commands prefixed with '!': ! pip install -U requests regex IPython ! pip install -U --user psfblessedset1 %run pip install -U --user psfblessedset1 %run? % autoreload? # http://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html This:

It's possible that this would result in *more* confusion than the current situation.

Why isn't it upgrading over top of the preinstalled set? At least with bash, the shell command history is logged to .bash_history by default. IMO, it's easier to assume that environment-modifying commands are logged in the shell log; and that running a ``%logstart -o logged-cmds.py`` won't change the versions of packages it builds upon. Is it bad form to put ``! pip install -U requests `` at the top of every jupyter notebook? ``! pip install requests==version `` is definitely more reproducible. (seeAlso: binder, jupyter/docker-stacks) I'll just include these here unnecessarily from the other curated package set discussions: https://github.com/jupyter/docker-stacks https://github.com/Kaggle/docker-python/blob/master/Dockerfile https://binderhub.readthedocs.io/en/latest/ I can see the appeal of something like this, but it's not as simple as

it looks. If you want to discuss this further, I'd definitely suggest making it a thread of its own.

Personally, as a pip maintainer, I'm -0 on this (possibly even -1).

Same.

On 2017-10-29 00:54, Nick Coghlan wrote:

The proposal in this thread thus stems from asking the question "Who is going to be best positioned to offer authoritative advice on which third party modules may be preferable to their standard library counterparts for end users of Python?" and answering it with "The standard library module maintainers that are already responsible for deciding whether or not to place appropriate See Also links in the module documentation".

All the proposal does is to suggest taking those existing recommendations from the documentation and converting them into a more readibly executable form.

So it sounds like you're just saying that there should be one more "awesome-python" style list, except it should be official, with the seal of approval of Python itself. The details of exactly how that would displayed to users are not so important as that it be easily accessible from python.org and clearly shown to be endorsed by the Python developers. I think that is a good idea. The current situation is quite confusing in many cases. You see a lot of questions on StackOverflow where someone is tearing their hair out trying to use urllib to do heavy lifting and the answer is "use requests instead". Likewise people trying to implement matrix multiplication who should probably be using numpy, As for regex, to be honest, I have yet to find any way in which it is not completely superior to the existing re library, and I find it somewhat frustrating that the "publisher-side" concerns you mention continue to hold it back from replacing re. The only problem I see with this sort of "Python seal of approval" library list is that it carries some of the same risks as incorporating such libraries into the stdlib. Not all, but some. For one thing, if people go to python.org and see a thing that says "You may also want to use these libraries that bear the Python Seal of Approval. . .", and then they download them, and something goes wrong (e.g., there is some kind of version conflict with another package), they will likely blame python.org (at least partially). Only now, since the libraries aren't in the stdlib, the Python devs won't really be able to do anything to fix that; all they could do is remove the offending package from the approved list. In practice I think this is unlikely to happen, since the idea would be that Python devs would be judicious in awarding the seal of approval only to projects that are robust and not prone to breakage. But it does change the nature of the approval from "we approve this *code* and we will fix it if it breaks" (which is how the existing stdlib works) to "we approve these *people* (the people working on requests or regex or whatever) and we will cease to do if they break their code". -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

The two use cases you describe (scripters and teachers) leave me luke-warm -- scripters live in the wild west and can just pip install whatever (that's what it means to be scripting) and teachers tend to want a customized bundle anyway -- let the edu world get together and create their own recommended bundle. As long as it's not going to be bundled, i.e. there's just going to be some list of packages that we recommend to 3rd party repackagers, then I'm fine with it. But they must remain clearly marked as 3rd party packages in whatever docs we provide, and live in site-packages. I would really like to see what you'd add to the list besides requests -- I really don't see why the teaching use case would need the regex module (unless it's a class in regular expressions). --Guido On Sun, Oct 29, 2017 at 12:54 AM, Nick Coghlan wrote:

On 29 October 2017 at 15:16, Guido van Rossum wrote:

...

Why? What's wrong with pip install?

At a technical level, this would just be a really thin wrapper around 'pip install' (even thinner than ensurepip in general, since these libraries *wouldn't* be bundled for offline installation, only listed by name).

...

Why complicate things? Your motivation is really weak here. "beneficial"? "difficult cases"?

The main recurring problems with "pip install" are a lack of discoverability and a potential lack of availability (depending on the environment).

This then causes a couple of key undesirable outcomes:

- folks using Python as a teaching language have to choose between teaching with just the standard library APIs, requiring that learners restrict themselves to a particular preconfigured learning environment, or make a detour into package management tools in order to ensure learners have access to the APIs they actually want to use (this isn't hypothetical - I was a technical reviewer for a book that justified teaching XML-RPC over HTTPS+JSON on the basis that xmlrpc was in the standard library, and requests wasn't) - folks using Python purely as a scripting language (i.e without app level dependency management) may end up having to restrict themselves to the standard library API, even when there's a well-established frequently preferred alternative for what they're doing (e.g. requests for API management, regex for enhanced regular expressions)

The underlying problem is that our reasons for omitting these particular libraries from the standard library relate mainly to publisher side concerns like the logistics of ongoing bug fixing and support, *not* end user concerns like software reliability or API usability. This means that if educators aren't teaching them, or redistributors aren't providing them, then they're actively doing their users a disservice (as opposed to other cases like web frameworks and similar, where there are multiple competing options, you're only going to want one of them in any given application, and the relevant trade-offs between the available options depend greatly on exactly what you're doing)

Now, the Python-for-data-science community have taken a particular direction around handling this, and there's an additional library set beyond the standard library that's pretty much taken for granted in a data science context. While conda has been the focal point for those efforts more recently, it started a long time ago with initiatives like Python(x, y) and the Enthought Python Distribution.

Similarly, initiatives like Raspberry Pi are able to assume a particular learning environment (Raspbian in the Pi's case), rather than coping with arbitrary starting points.

Curated lists like the "awesome-python" one that Stephan linked don't really help that much with the discoverability problem, since they become just another thing for people to learn: How do they find out such lists exist in the first place? Given such a list, how do they determine if the recommendations it offers are actually relevant to their needs? Since assessing a published package API against your needs as a user is a skill that has to be learned like any other, it can be a lot easier to get started in a more prescriptive environment that says "This is what you have to work with for now, we'll explain more about your options for branching out later".

The proposal in this thread thus stems from asking the question "Who is going to be best positioned to offer authoritative advice on which third party modules may be preferable to their standard library counterparts for end users of Python?" and answering it with "The standard library module maintainers that are already responsible for deciding whether or not to place appropriate See Also links in the module documentation".

All the proposal does is to suggest taking those existing recommendations from the documentation and converting them into a more readibly executable form.

I'm not particularly wedded to any particular approach to making the recommendations available in a more machine-friendly form, though - it's just the "offer something more machine friendly than scraping the docs for recommendation links" aspect that I'm interested in. For example, we could skip touching ensurepip or venv at all, and instead limit this to a documentation proposal to collect these recommendations from the documentation, and publish them within the `venv` module docs as a "recommended-libraries.txt" file (using pip's requirements.txt format). That would be sufficient to allow straightforward 3rd party automation, without necessarily committing to providing such automation ourselves.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-- --Guido van Rossum (python.org/~guido)

On 29 October 2017 at 18:56, Guido van Rossum wrote:

The two use cases you describe (scripters and teachers) leave me luke-warm -- scripters live in the wild west and can just pip install whatever (that's what it means to be scripting)

In my experience, "scripting" *does* include people for whom "being in a standard Python distribution" is a requirement. Situations I have encountered: 1. Servers where developers need to write administrative or monitoring scripts, but they don't control what's on that server. The Python installation that comes by default is all that's available. 2. Developers working in environments with limited internet access. For example, my employer has a Windows (NTLM) proxy that pip can't work with. Getting internet access for pip involves installing a separate application, and that's not always possible/desirable. 3. Developers writing scripts to be shared with non-developers. On Unix, this means "must work with the system Python", and on Windows "download and install Python from python.org" is typically all you can expect (although in that case "install Anaconda" is a possible alternative, although not one I've tried telling people to do myself). Nick's proposal doesn't actually help for (1) or (2), as the problem there is that "pip install" won't work. And bundling a script with its (pure Python) dependencies, for example as a zipapp, is always a solution - although it's nowhere near as easy as simply copying a single-file script to the destination where it's to be run. So these situations don't actually matter in terms of the value of the proposals being discussed here. But I did want to dispute the idea that "scripters can just pip install whatever" is inherent to the idea of being a scripter - my experience is the opposite, that scripters are the people *least* able to simply pip install things. Paul

On 29 October 2017 at 20:44, Alex Walters wrote:

Writing scripts for non-developers, in an unmanaged environment (IT cant push a python install to the system) on windows means running pyinstaller et. al., on your script, if it has dependencies or not. Its not worth it to walk someone through a python install to run a script, let alone installing optional dependencies.

Let's just say "not in the environments I work in", and leave it at that.

Not to sound too crass, but there are only so many edge cases that a third party platform developer can be expected to care about (enough to bend over backwards to support).

I never suggested otherwise. I just wanted to point out that "scripting in Python" covers a wider range of use cases than "can use pip install". I'm not asking python-dev to support those use cases (heck, I'm part of python-dev and *I* don't want to bend over backwards to support them), but I do ask that people be careful not to dismiss a group of users who are commonly under-represented in open source mailing lists and communities. Paul

On 30 October 2017 at 04:56, Guido van Rossum wrote:

The two use cases you describe (scripters and teachers) leave me luke-warm -- scripters live in the wild west and can just pip install whatever (that's what it means to be scripting)

For personal scripting, we can install whatever, but "institutional scripting" isn't the same thing - there we're scripting predefined "Standard Operating Environments", like those Mahmoud Hashemi describes for PayPal at the start of https://www.paypal-engineering.com/2016/09/07/python-packaging-at-paypal/ "Just use PyInstaller" isn't an adequate answer in large-scale environments because of the "zlib problem": you don't want to have to rebuild and redeploy the world to handle a security update in a low level frequently used component, you want to just update that component and have everything else pick it up dynamically. While "Just use conda" is excellent advice nowadays for any organisation contemplating defining their own bespoke Python SOE (hence Mahmoud's post), if that isn't being driven by the folks that already maintain the SOE (as happened in PayPal's case) convincing an org to add a handful of python-dev endorsed libraries to an established SOE is going to be easier than rebasing their entire Python SOE on conda.

and teachers tend to want a customized bundle anyway -- let the edu world get together and create their own recommended bundle.

As long as it's not going to be bundled, i.e. there's just going to be some list of packages that we recommend to 3rd party repackagers, then I'm fine with it. But they must remain clearly marked as 3rd party packages in whatever docs we provide, and live in site-packages.

Yep, that was my intent, although it may not have been clear in my initial proposal. I've filed two separate RFEs in relation to that: * Documentation only: https://bugs.python.org/issue31898 * Regression testing resource: https://bugs.python.org/issue31899 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 31 October 2017 at 00:28, Guido van Rossum wrote:

I just feel that when you're talking about an org like PayPal they can take care of themselves and don't need our help. They will likely have packages they want installed everywhere that would never make in on your list. So it feels this whole discussion is a distraction and a waste of time (yours, too).

Just because companies are big doesn't mean they necessarily have anyone internally that's already up to speed on the specifics of recommended practices in a sprawling open source community like Python's. The genesis of this idea is that I think we can offer a more consistent initial experience for those folks than "Here's PyPI and Google, y'all have fun now" (and in so doing, help folks writing books and online tutorials to feel more comfortable with the idea of assuming that libraries like requests will be available in even the most restrictive institutional environments that still allow the use of Python). One specific situation this idea is designed to help with is the one where: - there's a centrally managed Standard Operating Environment that dictates what gets installed - they've approved the python.org installers - they *haven't* approved anything else yet Now, a lot of large orgs simply won't get into that situation in the first place, since their own supplier management rules will push them towards a commercial redistributor, in which case they'll get their chosen redistributor's preferred package set, which will then typically cover at least a few hundred of the most popular PyPI packages. But some of them will start from the narrower "standard library only" baseline, and I spent enough time back at Boeing arguing for libraries to be added to our approved component list to appreciate the benefits of transitive declarations of trust ("we trust supplier X, they unambigously state their trust in supplier Y, so that's an additional point in favour of our also trusting supplier Y") when it comes time to make your case to your supplier management organisation. Such declarations still aren'y always sufficient, but they definitely don't hurt, and they sometimes help. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Oct 30, 2017 at 12:29 PM, Guido van Rossum wrote:

What's your proposed process to arrive at the list of recommended packages? And is it really just going to be a list of names, or is there going to be some documentation (about the vetting, not about the contents of the packages) for each name?

As I see it, the bootstrap would be a requirements.txt (requirements.pip) with the packages that more experienced developers prefer to use over stdlib, pinned to package versions known to work well with the CPython release. I think this is a great idea, specially if it's an easy opt-in. For example, sometimes I go into a programming niche for a few years, and I'd very much appreciate a curated list of packages when I move to another niche. A pay-forward kind of thing. The downside to the idea is that the list of packages will need a curator (which could be python.org).

On 31 October 2017 at 02:29, Guido van Rossum wrote:

What's your proposed process to arrive at the list of recommended packages?

I'm thinking it makes the most sense to treat inclusion in the recommended packages list as a possible outcome of proposals for standard library inclusion, rather than being something we'd provide a way to propose specifically. We'd only use it in cases where a proposal would otherwise meet the criteria for stdlib inclusion, but the logistics of actually doing so don't work for some reason. Running the initial 5 proposals through that filter: * six: a cross-version compatibility layer clearly needs to be outside the standard library * setuptools: we want to update this in line with the PyPA interop specs, not the Python language version * cffi: updates may be needed for PyPA interop specs, Python implementation updates or C language definition updates * requests: updates are more likely to be driven by changes in network protocols and client platform APIs than Python language changes * regex: we don't want two regex engines in the stdlib, transparently replacing _sre would be difficult, and _sre is still good enough for most purposes Of the 5, I'd suggest that regex is the only one that could potentially still make its way into the standard library some day - it would just require someone with both the time and inclination to create a CPython variant that used _regex instead of _sre as the default regex engine, and then gathered evidence to show that it was "compatible enough" with _sre to serve as the default engine for CPython. For the first four, there are compelling arguments that their drivers for new feature additions are such that their release cycles shouldn't ever be tied to the rate at which we update the Python language definition.

And is it really just going to be a list of names, or is there going to be some documentation (about the vetting, not about the contents of the packages) for each name?

I'm thinking a new subsection in https://docs.python.org/devguide/stdlibchanges.html for "Recommended Third Party Packages" would make sense, covering what I wrote above. It also occurred to me that since the recommendations are independent of the Python version, they don't really belong in the version specific documentation. While the Developer's Guide isn't really the right place for the list either (except as an easier way to answer "Why isn't <X> in the standard library?" questions), it could be a good interim option until I get around to actually writing a first draft of https://github.com/python/redistributor-guide/ (which I was talking to Barry about at the dev sprint, but didn't end up actually creating any content for since I went down a signal handling rabbit hole instead). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 1 November 2017 at 00:53, Guido van Rossum wrote:

On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan wrote:

...

On 31 October 2017 at 02:29, Guido van Rossum wrote:

...

What's your proposed process to arrive at the list of recommended packages?

I'm thinking it makes the most sense to treat inclusion in the recommended packages list as a possible outcome of proposals for standard library inclusion, rather than being something we'd provide a way to propose specifically.

I don't think that gets you off the hook for a process proposal. We need some criteria to explain why a module should be on the recommended list -- not just a ruling as to why it shouldn't be in the stdlib.

The developer guide already has couple of sections on this aspect: * https://devguide.python.org/stdlibchanges/#acceptable-types-of-modules * https://devguide.python.org/stdlibchanges/#requirements I don't think either of those sections is actually quite right (since we've approved new modules that wouldn't meet them), but they're not terrible as a starting point in general, and they're accurate for the recommended packages use case.

We'd only use it in cases where a proposal would otherwise meet the

...

criteria for stdlib inclusion, but the logistics of actually doing so don't work for some reason.

But that would exclude most of the modules you mention below, since one of the criteria is that their development speed be matched with Python's release cycle. I think there must be some form of "popularity" combined with "best of breed". In particular I'd like to have a rule that explains why flask and Django would never make the list. (I don't know what that rule is, or I would tell you -- my gut tells me it's something to do with having their own community *and* competing for the same spot.)

The developer guide words this as "The module needs to be considered best-of-breed.". In some categories (like WSGI frameworks), there are inherent trade-offs that mean there will *never* be a single best-of-breed solution, since projects like Django, Flask, and Pyramid occupy deliberately different points in the space of available design decisions. Running the initial 5 proposals through that filter:

...

* six: a cross-version compatibility layer clearly needs to be outside the standard library

Hm... Does six still change regularly? If not I think it *would* be a candidate for actual stdlib inclusion. Just like we added u"..." literals to Python 3.4.

It still changes as folks porting new projects discover additional discrepancies between the 2.x and 3.x standard library layouts and behaviour (e.g. we found recently that the 3.x subprocess module's emulation of the old commands module APIs actually bit shifts the status codes relative to the 2.7 versions). The rate of change has definitely slowed down a lot since the early days, but it isn't zero. In addition, the only folks that need it are those that already care about older versions of Python - if you can start with whatever the latest version of Python is, and don't have any reason to support users still running older version, you can happily pretend those older versions don't exist, and hence don't need a compatibility library like six. As a separate library, it can just gracefully fade away as folks stop depending on it as they drop Python 2.7 support. By contrast, if we were to bring it into the 3.x standard library, then we'd eventually have to figure out when we could deprecate and remove it again.

* setuptools: we want to update this in line with the PyPA interop specs,

...

not the Python language version

But does that exclude stdlib inclusion? Why would those specs change, and why couldn't they wait for a new Python release?

The specs mainly change when we want to offer publishers new capabilities while still maintaining compatibility with older installation clients (and vice-versa: we want folks still running Python 2.7 to be able to publish wheel files and use recently added metadata fields like Description-Content-Type). The reason we can't wait for new Python releases is because when we add such things, we need them to work on *all* supported Python releases (including 2.7 and security-release-only 3.x versions). There are also other drivers for setuptools updates, which include: - operating system build toolchain changes (e.g. finding new versions of Visual Studio or XCode) - changes to PyPI's operations (e.g. the legacy upload API getting turned off due to persistent service stability problems, switching to HTTPS only access) With setuptools as a separate project, a whole lot of package publication problems can be solved via "pip install --upgrade setuptools wheel" in a virtual environment, which is a luxury we don't have with plain distutils.

* cffi: updates may be needed for PyPA interop specs, Python implementation updates or C language definition updates

Hm, again, I don't recall that this was debated -- I think it's a failure that it's not in the stdlib.

A couple of years ago I would have agreed with you, but I've spent enough time on packaging problems now to realise that cffi actually qualifies as a build tool due to the way it generates extension module wrappers when used in "out-of-line" mode. Being outside the standard library means that cffi still has significant flexibility to evolve how it separates its buildtime functionality from its runtime functionality, and may eventually be adjusted so that only "_cffi_backend" needs to be installed at runtime for the out-of-line compilation mode, without the full C header parsing and inline extension module compilation capabilities of CFFI itself (see https://cffi.readthedocs.io/en/latest/cdef.html#preparing-and-distributing-m... for the details). Being separate also means that cffi can be updated to generate more efficient code even for existing Python versions.

* requests: updates are more likely to be driven by changes in network

...

protocols and client platform APIs than Python language changes

Here I agree. There's no alternative (except aiohttp, but that's asyncio-based) and it can't be in the stdlib because it's actively being developed.

...

* regex: we don't want two regex engines in the stdlib, transparently replacing _sre would be difficult, and _sre is still good enough for most purposes

I think this needn't be recommended at all. For 99.9% of regular expression uses, re is just fine. Let's just work on a strategy for introducing regex into the stdlib.

Given your informational PEP suggestion below, I'd probably still include it, but in a separate section from the others (e.g. the others might be listed as "Misaligned Feature Release Cycles", which is an inherent logistical problem that no amount of coding can fix, while regex would instead be categorised as "Technical Challenges").

Of the 5, I'd suggest that regex is the only one that could potentially

...

still make its way into the standard library some day - it would just require someone with both the time and inclination to create a CPython variant that used _regex instead of _sre as the default regex engine, and then gathered evidence to show that it was "compatible enough" with _sre to serve as the default engine for CPython.

For the first four, there are compelling arguments that their drivers for new feature additions are such that their release cycles shouldn't ever be tied to the rate at which we update the Python language definition.

As you can tell from my arguing, the reasons need to be written up in more detail.

...

And is it really just going to be a list of names, or is there going to

...

be some documentation (about the vetting, not about the contents of the packages) for each name?

I'm thinking a new subsection in https://docs.python.org/devgui de/stdlibchanges.html for "Recommended Third Party Packages" would make sense, covering what I wrote above.

That's too well hidden for my taste.

...

It also occurred to me that since the recommendations are independent of the Python version, they don't really belong in the version specific documentation.

But that doesn't mean they can't (also) be listed there. (And each probably has its version dependencies.)

...

While the Developer's Guide isn't really the right place for the list either (except as an easier way to answer "Why isn't <X> in the standard library?" questions), it could be a good interim option until I get around to actually writing a first draft of https://github.com/python/redi stributor-guide/ (which I was talking to Barry about at the dev sprint, but didn't end up actually creating any content for since I went down a signal handling rabbit hole instead).

Hm, let's not put more arbitrary check boxes in the way of progress. Maybe it can be an informational PEP that's occasionally updated?

If I'm correctly reading that as "Could the list of Recommended Third Party Packages be an informational PEP?", then I agree that's probably a good way to tackle it, since it will cover both the developer-centric "Why isn't this in the standard library yet?" aspect *and* the "Redistributors should probably provide this" aspect. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tuesday, October 31, 2017, Guido van Rossum wrote:

On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan javascript:_e(%7B%7D,'cvml','ncoghlan@gmail.com');> wrote:

...

On 31 October 2017 at 02:29, Guido van Rossum javascript:_e(%7B%7D,'cvml','guido@python.org');> wrote:

...

What's your proposed process to arrive at the list of recommended packages?

I'm thinking it makes the most sense to treat inclusion in the recommended packages list as a possible outcome of proposals for standard library inclusion, rather than being something we'd provide a way to propose specifically.

I don't think that gets you off the hook for a process proposal. We need some criteria to explain why a module should be on the recommended list -- not just a ruling as to why it shouldn't be in the stdlib.

...

We'd only use it in cases where a proposal would otherwise meet the criteria for stdlib inclusion, but the logistics of actually doing so don't work for some reason.

But that would exclude most of the modules you mention below, since one of the criteria is that their development speed be matched with Python's release cycle. I think there must be some form of "popularity" combined with "best of breed". In particular I'd like to have a rule that explains why flask and Django would never make the list. (I don't know what that rule is, or I would tell you -- my gut tells me it's something to do with having their own community *and* competing for the same spot.)

Running the initial 5 proposals through that filter:

...

* six: a cross-version compatibility layer clearly needs to be outside the standard library

Hm... Does six still change regularly? If not I think it *would* be a candidate for actual stdlib inclusion. Just like we added u"..." literals to Python 3.4.

...

* setuptools: we want to update this in line with the PyPA interop specs, not the Python language version

But does that exclude stdlib inclusion? Why would those specs change, and why couldn't they wait for a new Python release?

...

* cffi: updates may be needed for PyPA interop specs, Python implementation updates or C language definition updates

Hm, again, I don't recall that this was debated -- I think it's a failure that it's not in the stdlib.

...

* requests: updates are more likely to be driven by changes in network protocols and client platform APIs than Python language changes

Here I agree. There's no alternative (except aiohttp, but that's asyncio-based) and it can't be in the stdlib because it's actively being developed.

What about certifi (SSL bundles (from requests (?)) on PyPi) https://pypi.org/project/certifi/ ?

...

* regex: we don't want two regex engines in the stdlib, transparently replacing _sre would be difficult, and _sre is still good enough for most purposes

I think this needn't be recommended at all. For 99.9% of regular expression uses, re is just fine. Let's just work on a strategy for introducing regex into the stdlib.

...

Of the 5, I'd suggest that regex is the only one that could potentially still make its way into the standard library some day - it would just require someone with both the time and inclination to create a CPython variant that used _regex instead of _sre as the default regex engine, and then gathered evidence to show that it was "compatible enough" with _sre to serve as the default engine for CPython.

For the first four, there are compelling arguments that their drivers for new feature additions are such that their release cycles shouldn't ever be tied to the rate at which we update the Python language definition.

As you can tell from my arguing, the reasons need to be written up in more detail.

...

And is it really just going to be a list of names, or is there going to

...

be some documentation (about the vetting, not about the contents of the packages) for each name?

I'm thinking a new subsection in https://docs.python.org/devgui de/stdlibchanges.html for "Recommended Third Party Packages" would make sense, covering what I wrote above.

That's too well hidden for my taste.

...

It also occurred to me that since the recommendations are independent of the Python version, they don't really belong in the version specific documentation.

But that doesn't mean they can't (also) be listed there. (And each probably has its version dependencies.)

...

While the Developer's Guide isn't really the right place for the list either (except as an easier way to answer "Why isn't <X> in the standard library?" questions), it could be a good interim option until I get around to actually writing a first draft of https://github.com/python/redi stributor-guide/ (which I was talking to Barry about at the dev sprint, but didn't end up actually creating any content for since I went down a signal handling rabbit hole instead).

Hm, let's not put more arbitrary check boxes in the way of progress. Maybe it can be an informational PEP that's occasionally updated?

-- --Guido van Rossum (python.org/~guido)

On Tue, Oct 31, 2017 at 11:41 AM, MRAB wrote:

regex gets updated when the Unicode Consortium releases an update.

Is it a feature that that is more frequently than Python releases? There are other things in Python that must be updated whenever the UC releases an update, and they get treated as features (or perhaps as bugfixes, I'm not sure) but this means they generally don't get backported. -- --Guido van Rossum (python.org/~guido)

On Tue, Oct 31, 2017 at 12:24 PM, MRAB wrote:

On 2017-10-31 18:44, Guido van Rossum wrote:

...

On Tue, Oct 31, 2017 at 11:41 AM, MRAB mailto:python@mrabarnett.plus.com> wrote:

regex gets updated when the Unicode Consortium releases an update.

Is it a feature that that is more frequently than Python releases? There are other things in Python that must be updated whenever the UC releases an update, and they get treated as features (or perhaps as bugfixes, I'm not sure) but this means they generally don't get backported.

Here's a list of the updates to Unicode:

https://www.unicode.org/versions/enumeratedversions.html

Roughly yearly.

Those still on Python 2.7, for example, are now 8 years behind re Unicode.

Frankly I consider that a feature of the 2.7 end-of-life planning. :-)

At least Python 3.6 is only 1 year/release behind, which is fine!

OK, so presumably that argument doesn't preclude inclusion in the 3.7 (or later) stdlib. I'm beginning to warm up to the idea again... Maybe we should just bite the bullet. Nick, what do you think? Is it worth a small PEP? -- --Guido van Rossum (python.org/~guido)

On Oct 31, 2017 4:42 AM, "Nick Coghlan" wrote: On 31 October 2017 at 02:29, Guido van Rossum wrote:

What's your proposed process to arrive at the list of recommended packages?

I'm thinking it makes the most sense to treat inclusion in the recommended packages list as a possible outcome of proposals for standard library inclusion, rather than being something we'd provide a way to propose specifically. We'd only use it in cases where a proposal would otherwise meet the criteria for stdlib inclusion, but the logistics of actually doing so don't work for some reason. Running the initial 5 proposals through that filter: * six: a cross-version compatibility layer clearly needs to be outside the standard library * setuptools: we want to update this in line with the PyPA interop specs, not the Python language version * cffi: updates may be needed for PyPA interop specs, Python implementation updates or C language definition updates * requests: updates are more likely to be driven by changes in network protocols and client platform APIs than Python language changes * regex: we don't want two regex engines in the stdlib, transparently replacing _sre would be difficult, and _sre is still good enough for most purposes Some other packages that might meet these criteria, or at least be useful for honing them: - lxml - numpy - cryptography - idna -n

On Wed, Nov 1, 2017 at 7:41 AM, Guido van Rossum wrote:

Can you write 1-2 paragraphs with the argument for each?

On Tue, Oct 31, 2017 at 10:01 PM, Nathaniel Smith wrote:

...

- lxml

My impression (probably others are more knowledgeable) is that lxml has more or less replaced the stdlib 'xml' package as the de facto standard -- sort of similar to the urllib2/requests situation. AFAIK lxml has never been proposed for stdlib inclusion and I believe the fact that it's all in Cython would be a barrier even if the maintainers were amenable. But it might be helpful to our users to put a box at the top of the 'xml' docs suggesting people check out 'lxml', similar to the one on the urllib2 docs.

...

- numpy

Numpy's arrays are a foundational data structure and de facto standard, and would probably fit naturally in the stdlib semantically, but for a number of logistical/implementational reasons it doesn't make sense to merge. Probably doesn't make much difference whether python-dev "blesses" it or not in practice, since there aren't any real competitors inside or outside the stdlib; it'd more just be an acknowledgement of the status quo.

...

- cryptography

Conceptually, core cryptographic operations are the kind of functionality that you might expect to see in the stdlib, but the unique sensitivity of crypto code makes this a bad idea. Historically there have been a variety of basic crypto packages for Python, but at this point IIUC the other ones are all considered obsolete-and-potentially-dangerous and the consensus is everyone should move to 'cryptography', so documenting that in this PEP might help send people in the right direction.

...

- idna

This is a bit of a funny one. IDNA functionality is pretty fundamental -- you need it to do unicode<->bytes conversions on hostnames, so basically anyone doing networking needs it. Python ships with some built-in IDNA functionality (as the "idna" codec), but it's using an obsolete standard (IDNA2003, versus the current IDNA2008, see bpo-17305), and IIRC Christian thinks the whole codec-based design is the wrong approach... basically what we have in the stdlib has been broken for most of a decade and there doesn't seem to be anyone interested in fixing it. So... in the long run the stdlib support should either be fixed or deprecated. I'm not sure which is better. (The argument for deprecating it would be that IIUC you need to update the tables whenever a new unicode standard comes out, and since it's a networking thing you want to keep in sync with the rest of the world, which is easier with a standalone library. I don't know how much this matters in practice.) But right now, this library is just better than the stdlib functionality, and it wouldn't hurt to document that. -n -- Nathaniel J. Smith -- https://vorpus.org