On Sun, Feb 6, 2022 at 3:48 PM Damian Shaw <damian.peter.shaw@gmail.com> wrote:

Pip vendors requests for network calls: https://github.com/pypa/pip/tree/main/src/pip/_vendor/requests

But still does depend on functions from urllib.parse and urllib.request in many places: https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/urls.py

Aha, it doesn't use urllib.request to open a HTTP connection, it only uses pathname2url() and url2pathname() functions of urllib.request. Maybe we can keep these functions. I'm not sure why they don't belong to urllib.parse. If urllib.parse is widely used, maybe we can keep this module. Victor

El dom, 6 feb 2022 a las 9:12, Paul Moore (<p.f.moore@gmail.com>) escribió:

On Sun, 6 Feb 2022 at 16:51, Christian Heimes <christian@python.org> wrote:

...

The urllib package -- and to some degree also the http package -- are constant source of security bugs. The code is old and the parsers for HTTP and URLs don't handle edge cases well. Python core lacks a true maintainer of the code. To be honest, we have to admit defeat and be up front that urllib is not up to the task for this decade. It was designed written during a more friendly, less scary time on the internet.

If I had the power and time, then I would replace urllib with a simpler, reduced HTTP client that uses platform's HTTP library under the hood (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten, maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or aiohttp are much better suited than urllib.

The second best option is to reduce the feature set of urllib to core HTTP (no ftp, proxy, HTTP auth) and a partial rewrite with stricter, more standard conform parsers for urls, query strings, and RFC 2822 instead of RFC 822 for headers.

I'd likely be fine with either of these two options. I'm not worried about supporting "advanced" uses. But having no way of getting a file from the internet without relying on 3rd party packages seems like a huge gap in functionality for a modern language. And having to use a 3rd party library to parse URLs will simply push more people to use home-grown regexes rather than something safe and correct. Remember that a lot of Python users are not professional software developers, but scientists, data analysts, and occasional users, for whom the existence of something in the stdlib is the *only* reason they have any idea that URLs need specialised parsing in the first place.

And while we all like to say 3rd party modules are great, the reality is that they provide a genuine problem for many of these non-specialist users - and I say that as a packaging specialist and pip maintainer. The packaging ecosystem is *not* newcomer-friendly in the way that core Python is, much as we're trying to improve that situation.

I've said it previously, but I'll reiterate - IMO this *must* have a PEP, and that PEP must be clear that the intention is to *remove* urllib, not simply to "deprecate and then think about it". That could be making it part of PEP 594, or a separate PEP, but one way or another it needs a PEP.

PEP 594 is meant to be a set of uncontroversial removals of mostly unused modules. Removing urllib is obviously not going to be uncontroversial, so it should be discussed in a separate PEP.

Paul _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KT6TGUBT... Code of Conduct: http://python.org/psf/codeofconduct/

My two cents: 1. I’ve grepped ("ag-ed" acually) through all my code bases, and as noted by others, urllib.parse is used in many places. urllib.request never is, as we've been using requests or httpx instead. IMHO and for the context I'm using it (YMMV), urllib.parse is useful and should be kept (could be replaced by third-party libs like furl or yarl, but probably not a great idea). 2. Obviously bootstrapping pip or similar package management tools should be a concern. 3. Overall, I think the days were "battery included" was a positive argument are over. I'd rather make the standard library leaner, and focussed on core language constructs. The only advantage that I can see is that having stuff in the standard lib can reduce fragmentation, at least initially, and ensure a very high level of quality and support, but at some point in the future history has shown us that usually a better alternative ends up emerging. 4. When deprecating and removing stuff from the stdlib and if there are no dependency issues, it should be possible to more the components to their own dedicated packages, maybe under an "extra" or "legacy" organisation. The question of support stays open, though. Regards, S. On 6 Feb 2022 at 23:39:55, Gregory P. Smith <greg@krypto.org> wrote:

On Sun, Feb 6, 2022 at 9:13 AM Paul Moore <p.f.moore@gmail.com> wrote:

...

On Sun, 6 Feb 2022 at 16:51, Christian Heimes <christian@python.org> wrote:

...

The urllib package -- and to some degree also the http package -- are constant source of security bugs. The code is old and the parsers for HTTP and URLs don't handle edge cases well. Python core lacks a true maintainer of the code. To be honest, we have to admit defeat and be up front that urllib is not up to the task for this decade. It was designed written during a more friendly, less scary time on the internet.

If I had the power and time, then I would replace urllib with a simpler, reduced HTTP client that uses platform's HTTP library under the hood (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten, maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or aiohttp are much better suited than urllib.

The second best option is to reduce the feature set of urllib to core HTTP (no ftp, proxy, HTTP auth) and a partial rewrite with stricter, more standard conform parsers for urls, query strings, and RFC 2822 instead of RFC 822 for headers.

I'd likely be fine with either of these two options. I'm not worried about supporting "advanced" uses. But having no way of getting a file from the internet without relying on 3rd party packages seems like a huge gap in functionality for a modern language. And having to use a 3rd party library to parse URLs will simply push more people to use home-grown regexes rather than something safe and correct. Remember that a lot of Python users are not professional software developers, but scientists, data analysts, and occasional users, for whom the existence of something in the stdlib is the *only* reason they have any idea that URLs need specialised parsing in the first place.

And while we all like to say 3rd party modules are great, the reality is that they provide a genuine problem for many of these non-specialist users - and I say that as a packaging specialist and pip maintainer. The packaging ecosystem is *not* newcomer-friendly in the way that core Python is, much as we're trying to improve that situation.

I've said it previously, but I'll reiterate - IMO this *must* have a PEP, and that PEP must be clear that the intention is to *remove* urllib, not simply to "deprecate and then think about it". That could be making it part of PEP 594, or a separate PEP, but one way or another it needs a PEP.

This would need to be it's own PEP. urllib et. al. are used by virtually everybody. They're highly used batteries.

I'm -1 on deprecating it for that reason alone.

Christian proposes that having a simpler scope rewrite of it might be nice, but I think disruption to the world and loss of trust in Python would be similar either way.

-gps

...

Paul _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KT6TGUBT... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QMSFZBQJ... Code of Conduct: http://python.org/psf/codeofconduct/

-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Co-Founder & Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Co-Founder & Chairman, Association Professionnelle Européenne du Logiciel Libre (APELL) - https://www.apell.info/ Co-Founder & Spokesperson, European Cloud Industrial Alliance (EUCLIDIA) - https://www.euclidia.eu/ Founder, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/

On Sun, 27 Mar 2022 at 00:52, Ethan Furman <ethan@stoneleaf.us> wrote:

[apologies for the late post, just found this in my drafts folder]

On 2/7/22 12:49 AM, Stéfane Fermigier wrote:

...

3. Overall, I think the days where "battery included" was a positive argument are over

I strongly disagree. Being able to download something and immediately get something to work and see results is hugely rewarding; on the other hand, having to research, find, compare & contrast available third-party modules (especially for new-comers) can be extremely discouraging.

100% agreed. I think people massively underestimate how frustrating it can be to have to hunt out some piece of functionality that *doesn't* come with the base download. Or worse, to find 5 different implementations, and have no idea which one you "should" use. Paul

Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

Agreed. I think one thing that's easy to forget is that Python is useful not just for very large applications but also for your standard tiny one-off scripts (i.e., cron jobs and the like). The big standard library is indispensible for writing clean code in those somewhat inherently hacky situations. Note that this isn't really unique to Python. If we look across the landscape for scripting lanugages useful in the same areas, Bash stands out as being a bit of an odd language on its own, but one that pairs well with all your traditional UNIX tools. These tools make for a decently capable 'standard library', and Bash a surprisingly versatile tool. My point obviously isn't that Python is or should be Bash, but that Python gets a tangible benefit from its batteries in much the same way as Bash gets a benefit from sed, awk, &c. Namely, these languages are both made so much more /versatile/ by their respective included tools. Being able to count on some common problems *just being solved already* is a highly appreciated language feature, and something I'd love to see stable and supported going forward. So, let's not remove urllib. The fact that we're having this discussion indicates that the scope of urllib is somewhat undefined, so perhaps it would benefit from a firm and semi-formal (re)definition of scope (hopefully in a manner motivated by real code). After that, it would probably benefit from patching up all the rough spots in the implementation side of things. To keep the metaphor going, let's recharge our dead batteries. -- lincoln auster they/them As an aside, I've opened PR#30520 on the CPython repo, which makes urllib.urlparse less dependent on the hard-coded list of schemes it uses to determine parsing behavior. In the interest of recharging batteries, would anyone be willing to review that? @orsenthil volunteered a while back, but with all of my first-PR issues, it's sort of stalled.

Time for a __legacy__ package? Kind regards, Steve On Sun, Mar 27, 2022 at 7:06 PM Paul Moore <p.f.moore@gmail.com> wrote:

On Sun, 27 Mar 2022 at 17:11, Christopher Barker <pythonchb@gmail.com> wrote:

...

With the json package included, all they need to do is `import json`. If

...

on PYPI, and it's HUGE -- most of them are for specialized JSON-using

that wasn't there, they's look in PyPi for a JSON implementation, and find an absolutely astonishing number of options. I just did a search for "JSON" protocols of some sort. I was actually really surprised that couple I know about of the top of my head (ujson, orjson) are actually hard to find.

...

"You can just pip install it" is really not a good solution.

In fact, this is an example, I think, of where we should put some effort

into making the included batteries better -- it's great to have a JSON lib built in, but it's too bad that it's not best-of-bread by pretty much any definition (speed, memory performance, flexibility) -- there are quite a few feature requests open for it -- it would be nice to actually implement some of those. (but yes, that's a lot of work that someone(s) would have to do)

...

Back to the topic at hand, rather than remove urllib, maybe it could be

made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

I think that's where the mistake happens, though. Someone who needs "best of breed" is motivated (and likely knowledgeable enough) to make informed decisions about what's on PyPI. But someone who just wants to get the job done probably doesn't - and that's the audience for the stdlib. A stdlib module needs to be a good, reliable set of basic functionality that non-experts can use successfully. There can be better libraries on PyPI, but that doesn't mean the stdlib module is unnecessary, nor does it mean that the stdlib has to match the PyPI library feature for feature.

So here, specifically, I'd rather see urlllib be the best urlllib it can be, and not demand that it turn into requests. Requests is there if people need/want it (as is httpx, and urllib3, and aiohttp). But urllib is for people who want to get a file from the web, and *not* have to deal with dependencies, 3rd party libraries, etc.

The "batteries included" standard library and PyPI complement each other. Neither is redundant, and neither implies the other is unnecessary.

Paul _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/E2FGXTYO... Code of Conduct: http://python.org/psf/codeofconduct/

On Sun, Mar 27, 2022, 11:07 AM Paul Moore <p.f.moore@gmail.com> wrote:

On Sun, 27 Mar 2022 at 17:11, Christopher Barker <pythonchb@gmail.com> wrote:

...

Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

I think that's where the mistake happens, though. Someone who needs "best of breed" is motivated (and likely knowledgeable enough) to make informed decisions about what's on PyPI. But someone who just wants to get the job done probably doesn't - and that's the audience for the stdlib. A stdlib module needs to be a good, reliable set of basic functionality that non-experts can use successfully. There can be better libraries on PyPI, but that doesn't mean the stdlib module is unnecessary, nor does it mean that the stdlib has to match the PyPI library feature for feature.

So here, specifically, I'd rather see urlllib be the best urlllib it can be, and not demand that it turn into requests. Requests is there if people need/want it (as is httpx, and urllib3, and aiohttp). But urllib is for people who want to get a file from the web, and *not* have to deal with dependencies, 3rd party libraries, etc.

One thing about talking about "make urllib more like requests" that is different than any of the other libs, though, is that requests aims to be easier to use than anything else (which I note Chris Barker called out as why he wanted urllib to be more like it). I think that's important to think about because i think ease of use is also the number one thing that the audience you talk about is also looking for. Of course, figuring out whether an api like request's is actually easier to use than urllib or merely more featureful is open to debate. -toshio

On Tue, 29 Mar 2022, 9:44 am Toshio Kuratomi, <a.badger@gmail.com> wrote:

One thing about talking about "make urllib more like requests" that is different than any of the other libs, though, is that requests aims to be easier to use than anything else (which I note Chris Barker called out as why he wanted urllib to be more like it). I think that's important to think about because i think ease of use is also the number one thing that the audience you talk about is also looking for.

Of course, figuring out whether an api like request's is actually easier to use than urllib or merely more featureful is open to debate.

There's a concrete scope difference between the two: * requests was written as a way to make HTTP(S) requests from Python, including authenticated JSON based API requests * urllib was designed to work with a variety of URL schemas which may or may not support authentication and encryption (before the rise of the web and restrictive corporate firewalls helped HTTP become the de facto standard for data exchange over the public internet) I wrote more on that topic back in 2016 [1], and the gist of that article still holds true: urllib is OK for the basic HTTP GET use case, but having to tell urllib that you want to use HTTP-only features exposes a lot of low level plumbing that end users ideally wouldn't have to care about. Unfortunately, it isn't clear where a useful subset of a full-fledged HTTP client library lies. "Make it easy to send a TLS protected JSON API POST request with Basic authentication and check the status code of the response" *might* qualify, but even that isn't particularly easy to define or maintain (What about proxies? What about redirects? What about other authentication methods?). Thus the status quo persists - adding a fully featured HTTP client library to the existing stdlib maintenance burden doesn't (or at least shouldn't) strike *anyone* as a good idea, but it also isn't clear how much of the ease of use of the existing 3rd party libraries comes from the sheer amount of functionality they have built-in so users don't have to worry about it themselves, which means defining something "lighter and easier to maintain than a fully featured HTTP client library, but still easier to use than calling in to urllib directly" isn't a viable option either. At least with urllib present in the stdlib, it's *possible* to share "No dependency" recipes for crafting particular HTTP requests, even though the default recommendation remains to grab a higher level 3rd party library instead of rolling your own solution. Cheers, Nick. [1] https://www.curiousefficiency.org/posts/2016/08/what-problem-does-it-solve.h...

-toshio _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4ZQC4H7H... Code of Conduct: http://python.org/psf/codeofconduct/

"Coyot Linden (Glenn Glazer)" <coyot@lindenlab.com> writes:

P.S. There are uses for urllib outside of standard web programming. I recently needed to handle strings in query parameter format (`?...&...&...`) and found urllib parse() the easiest way of doing that even though I wasn't taking them in as a webservice. Deleting the library would break this code. :(

+1. If urllib is removed, I would very much like to preserve at least the functionality of urlparse /somewhere/. -- Lincoln Auster they/them

I think to most people "batteries included" doesn't necessarily mean "standard library," with all that implies. It just means "it came with the first thing that I installed" (or alternatively, "at no additional charge"). Given there are *plenty* of existing distros out there that install more than just the standard library for the benefit of their users, and yet the people asking for a fatter stdlib don't seem to want them, perhaps we just need to officially endorse one? I've got no issue preinstalling a curated sumo.txt into the Windows installers, provided we're clear about support (i.e. none from us) and someone besides me is tracking everything in that curated list. That probably requires new volunteers who are somehow already trustworthy, but that should be easier than people to actually write and maintain new functionality on stdlib timelines. We would still have to have the "light" distribution, as many of our users wouldn't be able to use a "this is full of unsupported stuff" package, but at least most users could be pointed to the bigger install by default and not even care that the batteries just came in the same package and aren't actually an integral part of the core product. Cheers, Steve

On Mon, Mar 28, 2022 at 11:29 AM Paul Moore <p.f.moore@gmail.com> wrote:

To be honest, I feel like I'm just reiterating stuff I've said before here, and I think the same is true of the points I'm responding to

...

(I'm not *against* going over the debate again, it helps make sure people haven't changed their minds, but it's important to be clear that none of the practical facts have changed, if that is the case).

Maybe there's a way to make this discussion (it feels more like a discussion than debate at the moment) more productive by writing some things down. I'm not sure it's a PEP, but some kind of: "policy for the stdlib" document in which we could capture the primary points of view, places where there's consensus, etc. would be helpful to keep us retreading this over and over again. I suggest this without the bandwidth to actually shepherd the project, but if someone wants to, I think it would be a great idea. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On 29 Mar 2022, at 00:34, Brett Cannon <brett@python.org> wrote:

On Mon, Mar 28, 2022 at 11:52 AM Christopher Barker <pythonchb@gmail.com <mailto:pythonchb@gmail.com>> wrote: On Mon, Mar 28, 2022 at 11:29 AM Paul Moore <p.f.moore@gmail.com <mailto:p.f.moore@gmail.com>> wrote: To be honest, I feel like I'm just reiterating stuff I've said before here, and I think the same is true of the points I'm responding to ... (I'm not *against* going over the debate again, it helps make sure people haven't changed their minds, but it's important to be clear that none of the practical facts have changed, if that is the case).

Maybe there's a way to make this discussion (it feels more like a discussion than debate at the moment) more productive by writing some things down. I'm not sure it's a PEP, but some kind of:

"policy for the stdlib" document in which we could capture the primary points of view, places where there's consensus, etc. would be helpful to keep us retreading this over and over again.

I suggest this without the bandwidth to actually shepherd the project, but if someone wants to, I think it would be a great idea.

Once https://mail.python.org/archives/list/python-committers@python.org/thread/5E... <https://mail.python.org/archives/list/python-committers@python.org/thread/5E...> is considered resolved, the next part of my "what is the stdlib" plan is to finally try to suss all of this out and more-or-less write a stdlib policy PEP so we stop talking about this. My guess it will be more of guidance about what we want the stdlib to be and thus guide what things belong in it. No ETA on that work since I also have four other big Python projects on the go right now whose work I am constantly alternating between.

Having such a policy is a good thing and helps in evolving the stdlib, but I wonder if the lack of such a document is the real problem. IMHO the main problem is that the CPython team is very small and therefore has little bandwidth for maintaining, let alone evolving, large parts of the stdlib. In that it doesn’t help that some parts of the stdlib have APIs that make it hard to make modifications (such as distutils where effectively everything is part of the public API). Shrinking the stdlib helps in the maintenance burden, but feels as a partial solution. That said, I have no ideas on how a better stdlib development proces would look like, let alone on how to get there. Ronald

-Brett

-CHB

-- Christopher Barker, PhD (Chris)

Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GAZAFRFV... <https://mail.python.org/archives/list/python-dev@python.org/message/GAZAFRFV...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/> _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PC67DOLD... <https://mail.python.org/archives/list/python-dev@python.org/message/PC67DOLD...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>

— Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/

I was trying to think through how a "remote" stdlib might work. In the process, I got to wondering if there are known "specialists" for various current modules. Every now and then I still get assigned (or at least made nosy) about something to do with the csv module. Is there an official module-by-module list of maintainers? I was thinking about this in the context of the urllib discussion. Whether or not it might be a candidate to keep long term (as one of the handful of modules required to build and test CPython), if there are known maintainers of specific modules or packages, I think it might be worthwhile to give them the chance to chime in. Skip

There's the CODEOWNERS file: https://github.com/python/cpython/blob/main/.github/CODEOWNERS

Thanks. Never would have thought there was such a thing. I was looking for files with "maintain" in them. Skimming it, it would seem that most of the stuff in Lib or Modules isn't really associated with a particular person or small group. Skip

I'm probably overly stressing a well understood point here that urllib.parse is incredibly widely used and critical to many foundational libraries in Python. But I just came across this today that: conda is migrating from urllib3 for parsing to urllib.parse: https://github.com/conda/conda/pull/11373 And not to mention urllib3 itself somewhat depends on urllib.parse: https://github.com/urllib3/urllib3/search?q=urllib.parse Damian (he/him) On Tue, Mar 29, 2022 at 6:59 PM Alex Waygood <alex.waygood@gmail.com> wrote:

There's also the "Experts index" in the devguide: https://devguide.python.org/experts/#experts

Best, Alex

-------- Original message -------- From: Skip Montanaro <skip.montanaro@gmail.com> Date: 29/03/2022 22:36 (GMT+00:00) To: "Eric V. Smith" <eric@trueblade.com> Cc: Python Dev <python-dev@python.org> Subject: [Python-Dev] Re: Are "Batteries Included" still a Good Thing? [was: It's now time to deprecate the stdlib urllib module]

There's the CODEOWNERS file:

...

https://github.com/python/cpython/blob/main/.github/CODEOWNERS

Thanks. Never would have thought there was such a thing. I was looking for files with "maintain" in them. Skimming it, it would seem that most of the stuff in Lib or Modules isn't really associated with a particular person or small group.

Skip _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CN7SXNP2... Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Mar 29, 2022, 10:55 AM Brett Cannon <brett@python.org> wrote:

On Tue, Mar 29, 2022 at 8:58 AM Ronald Oussoren <ronaldoussoren@mac.com> wrote:

...

On 29 Mar 2022, at 00:34, Brett Cannon <brett@python.org> wrote:

On Mon, Mar 28, 2022 at 11:52 AM Christopher Barker <pythonchb@gmail.com> wrote:

...

On Mon, Mar 28, 2022 at 11:29 AM Paul Moore <p.f.moore@gmail.com> wrote:

...

Having such a policy is a good thing and helps in evolving the stdlib, but I wonder if the lack of such a document is the real problem. IMHO the main problem is that the CPython team is very small and therefore has little bandwidth for maintaining, let alone evolving, large parts of the stdlib. In that it doesn’t help that some parts of the stdlib have APIs that make it hard to make modifications (such as distutils where effectively everything is part of the public API). Shrinking the stdlib helps in the maintenance burden, but feels as a partial solution.

You're right that is the fundamental problem. But for me this somewhat stems from the fact that we don't have a shared understanding of what the stdlib *is*, and so the stdlib is a bit unbounded in its size and scope. That leads to a stdlib which is hard to maintain. It's just like dealing with any scarce resource: you try to cut back on your overall use as best as you can and then become more efficient with what you must still consume; I personally think we don't have an answer to the "must consume" part of that sentence that leads us to "cut back" to a size we can actually keep maintained so we don't have 1.6K open PRs <https://github.com/python/cpython/pulls>.

One of the things that's often missed in discussions is that a *good* policy document can also help grow the number of maintainers. As just one example, i found two interesting items in the discussion started by Skip about determining what modules don't have maintainers just downstream if this. (1) There's a file which matches maintainers to modules in the stdlib (this is documented but i only found out about it a few years ago and Skip, who's been around even longer than me didn't know about it either... So this means something about how our policy docs are currently structured could be improved). (2) Terry brought up that you don't have to be a core maintainer in order to take up ownership of something in the stdlib. That's awesome! But this is definitely something i didn't know. I've been "focusing"[1] on becoming a core maintainer because i thought that was a prerequisite to getting anything done in the stdlib. Knowing that getting involved with stdlib maintenance is different could be vastly helpful. [1] focusing is the wrong word... It expresses the feeling of "directed action" correctly but doesn't convey the lack of activity that sprinkles my attempts. Nor does it account for discouragement, helplessness, and imposter-y feelings which are the reasons for that lack. -toshio

On Tue, Mar 29, 2022 at 11:55 AM Brett Cannon <brett@python.org> wrote:

You're right that is the fundamental problem. But for me this somewhat stems from the fact that we don't have a shared understanding of what the stdlib is, and so the stdlib is a bit unbounded in its size and scope. That leads to a stdlib which is hard to maintain. It's just like dealing with any scarce resource: you try to cut back on your overall use as best as you can and then become more efficient with what you must still consume; I personally think we don't have an answer to the "must consume" part of that sentence that leads us to "cut back" to a size we can actually keep maintained so we don't have 1.6K open PRs.

This sent me down a rabbit hole of trying to analyze the PRs to see where things are failing. Using a couple libraries from PyPI (pygithub and gitignore_parser) I've mined out some interesting things. Some of the lines in CODEOWNERS are not matching what people think they are matching so notifications are being missed. The "partial matches" are PRs which have at least one matching file in CODEOWNERS so I think that those people should have been notified as well as those where the entire changeset matches via CODEOWNERS. I'm still fine tuning the script, so if more detail is desired (or some other ignorable entries like /Misc/NEWS.d/ or /**/clinic/*.h) please give feedback. So with that, here are some results: == PR Totals ======== core developers: 281 triage team: 89 draft PRs: 37 CODEOWNERS: 139 partial matches: 653 orphans: 396 ----- Total PRs: 1595 Of the files changed in all of the PRs, the following that do not match via CODEOWNERS: -- Core (/Grammar/, /Include/, /Objects/, /Python/) that should always be maintained Include/Python.h Include/bltinmodule.h Include/boolobject.h Include/buffer.h Include/bytes_methods.h Include/ceval.h Include/cpython/code.h Include/cpython/dictobject.h Include/cpython/fileutils.h Include/cpython/object.h Include/cpython/pyerrors.h Include/cpython/pystate.h Include/cpython/pythonrun.h Include/cpython/sysmodule.h Include/descrobject.h Include/exports.h Include/internal/pycore_atomic.h Include/internal/pycore_bitutils.h Include/internal/pycore_dict.h Include/internal/pycore_format.h Include/internal/pycore_global_strings.h Include/internal/pycore_interp.h Include/internal/pycore_object.h Include/internal/pycore_pathconfig.h Include/internal/pycore_pylifecycle.h Include/internal/pycore_runtime.h Include/internal/pycore_runtime_init.h Include/internal/pycore_sysmodule.h Include/methodobject.h Include/modsupport.h Include/moduleobject.h Include/object.h Include/obmalloc.h Include/opcode.h Include/opcode_names.h Include/py_curses.h Include/pydtrace.h Include/pyerrors.h Include/pymath.h Include/pyport.h Include/pyportosf.h Include/pystate.h Include/pystrtod.h Include/pythread.h Include/structmember.h Objects/abstract.c Objects/boolobject.c Objects/bytearrayobject.c Objects/bytes_methods.c Objects/bytesobject.c Objects/classobject.c Objects/complexobject.c Objects/descrobject.c Objects/fileobject.c Objects/funcobject.c Objects/genericaliasobject.c Objects/listobject.c Objects/longobject.c Objects/memoryobject.c Objects/methodobject.c Objects/moduleobject.c Objects/namespaceobject.c Objects/object.c Objects/obmalloc.c Objects/picklebufobject.c Objects/rangeobject.c Objects/sliceobject.c Objects/stringlib/split.h Objects/structseq.c Objects/tupleobject.c Objects/unicodectype.c Objects/unicodeobject.c Objects/unicodetype_db.h Objects/unionobject.c Objects/weakrefobject.c Python/Python-ast.c Python/_warnings.c Python/bltinmodule.c Python/ceval_gil.h Python/codecs.c Python/condvar.h Python/dtoa.c Python/errors.c Python/fileutils.c Python/formatter_unicode.c Python/getargs.c Python/hashtable.c Python/importlib.h Python/importlib_external.h Python/initconfig.c Python/makeopcodetargets.py Python/modsupport.c Python/opcode_targets.h Python/pathconfig.c Python/preconfig.c Python/pylifecycle.c Python/pystate.c Python/pystrtod.c Python/structmember.c Python/symtable.c Python/sysmodule.c Python/thread_nt.h Python/thread_pthread.h -- Truly Orphaned .azure-pipelines/ci.yml .azure-pipelines/libffi-build.yml .azure-pipelines/pr .azure-pipelines/pr.yml .azure-pipelines/windows-steps.yml .github/workflows/build.yml .github/workflows/build_msi.yml .github/workflows/deploy-previews.yml .github/workflows/doc.yml .github/workflows/verify-bundled-wheels.yml Doc/c-api/arg.rst Doc/c-api/dict.rst Doc/c-api/init.rst Doc/c-api/intro.rst Doc/c-api/long.rst Doc/c-api/object.rst Doc/c-api/structures.rst Doc/c-api/type.rst Doc/c-api/typeobj.rst Doc/c-api/unicode.rst Doc/conf.py Doc/data/refcounts.dat Doc/extending/embedding.rst Doc/extending/extending.rst Doc/extending/newtypes.rst Doc/extending/newtypes_tutorial.rst Doc/extending/windows.rst Doc/glossary.rst Doc/howto/functional.rst Doc/includes/custom.c Doc/includes/custom2.c Doc/includes/custom3.c Doc/includes/custom4.c Doc/includes/sqlite3/blob.py Doc/includes/sqlite3/blob_with.py Doc/includes/sublist.c Doc/library/2to3.rst Doc/library/__main__.rst Doc/library/abc.rst Doc/library/aifc.rst Doc/library/argparse.rst Doc/library/array.rst Doc/library/asyncio-tutorial/async-functions.rst Doc/library/asyncio-tutorial/case-study-chat-client-cli.rst Doc/library/asyncio-tutorial/case-study-chat-server.rst Doc/library/asyncio-tutorial/client05.py Doc/library/asyncio-tutorial/index.rst Doc/library/asyncio-tutorial/pttest.py Doc/library/asyncio-tutorial/running-async-functions.rst Doc/library/asyncio-tutorial/server01.py Doc/library/asyncio-tutorial/server02.py Doc/library/asyncio-tutorial/server03.py Doc/library/asyncio-tutorial/server04.py Doc/library/asyncio-tutorial/server05.py Doc/library/asyncio-tutorial/server20.py Doc/library/asyncio-tutorial/utils01.py Doc/library/asyncio-tutorial/utils20.py Doc/library/base64.rst Doc/library/bdb.rst Doc/library/cmd.rst Doc/library/code.rst Doc/library/codecs.rst Doc/library/concurrent.futures.rst Doc/library/configparser.rst Doc/library/crypt.rst Doc/library/csv.rst Doc/library/ctypes.rst Doc/library/curses.ascii.rst Doc/library/curses.panel.rst Doc/library/curses.rst Doc/library/difflib.rst Doc/library/dis.rst Doc/library/distutils.rst Doc/library/doctest.rst Doc/library/ensurepip.rst Doc/library/errno.rst Doc/library/exceptions.rst Doc/library/fileinput.rst Doc/library/formatter.rst Doc/library/fractions.rst Doc/library/ftplib.rst Doc/library/functions.rst Doc/library/getpass.rst Doc/library/gettext.rst Doc/library/gzip.rst Doc/library/http.client.rst Doc/library/http.cookiejar.rst Doc/library/http.cookies.rst Doc/library/http.rst Doc/library/http.server.rst Doc/library/idle.rst Doc/library/importlib.metadata.rst Doc/library/importlib.rst Doc/library/inspect.rst Doc/library/io.rst Doc/library/ipaddress.rst Doc/library/locale.rst Doc/library/math.rst Doc/library/mmap.rst Doc/library/multiprocessing.rst Doc/library/netrc.rst Doc/library/numbers.rst Doc/library/operator.rst Doc/library/os.path.rst Doc/library/os.rst Doc/library/pathlib.rst Doc/library/pdb.rst Doc/library/pickle.rst Doc/library/pkgutil.rst Doc/library/platform.rst Doc/library/plistlib.rst Doc/library/profile.rst Doc/library/pyexpat.rst Doc/library/re.rst Doc/library/reprlib.rst Doc/library/rlcompleter.rst Doc/library/runpy.rst Doc/library/sched.rst Doc/library/secrets.rst Doc/library/selectors.rst Doc/library/shlex.rst Doc/library/shutil.rst Doc/library/signal.rst Doc/library/site.rst Doc/library/socket.rst Doc/library/socketserver.rst Doc/library/ssl.rst Doc/library/stat.rst Doc/library/stdtypes.rst Doc/library/string.rst Doc/library/struct.rst Doc/library/subprocess.rst Doc/library/sunau.rst Doc/library/sys.rst Doc/library/telnetlib.rst Doc/library/tempfile.rst Doc/library/test.rst Doc/library/textwrap.rst Doc/library/timeit.rst Doc/library/tkinter.dnd.rst Doc/library/tkinter.font.rst Doc/library/tkinter.rst Doc/library/trace.rst Doc/library/traceback.rst Doc/library/tty.rst Doc/library/unicodedata.rst Doc/library/unittest.mock-examples.rst Doc/library/unittest.mock.rst Doc/library/unittest.rst Doc/library/urllib.parse.rst Doc/library/urllib.request.rst Doc/library/warnings.rst Doc/library/wave.rst Doc/library/weakref.rst Doc/library/winreg.rst Doc/library/wsgiref.rst Doc/library/xml.dom.pulldom.rst Doc/library/xml.etree.elementtree.rst Doc/library/xml.sax.handler.rst Doc/library/zipfile.rst Doc/library/zlib.rst Doc/license.rst Doc/reference/compound_stmts.rst Doc/reference/datamodel.rst Doc/reference/executionmodel.rst Doc/reference/expressions.rst Doc/reference/import.rst Doc/reference/lexical_analysis.rst Doc/reference/simple_stmts.rst Doc/requirements.txt Doc/tools/extensions/pyspecific.py Doc/tools/extensions/unidata_version.py Doc/tutorial/appendix.rst Doc/tutorial/errors.rst Doc/tutorial/inputoutput.rst Doc/tutorial/interactive.rst Doc/tutorial/modules.rst Doc/tutorial/stdlib.rst Doc/using/cmdline.rst Doc/using/windows.rst Lib/__init__.py Lib/_dummy_thread.py Lib/_py_abc.py Lib/_pyio.py Lib/_weakrefset.py Lib/abc.py Lib/aifc.py Lib/argparse.py Lib/asyncio/__main__.py Lib/asyncio/base_events.py Lib/asyncio/base_subprocess.py Lib/asyncio/events.py Lib/asyncio/futures.py Lib/asyncio/locks.py Lib/asyncio/proactor_events.py Lib/asyncio/runners.py Lib/asyncio/selector_events.py Lib/asyncio/sslproto.py Lib/asyncio/streams.py Lib/asyncio/subprocess.py Lib/asyncio/tasks.py Lib/asyncio/unix_events.py Lib/asyncio/windows_events.py Lib/asyncore.py Lib/base64.py Lib/bdb.py Lib/cProfile.py Lib/cmd.py Lib/collections/__init__.py Lib/compileall.py Lib/concurrent/futures/_base.py Lib/concurrent/futures/process.py Lib/concurrent/futures/thread.py Lib/configparser.py Lib/crypt.py Lib/csv.py Lib/ctypes/__init__.py Lib/ctypes/_aix.py Lib/ctypes/macholib/dyld.py Lib/ctypes/macholib/dylib.py Lib/ctypes/macholib/framework.py Lib/ctypes/test/test_callbacks.py Lib/ctypes/test/test_find.py Lib/ctypes/test/test_functions.py Lib/ctypes/test/test_loading.py Lib/ctypes/test/test_macholib.py Lib/ctypes/test/test_pep3118.py Lib/ctypes/test/test_python_api.py Lib/ctypes/test/test_refcounts.py Lib/ctypes/test/test_structures.py Lib/ctypes/test/test_win32.py Lib/ctypes/util.py Lib/curses/textpad.py Lib/difflib.py Lib/dis.py Lib/distutils/_msvccompiler.py Lib/distutils/ccompiler.py Lib/distutils/command/build_ext.py Lib/distutils/tests/test_build_ext.py Lib/distutils/tests/test_check.py Lib/distutils/tests/test_dist.py Lib/distutils/tests/test_sdist.py Lib/distutils/tests/test_sysconfig.py Lib/distutils/version.py Lib/doctest.py Lib/email/_header_value_parser.py Lib/email/contentmanager.py Lib/email/feedparser.py Lib/email/generator.py Lib/email/header.py Lib/email/message.py Lib/email/mime/signed.py Lib/email/mime/text.py Lib/email/parser.py Lib/email/policy.py Lib/email/utils.py Lib/encodings/aliases.py Lib/encodings/cp1250.py Lib/encodings/cp1251.py Lib/encodings/cp1252.py Lib/encodings/cp1253.py Lib/encodings/cp1254.py Lib/encodings/cp1255.py Lib/encodings/cp1256.py Lib/encodings/cp1257.py Lib/encodings/cp1258.py Lib/encodings/cp874.py Lib/encodings/idna.py Lib/encodings/punycode.py Lib/encodings/undefined.py Lib/encodings/utf_16.py Lib/encodings/utf_32.py Lib/ensurepip/__init__.py Lib/ensurepip/_bundled/.gitignore Lib/ensurepip/_bundled/pip-19.0.3-py2.py3-none-any.whl Lib/ensurepip/_bundled/setuptools-40.8.0-py2.py3-none-any.whl Lib/ensurepip/_bundler.py Lib/ensurepip/bundle.py Lib/fileinput.py Lib/fnmatch.py Lib/fractions.py Lib/ftplib.py Lib/genericpath.py Lib/getpass.py Lib/gettext.py Lib/gzip.py Lib/http/__init__.py Lib/http/client.py Lib/http/cookiejar.py Lib/http/cookies.py Lib/http/server.py Lib/idlelib/Icons/debug_current.gif Lib/idlelib/Icons/debug_go.gif Lib/idlelib/Icons/debug_go_disabled.gif Lib/idlelib/Icons/debug_line.gif Lib/idlelib/Icons/debug_out.gif Lib/idlelib/Icons/debug_out_disabled.gif Lib/idlelib/Icons/debug_over.gif Lib/idlelib/Icons/debug_over_disabled.gif Lib/idlelib/Icons/debug_prefs.gif Lib/idlelib/Icons/debug_prefs_disabled.gif Lib/idlelib/Icons/debug_step.gif Lib/idlelib/Icons/debug_step_disabled.gif Lib/idlelib/Icons/debug_stop.gif Lib/idlelib/Icons/debug_stop_disabled.gif Lib/idlelib/ParenClose.py Lib/idlelib/README.txt Lib/idlelib/config-extensions.def Lib/idlelib/config-keys.def Lib/idlelib/config-main.def Lib/idlelib/config.py Lib/idlelib/configdialog.py Lib/idlelib/debugger.py Lib/idlelib/debugger_r.py Lib/idlelib/dynoption.py Lib/idlelib/editor.py Lib/idlelib/filelist.py Lib/idlelib/help.html Lib/idlelib/idle_test/htest.py Lib/idlelib/idle_test/test_calltip.py Lib/idlelib/idle_test/test_config.py Lib/idlelib/idle_test/test_configdialog.py Lib/idlelib/idle_test/test_debugger.py Lib/idlelib/idle_test/test_editor.py Lib/idlelib/idle_test/test_parenclose.py Lib/idlelib/idle_test/test_query.py Lib/idlelib/idle_test/test_scrolledlist.py Lib/idlelib/idle_test/test_textview.py Lib/idlelib/iomenu.py Lib/idlelib/mainmenu.py Lib/idlelib/navigatebar.py Lib/idlelib/pyshell.py Lib/idlelib/query.py Lib/idlelib/replace.py Lib/idlelib/run.py Lib/idlelib/scrolledlist.py Lib/idlelib/search.py Lib/idlelib/textview.py Lib/imghdr.py Lib/importlib/_bootstrap_external.py Lib/importlib/abc.py Lib/inspect.py Lib/ipaddress.py Lib/json/__init__.py Lib/json/decoder.py Lib/json/encoder.py Lib/json/scanner.py Lib/json/tool.py Lib/lib2to3/tests/support.py Lib/lib2to3/tests/test_parser.py Lib/linecache.py Lib/locale.py Lib/logging/__init__.py Lib/logging/config.py Lib/logging/handlers.py Lib/modulefinder.py Lib/multiprocessing/__init__.py Lib/multiprocessing/connection.py Lib/multiprocessing/dummy/__init__.py Lib/multiprocessing/forkserver.py Lib/multiprocessing/heap.py Lib/multiprocessing/managers.py Lib/multiprocessing/pool.py Lib/multiprocessing/popen_fork.py Lib/multiprocessing/popen_forkserver.py Lib/multiprocessing/popen_spawn_posix.py Lib/multiprocessing/popen_spawn_win32.py Lib/multiprocessing/process.py Lib/multiprocessing/reduction.py Lib/multiprocessing/resource_tracker.py Lib/multiprocessing/spawn.py Lib/multiprocessing/synchronize.py Lib/multiprocessing/util.py Lib/netrc.py Lib/ntpath.py Lib/numbers.py Lib/opcode.py Lib/os.py Lib/pathlib.py Lib/pdb.py Lib/pickle.py Lib/pickletools.py Lib/pkgutil.py Lib/platform.py Lib/plistlib.py Lib/pprint.py Lib/profile.py Lib/pstats.py Lib/pty.py Lib/pyclbr.py Lib/pydoc.py Lib/pydoc_data/topics.py Lib/quopri.py Lib/re.py Lib/reprlib.py Lib/rlcompleter.py Lib/runpy.py Lib/sched.py Lib/selectors.py Lib/shelve.py Lib/shlex.py Lib/shutil.py Lib/site.py Lib/sndhdr.py Lib/socket.py Lib/socketserver.py Lib/sqlite3/dbapi2.py Lib/sqlite3/dump.py Lib/sqlite3/test/dbapi.py Lib/sqlite3/test/regression.py Lib/sqlite3/test/test_dump.py Lib/sqlite3/test/test_namedrow.py Lib/sre_compile.py Lib/sre_constants.py Lib/ssl.py Lib/stat.py Lib/string.py Lib/sunau.py Lib/sysconfig.py Lib/tabnanny.py Lib/telnetlib.py Lib/tempfile.py Lib/test/_test_multiprocessing.py Lib/test/_typed_dict_helper.py Lib/test/ann_module.py Lib/test/audiodata/pluck-pcm24-ext.wav Lib/test/audiotests.py Lib/test/audit-tests.py Lib/test/clinic.test Lib/test/dataclass_forward_ref.py Lib/test/dataclass_forward_ref_child.py Lib/test/dataclass_forward_ref_str.py Lib/test/final_a.py Lib/test/final_b.py Lib/test/imghdrdata/python.jpg Lib/test/imghdrdata/python2.jpeg Lib/test/inspect_fodder.py Lib/test/libregrtest/refleak.py Lib/test/libregrtest/runtest.py Lib/test/libregrtest/win_utils.py Lib/test/list_tests.py Lib/test/mod_generics_cache.py Lib/test/pickletester.py Lib/test/sndhdrdata/input1.bad Lib/test/sndhdrdata/input2.bad Lib/test/string_tests.py Lib/test/support/__init__.py Lib/test/test__opcode.py Lib/test/test__xxsubinterpreters.py Lib/test/test_abc.py Lib/test/test_aifc.py Lib/test/test_argparse.py Lib/test/test_array.py Lib/test/test_ast.py Lib/test/test_asyncgen.py Lib/test/test_asyncio/test_base_events.py Lib/test/test_asyncio/test_events.py Lib/test/test_asyncio/test_futures.py Lib/test/test_asyncio/test_futures2.py Lib/test/test_asyncio/test_proactor_events.py Lib/test/test_asyncio/test_selector_events.py Lib/test/test_asyncio/test_sslproto.py Lib/test/test_asyncio/test_streams.py Lib/test/test_asyncio/test_subprocess.py Lib/test/test_asyncio/test_tasks.py Lib/test/test_asyncio/test_unix_events.py Lib/test/test_asyncio/test_unix_pipes.py Lib/test/test_asyncio/utils.py Lib/test/test_audioop.py Lib/test/test_audit.py Lib/test/test_base64.py Lib/test/test_bdb.py Lib/test/test_buffer.py Lib/test/test_builtin.py Lib/test/test_bytes.py Lib/test/test_bz2.py Lib/test/test_c_locale_coercion.py Lib/test/test_calendar.py Lib/test/test_capi.py Lib/test/test_cmd.py Lib/test/test_cmd_line.py Lib/test/test_cmd_line_script.py Lib/test/test_codecs.py Lib/test/test_compileall.py Lib/test/test_concurrent_futures.py Lib/test/test_configparser.py Lib/test/test_coroutines.py Lib/test/test_cprofile.py Lib/test/test_crypt.py Lib/test/test_csv.py Lib/test/test_curses.py Lib/test/test_decorators.py Lib/test/test_descr.py Lib/test/test_dis.py Lib/test/test_doctest.py Lib/test/test_dynamicclassattribute.py Lib/test/test_email/test__header_value_parser.py Lib/test/test_email/test_contentmanager.py Lib/test/test_email/test_defect_handling.py Lib/test/test_email/test_generator.py Lib/test/test_email/test_headerregistry.py Lib/test/test_email/test_message.py Lib/test/test_email/test_multipart_signed.py Lib/test/test_email/test_parser.py Lib/test/test_email/test_pickleable.py Lib/test/test_email/test_policy.py Lib/test/test_embed.py Lib/test/test_ensurepip.py Lib/test/test_faulthandler.py Lib/test/test_fileinput.py Lib/test/test_fileio.py Lib/test/test_float.py Lib/test/test_fnmatch.py Lib/test/test_format.py Lib/test/test_fractions.py Lib/test/test_future.py Lib/test/test_gc.py Lib/test/test_gdb.py Lib/test/test_genericalias.py Lib/test/test_genericpath.py Lib/test/test_getargs2.py Lib/test/test_getpass.py Lib/test/test_gettext.py Lib/test/test_grp.py Lib/test/test_gzip.py Lib/test/test_hmac.py Lib/test/test_http/test_http.py Lib/test/test_http_cookiejar.py Lib/test/test_http_cookies.py Lib/test/test_httplib.py Lib/test/test_httpservers.py Lib/test/test_imghdr.py Lib/test/test_import/__init__.py Lib/test/test_importlib/extension/test_loader.py Lib/test/test_importlib/fixtures.py Lib/test/test_importlib/import_/test_path.py Lib/test/test_importlib/test_abc.py Lib/test/test_importlib/test_main.py Lib/test/test_importlib/test_metadata_api.py Lib/test/test_importlib/test_util.py Lib/test/test_inspect.py Lib/test/test_io.py Lib/test/test_ipaddress.py Lib/test/test_json/test_decode.py Lib/test/test_json/test_dump.py Lib/test/test_json/test_indent.py Lib/test/test_json/test_separators.py Lib/test/test_json/test_speedups.py Lib/test/test_json/test_tool.py Lib/test/test_linecache.py Lib/test/test_list.py Lib/test/test_lltrace.py Lib/test/test_locale.py Lib/test/test_lzma.py Lib/test/test_memoryio.py Lib/test/test_memoryview.py Lib/test/test_minidom.py Lib/test/test_mmap.py Lib/test/test_module.py Lib/test/test_modulefinder.py Lib/test/test_multiprocessing_dummy.py Lib/test/test_netrc.py Lib/test/test_nntplib.py Lib/test/test_ntpath.py Lib/test/test_numeric_tower.py Lib/test/test_os.py Lib/test/test_pathlib.py Lib/test/test_pdb.py Lib/test/test_peg_parser.py Lib/test/test_pkg.py Lib/test/test_pkgutil.py Lib/test/test_plistlib.py Lib/test/test_posix.py Lib/test/test_pprint.py Lib/test/test_profile.py Lib/test/test_property.py Lib/test/test_pstats.py Lib/test/test_pulldom.py Lib/test/test_pyclbr.py Lib/test/test_pydoc.py Lib/test/test_pyexpat.py Lib/test/test_range.py Lib/test/test_re.py Lib/test/test_regrtest.py Lib/test/test_repl.py Lib/test/test_reprlib.py Lib/test/test_rlcompleter.py Lib/test/test_selectors.py Lib/test/test_set.py Lib/test/test_shlex.py Lib/test/test_shutil.py Lib/test/test_signal.py Lib/test/test_site.py Lib/test/test_sndhdr.py Lib/test/test_socket.py Lib/test/test_socketserver.py Lib/test/test_source_encoding.py Lib/test/test_ssl.py Lib/test/test_stat.py Lib/test/test_str.py Lib/test/test_string.py Lib/test/test_struct.py Lib/test/test_sunau.py Lib/test/test_super.py Lib/test/test_support.py Lib/test/test_syntax.py Lib/test/test_sys.py Lib/test/test_sysconfig.py Lib/test/test_tabnanny.py Lib/test/test_telnetlib.py Lib/test/test_tempfile.py Lib/test/test_textwrap.py Lib/test/test_thread.py Lib/test/test_threading.py Lib/test/test_timeit.py Lib/test/test_tk.py Lib/test/test_tokenize.py Lib/test/test_tools/test_c_analyzer/test_cpython/test_supported.py Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py Lib/test/test_tools/test_c_analyzer/test_parser/test_preprocessor.py Lib/test/test_tools/test_c_analyzer/test_variables/test_known.py Lib/test/test_tools/test_i18n.py Lib/test/test_tools/test_pindent.py Lib/test/test_trace.py Lib/test/test_ttk_guionly.py Lib/test/test_ttk_textonly.py Lib/test/test_tuple.py Lib/test/test_types.py Lib/test/test_unicode.py Lib/test/test_unicodedata.py Lib/test/test_unparse.py Lib/test/test_urllib.py Lib/test/test_urllib2.py Lib/test/test_urllib2net.py Lib/test/test_urlparse.py Lib/test/test_userstring.py Lib/test/test_utf8_mode.py Lib/test/test_warnings/__init__.py Lib/test/test_wave.py Lib/test/test_weakref.py Lib/test/test_winconsoleio.py Lib/test/test_winreg.py Lib/test/test_wsgiref.py Lib/test/test_xml_etree.py Lib/test/test_zipapp.py Lib/test/test_zipfile.py Lib/test/test_zlib.py Lib/test/test_zoneinfo/test_zoneinfo.py Lib/test/tkinter_test/README Lib/test/tkinter_test/__init__.py Lib/test/tkinter_test/runtktests.py Lib/test/tkinter_test/support.py Lib/test/tkinter_test/test_tkinter/__init__.py Lib/test/tkinter_test/test_tkinter/test_font.py Lib/test/tkinter_test/test_tkinter/test_geometry_managers.py Lib/test/tkinter_test/test_tkinter/test_images.py Lib/test/tkinter_test/test_tkinter/test_loadtk.py Lib/test/tkinter_test/test_tkinter/test_misc.py Lib/test/tkinter_test/test_tkinter/test_text.py Lib/test/tkinter_test/test_tkinter/test_variables.py Lib/test/tkinter_test/test_tkinter/test_widgets.py Lib/test/tkinter_test/test_ttk/__init__.py Lib/test/tkinter_test/test_ttk/test_extensions.py Lib/test/tkinter_test/test_ttk/test_functions.py Lib/test/tkinter_test/test_ttk/test_style.py Lib/test/tkinter_test/test_ttk/test_widgets.py Lib/test/tkinter_test/widget_tests.py Lib/textwrap.py Lib/threading.py Lib/timeit.py Lib/tkinter/__init__.py Lib/tkinter/constants.py Lib/tkinter/dnd.py Lib/tkinter/simpledialog.py Lib/tkinter/test/test_tkinter/test_event.py Lib/tkinter/test/test_tkinter/test_misc.py Lib/tkinter/test/test_tkinter/test_threads.py Lib/tkinter/test/test_tkinter/test_tkdnd.py Lib/tokenize.py Lib/trace.py Lib/tty.py Lib/turtle.py Lib/types.py Lib/unittest/case.py Lib/unittest/loader.py Lib/unittest/main.py Lib/unittest/result.py Lib/unittest/runner.py Lib/unittest/signals.py Lib/unittest/suite.py Lib/unittest/test/support.py Lib/unittest/test/test_assertions.py Lib/unittest/test/test_break.py Lib/unittest/test/test_case.py Lib/unittest/test/test_discovery.py Lib/unittest/test/test_loader.py Lib/unittest/test/test_program.py Lib/unittest/test/test_result.py Lib/unittest/test/test_runner.py Lib/unittest/test/test_setups.py Lib/unittest/test/test_suite.py Lib/urllib/parse.py Lib/urllib/request.py Lib/uuid.py Lib/venv/__init__.py Lib/venv/scripts/common/Activate.ps1 Lib/venv/scripts/common/activate Lib/venv/scripts/posix/activate.csh Lib/venv/scripts/posix/activate.fish Lib/warnings.py Lib/wave.py Lib/weakref.py Lib/webbrowser.py Lib/wsgiref/handlers.py Lib/wsgiref/headers.py Lib/wsgiref/simple_server.py Lib/wsgiref/validate.py Lib/xdrlib.py Lib/xml/dom/expatbuilder.py Lib/xml/dom/minidom.py Lib/xml/dom/pulldom.py Lib/xml/etree/ElementPath.py Lib/xml/etree/ElementTree.py Lib/xml/sax/xmlreader.py Lib/xmlrpc/client.py Lib/xmlrpc/server.py Lib/zipapp.py Lib/zipfile.py Lib/zoneinfo/_common.py Makefile.pre.in Misc/ACKS Misc/NEWS Misc/README.valgrind Misc/python-config.in Misc/python-config.sh.in Misc/python.man Misc/python.pc.in Misc/valgrind-python.supp Misc/verify-bundled-wheels.sh Modules/Makefile Modules/Setup Modules/_abc.c Modules/_bz2module.c Modules/_csv.c Modules/_ctypes/_ctypes.c Modules/_ctypes/_ctypes_test.c Modules/_ctypes/callbacks.c Modules/_ctypes/callproc.c Modules/_ctypes/cfield.c Modules/_ctypes/ctypes.h Modules/_ctypes/stgdict.c Modules/_cursesmodule.c Modules/_decimal/libmpdec/io.c Modules/_decimal/tests/formathelper.py Modules/_elementtree.c Modules/_hashopenssl.c Modules/_io/_iomodule.c Modules/_io/bufferedio.c Modules/_io/bytesio.c Modules/_io/fileio.c Modules/_io/iobase.c Modules/_io/textio.c Modules/_io/winconsoleio.c Modules/_json.c Modules/_localemodule.c Modules/_lsprof.c Modules/_lzmamodule.c Modules/_multiprocessing/clinic/posixshmem.c.h Modules/_multiprocessing/multiprocessing.h Modules/_multiprocessing/posixshmem.c Modules/_multiprocessing/semaphore.c Modules/_operator.c Modules/_pickle.c Modules/_sqlite/blob.c Modules/_sqlite/blob.h Modules/_sqlite/connection.c Modules/_sqlite/connection.h Modules/_sqlite/cursor.c Modules/_sqlite/module.c Modules/_sqlite/named_row.c Modules/_sqlite/named_row.h Modules/_sre.c Modules/_ssl.c Modules/_ssl_data.h Modules/_stat.c Modules/_struct.c Modules/_testcapimodule.c Modules/_threadmodule.c Modules/_tkinter.c Modules/_tracemalloc.c Modules/_uuidmodule.c Modules/_winapi.c Modules/_xxsubinterpretersmodule.c Modules/_zoneinfo.c Modules/arraymodule.c Modules/audioop.c Modules/cjkcodecs/clinic/multibytecodec.c.h Modules/cjkcodecs/multibytecodec.c Modules/errnomodule.c Modules/fcntlmodule.c Modules/getpath.c Modules/grapheme_cluster_break_automaton.h Modules/main.c Modules/makesetup Modules/mmapmodule.c Modules/nismodule.c Modules/ossaudiodev.c Modules/overlapped.c Modules/posixmodule.c Modules/pyexpat.c Modules/selectmodule.c Modules/signalmodule.c Modules/socketmodule.c Modules/socketmodule.h Modules/sre.h Modules/sre_constants.h Modules/sre_lib.h Modules/unicodedata.c Modules/unicodedata_db.h Modules/unicodename_db.h Modules/xxsubtype.c Modules/zlibmodule.c Programs/_testembed.c Tools/c-analyzer/c_parser/_state_machine.py Tools/c-analyzer/c_parser/preprocessor/__init__.py Tools/c-analyzer/cpython/_parser.py Tools/ccbench/ccbench.py Tools/clinic/clinic.py Tools/gdb/libpython.py Tools/i18n/msgfmt.py Tools/scripts/README Tools/scripts/deepfreeze.py Tools/scripts/eptags.py Tools/scripts/generate_global_objects.py Tools/scripts/generate_opcode_h.py Tools/scripts/ptags.py Tools/scripts/win_add2path.py Tools/ssl/multissltests.py Tools/unicode/Makefile Tools/unicode/makeunicodedata.py abc_bench.py aclocal.m4 configure configure.ac m4/ax_pthread.m4 pyconfig.h.in setup.py test.py -- Jeremy Kloth

On 3/28/2022 7:26 PM, Paul Moore wrote:

To be honest, I feel like I'm just reiterating stuff I've said before here, and I think the same is true of the points I'm responding to. Is there actually any new development here, or is it just a repeat of the same positions people have expressed in the past, based on the same underlying realities? (I'm not *against* going over the debate again, it helps make sure people haven't changed their minds, but it's important to be clear that none of the practical facts have changed, if that is the case).

Yep, same. Nothing much has changed, but if those of us who know that nothing has changed fail to reiterate those points, everyone new to the discussion will assume that everyone agrees with them. Just deleted a long rant to the discussion in general (not directed at Paul at all ;) ) that would have achieved nothing, but I'll summarise to this: email doesn't fix bugs; maintainers fix bugs. Please let us know *publicly* if you want to become the maintainer for a stdlib module and then we can support them, but if nobody is willing/able/ready to care for them it's irresponsible for us to keep shipping them to users.

email doesn't fix bugs; maintainers fix bugs. Please let us know *publicly* if you want to become the maintainer for a stdlib module and then we can support them, but if nobody is willing/able/ready to care for them it's irresponsible for us to keep shipping them to users.

At the moment, I don't have the time, experience, skill, nor involvement with the Python project to volunteer to maintain urllib, but I would like to take a more active role in its maintenance as a contributor such that we continue to see it in the stdlib. This may be the second time I've mentioned this on this thread, but please see PR#30520 on the CPython repo (BPO-46337), which attempts to patch up a rough edge in urlparse's scheme-based API. Feedback/review/etc on that would be greatly appreciated, and so would some pointers on what next needs the most work with regards to urllib as a whole. Thanks so much! -- Lincoln Auster they/them

On 30Mar2022 1124, Barney Gale wrote:

I'd like to become a maintainer for the pathlib module, if possible. I know the code and tests inside-out, and I've been wrestling the internals for past few Python releases. I check the bugs/PRs at least every week and help wherever I can.

Antoine is still active, so if he's fine with that, I'm also supportive. You're certainly familiar enough with that module.

I can also look after urllib if it would help. I have much less experience with that module vs pathlib, so I'll need to spend some time familiarizing myself with the code and the outstanding bug reports. But I do have plenty of experience with networking, HTTP, URLs, etc :-). I'll spend some time reading the bug tracker to start.

Please! I skim the open issues occasionally, but definitely feel out of my depth to decide on issues (e.g. should urlparse("http://[::1]spam:80") raise? - I have literally no idea :) ), but feel free to nosy me on issues you think you've figured out and I can help confirm your logic and do merges. Or if another core dev has been helping you, keep nosying them. Cheers, Steve

On Wed, 30 Mar 2022 at 12:20, Antoine Pitrou <antoine@python.org> wrote:

On Wed, 30 Mar 2022 12:03:58 +0100 Steve Dower <steve.dower@python.org> wrote:

...

On 30Mar2022 1124, Barney Gale wrote:

...

I'd like to become a maintainer for the pathlib module, if possible. I know the code and tests inside-out, and I've been wrestling the internals for past few Python releases. I check the bugs/PRs at least every week and help wherever I can.

Antoine is still active, so if he's fine with that, I'm also supportive. You're certainly familiar enough with that module.

I'm entirely supportive. Huge thank you for working on pathlib!

Thank you Antoine, Steve. It's humbling to stand on your shoulders and thrilling to give back to Python. If I'm successful in my application I'll do my best to take good care of pathlib and hopefully urllib too. These batteries ain't dead yet! :D Best, Barney

Le 28/03/2022 à 18:44, Steve Dower a écrit :

I think to most people "batteries included" doesn't necessarily mean "standard library," with all that implies. It just means "it came with the first thing that I installed" (or alternatively, "at no additional charge").

A point I have not seen made, is that some uses really need *standard* batteries. In scientific research, you'll sometimes share a data-processing script among a large group of non-computer-specialists, who hopefully all have *some* Python+NumPy installed: it may be a full up-to-date Conda, or it may be "a very old version that a former colleague installed for me years ago and now I won't update it for fear I break it". You have to cater to all versions. It may be more glamorous to compete with Rust and JS for "best modern language", than with Excel for "lowest common denominator for calculation". Still, that's a use case where Python has historically been strong, and it's one of the reasons it works so well in the research world. Hopefully this use case is not forgotten, even if those non-computer-specialists users are less likely to be involved here in core development. Cheers, Baptiste

On Wed, 30 Mar 2022 at 12:28, Steve Dower <steve.dower@python.org> wrote:

On 30Mar2022 1132, Baptiste Carvello wrote:

...

Le 28/03/2022 à 18:44, Steve Dower a écrit :

...

I think to most people "batteries included" doesn't necessarily mean "standard library," with all that implies. It just means "it came with the first thing that I installed" (or alternatively, "at no additional charge").

A point I have not seen made, is that some uses really need *standard* batteries.

I've certainly not forgotten it, it's just every time I try to make the point it doesn't seem to help. So until I find a clear way to argue that, yes, a standard "datetime" value makes sense, but no, a standard "HTTP headers dictionary" is unnecessary, I'm avoiding bringing it up again ;)

I also strongly agree with this position, and do my best to argue it. But as you say, it is hard to draw the line. Here's some examples, which while not exhaustive, might give a flavour of my views: "Obviously" things that should be in the stdlib (basic data types or algorithms): re, datetime, collections, enum, dataclasses, decimal, fractions (maybe borderline), statistics, itertools, functools, pathlib Obvious stuff that's OS related: os, io, threading, multiprocessing, concurrent.futures, subprocess, asyncio, urllib.request Core services: sys, venv, sysconfig, warnings, atexit, gc, zipimport, runpy, importlib, marshal, pickle Standard development tools: argparse, pdb, cprofile, typing (arguable in some ways) Handling of key file and data formats: struct, csv, json, tomllib, zlib, gzip, bz2 (borderline), zipfile, tarfile, base64 Arguable cases of file formats, but with important use cases xml (you may not like it, but it's still used a lot) configparser (similarly, ini-style files are used a lot, for better or worse) email (this is an outlier, but the packaging metadata file format uses it, so there's a bootstrapping issue if it's not in the stdlib) I've missed a lot out, and I'm sure people will argue most of these, but that's sort of the point - it's *hard* to come up with a blanket statement on how we should approach "the stdlib", because there's so much diversity in there. That's why my starting position is that we should retain the stdlib - sure, we can remove (and add!) stuff, but we should do so on a case by case basis, not based on some blanket "minimise it because we don't have the manpower or we don't want to support it" basis. If writing and sharing small, one-off scripts that depended on non-stdlib packages were *significantly* easier, I might revise my position on some of the arguable cases. PEP 582 (Python local packages directory) might make a big difference here. But at the moment, adding a 3rd party dependency effectively alters what you're doing from "writing a script I can share" to "developing an application" - and that's a *big* step for many people. Quoting Nathaniel Smith from https://mail.python.org/archives/list/distutils-sig@python.org/message/YFJIT... (albeit in a slightly different context): """ This last point is important: as projects grow in complexity, and are used by a wider audience, they often end up with fairly complex tool specifications that have to be shared among a team. """ My argument is that relying only on the stdlib allows people putting off the need to take that step up in complexity, and we shouldn't dismiss how significant that is. Paul

Am 27.03.22 um 18:11 schrieb Christopher Barker:

On Sun, Mar 27, 2022 at 3:08 AM Paul Moore <p.f.moore@gmail.com> wrote:

> > 3. Overall, I think the days where "battery included" was a positive argument are over > > I strongly disagree.  Being able to download something and immediately get something to work and see results is hugely > rewarding; on the other hand, having to research, find, compare & contrast available third-party modules (especially for > new-comers) can be extremely discouraging.

exactly - let's say someone needs to write some JSON for the first time.

I think the truth is somewhere in the middle. Python should include support for commonly needed functionality. This includes all kinds of language support (functools, collections, asyncio, wsgiref, re etc.), common file formats like XML, JSON, tar, zip etc., OS support (os, sys, stat etc.), basic networking (TCP/IP, making HTTP requests, sending mail), but also basic math modules like decimal, fractions, or statistics. I don't think it should support esoteric, niche, or long obsolete file formats or networking protocols like FTP, sunau etc. I also don't think that Python should come with "basic" servers that can't be used for anything useful. The same is true for domain-specific applications. For example, I wouldn't want a complex numerics package as part of Python (although having some more basic data types would make sense). Another problem are "optional" modules like sqlite or tk/tcl. Because they are optional, you can't depend on them being present, but you also can't ensure their presence by "pip install"ing them.

In fact, this is an example, I think, of where we should put some effort into making the included batteries better -- it's great to have a JSON lib built in, but it's too bad that it's not best-of-bread by pretty much any definition (speed, memory performance, flexibility) -- there are quite a few feature requests open for it -- it would be nice to actually implement some of those. (but yes, that's a lot of work that someone(s) would have to do)

Back to the topic at hand, rather than remove urllib, maybe it could be made better -- an as-easy-to-use-as-requests package in the stdlib would be really great.

I agree 100%. On the one hand the stdlib is missing some functionality I would consider "basic" (for example, async file and HTTP fetching support), on the other hand some of the existing modules would benefit from a more modern API that makes the common case easy and the uncommon case possible.  - Sebastian

On 2022-03-27 19:52, Dan Stromberg wrote:

On Sat, Mar 26, 2022 at 5:58 PM Ethan Furman <ethan@stoneleaf.us <mailto:ethan@stoneleaf.us>> wrote:

[apologies for the late post, just found this in my drafts folder]

On 2/7/22 12:49 AM, Stéfane Fermigier wrote:

> 3. Overall, I think the days where "battery included" was a positive argument are over

I strongly disagree.  Being able to download something and immediately get something to work and see results is hugely rewarding; on the other hand, having to research, find, compare & contrast available third-party modules (especially for new-comers) can be extremely discouraging.

It might make sense to have CPython's release cadence decoupled from the Standard Library's release cadence.  That is, maybe they should be separate downloads.

On the other hand, it's nice to get it all in one go. Perhaps the installer could gain controls to download optional parts of the standard library or update those parts that have been installed.

On Mar 27, 2022, at 11:52, Dan Stromberg <drsalists@gmail.com> wrote:

It might make sense to have CPython's release cadence decoupled from the Standard Library's release cadence. That is, maybe they should be separate downloads.

While I don’t underestimate the work and complexity, we can do both. I.e. separate the stdlib development cycle from the interpreter (for all but a handful of required packages perhaps). We could still distribute “sumo” releases which include all the batteries, but develop and maintain them outside the cpython repo, and even release them separately on PyPI. It’s *possible* but I don’t know if it’s *practical*. -Barry

Barry writes (in part):

We could still distribute “sumo” releases which include all the batteries, but develop and maintain them outside the cpython repo, and even release them separately on PyPI. It’s *possible* but I don’t know if it’s *practical*.

to which Stephen responds (in part):

[Emacs really didn't change much in 20 years.] .... [I]n Python, every new release makes the Mailman crew want to stop supporting all previous releases of Python because there's some feature that can't be emulated that we really love: genexps or async or walrus operator or ....

It seems to me that what we have is the possibility that, say, package P migrates to PyPI in concert with the release of Python version X where maintenance can be picked up by the broader Python community. Whether or not it would proceed to track ongoing changes to the language isn't clear, and it's not obvious to me that less effort would be required other than perhaps by core devs (though some of them would likely pitch in to maintain packages with which they are currently involved). If you go with the "discard the batteries" approach, I think it would at least be worthwhile distributing a requirements.txt file ("sumo.txt" or "batteries.txt"?) which would tell someone installing Python how to reclaim the batteries of their Python youth, and for other Python implementations to track a set of packages which would make them plausibly compatible with CPython. CI could still rely on that to provide as much test coverage as current today (I think). If nothing else, it might alert the core devs to the potential for breakage of presumably widely used packages by changes to CPython. What happens to P when Python version Y grows a new syntactic feature? Do P's maintainers fork to both continue feature growth as well as syntactic modernization? If something like batteries.txt is created to tie a slimmed-down CPython distribution to the batteries it once contained, would the core group exert any control over unit testing, documentation, package variations and such? Just thinking out loud... Skip

What happens when the new maintainer puts malware in the next release of a package in sumo.txt? Will core devs be blamed for listing it? As a user, how do I determine if I can trust the packages there? (This is easily the hardest part of finding and installing a package from PyPI, though many people seem to skip it.)

I will point out that you quoted my entire post except for the most important line, the one which reads:

Just thinking out loud...

I am still thinking out loud, so keep that in mind. (Perhaps this belongs on one of the two ideas groups, but this is where things started, so I will continue here for now.) Consider a hypothetical situation. - Suppose Python v. X is the last version delivered with batteries, so v. Y lacks a json module (assuming that's not determined to be so important that it must be delivered as part of the CPython installer proper) . - Downstream, some Linux distro realizes their users still want batteries (as do their Python admin tools), so recreates a Python package with them. Their package of v. Y *does* include (at least) the json battery. - One of the maintainers of the now externally maintained json battery takes it upon themselves to protest an incursion into East WhatsItStan by West WhatsItStan in the worst manner possible and inserts code to scrub disks of all West WhatsItStan-ian homed computers (as the primary maintainer of some widely installed (Node? JS?) package package did). - Some poor West WhatsItStan-ian upgrades their Linux distro, only to find when they next run their favorite Python application that their disk drive is wiped clean. Note that I haven't postulated the existence of a sumo.txt file. Whether the CPython distribution contains batteries or not, someone will recreate the batteries, if not in toto, at least piecemeal (some application or package depends on json and will blindly pull it in). Who gets the blame here? The core Python devs (because json "always came with Python")? The maintainers of the Linux distro for recreating the batteries but not having any West WhatsItStanian-based testers? The new maintainers of the now external (and corrupted) json package? Maybe the several levels of indirection would serve to insulate the Python devs, but maybe not.

If they do exert control, why not keep it in stdlib?

Perhaps that is the best route. Someone here (Stéfane Fermigier, it seems) questioned whether batteries included are such a good idea. It's quite possible that enough core devs disagree with that sentiment that the batteries will stay (and perhaps grow in number, though more slowly than in the past). That said, if enough devs agree with Stéfane, then what's the best route forward? Discard all batteries and let the chips fall where they may? Extract the dead batteries (and presumably their docs and test bits) into separate github.com/python repositories? Ask others to extract them into their own non-github.com/python repositories? One of the common reasons old platform support is dropped from CPython (OS/2 or AmigaOS anyone?) is that the maintenance load for the core devs is too high relative to the community benefit derived from supporting small minority platforms. The unlucky modules named by PEP 594 are about to suffer the same fate. (I'm not arguing that they shouldn't be deleted.) Once PEP 594 has been implemented, all the low-hanging fruit will have been picked. At that point, it's keep everything or keep (almost) nothing. I think that will largely depend on the willingness of the core devs to keep maintaining modules they may well never use in their own work (like getopt and wave). One thing I think is obvious. If you remove all the batteries not deemed absolutely essential to build and test CPython, I think you have to somehow symbolically say, "these correspond to the various batteries which used to be in the CPython distribution." Maybe it's text in a README file, a new PEP, a sumo.txt file, or a Linux distro's packaging. Skip P.S. Personally, I was never fond of shutil or the logging package (as two examples — though for different reasons). We might have something better in both realms by now if they hadn't been added to the stdlib long ago. This ability of presence in the stdlib to forestall further development might well be the strongest argument to remove most batteries. On the other hand, we seem to have had little trouble cycling through several completely different regular expression modules. Maybe I'm just imagining a barrier where none exists.

On 2/6/2022 4:44 PM, Christian Heimes wrote:

If I had the power and time, then I would replace urllib with a simpler, reduced HTTP client that uses platform's HTTP library under the hood (WinHTTP on Windows, NSURLSession (?) on macOS, Web API for Emscripten, maybe curl on Linux/BSD). For non-trivial HTTP requests, httpx or aiohttp are much better suited than urllib.

I'm +1 on this, though I think it would have to be in place before the "two releases until removal" kicked in for urllib.request. The stdlib can't get by without at least the basic functionality of curl built in natively. But we can do this on most platforms without vendoring OpenSSL, which is a HUGE win. Then our default behaviour could correctly use proxies (including auto-config), CA certificate bundles, integrated authentication, and other OS features that are currently ignored by our core. Chances are we could keep simple urlopen() calls in place, and use the deprecation as a "potential change of behaviour" without necessarily having to break the API. I'm yet to come across a case where making a trivial urlopen() call _better_ would break things (the cases I've seen that would break are things like "using an OpenSSL environment variable to configure something that I wish had been automatic"). The nature of network/internet access is that we have to break things periodically anyway, because all the code that was written over the last 30+ years is eventually going to be found to be exploitable. I'd be quite happy to say "Python gives you what your OS gives you; update the OS for fixes". Cheers, Steve

... and there are also plenty examples out there of using http.server as a quick HTTP server for trying out new things, testing and teaching. FWIW: I find this discussion a bit strange. Python's stdlib is supposed to provide basic tooling for a breadth of use cases, with emphasis on "basic" and "breadth". urllib is such a basic library and covers one of the main use cases for Python we have. It would be pretty much beside the point of the stdlib to remove such basic functionality and make Python less attractive and less useful for beginners. Anything more complex can be dealt with on PyPI, as it is already happening. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 08 2022)

...

...

Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/

::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/

On Wed, 9 Feb 2022 at 04:50, Christopher Barker <pythonchb@gmail.com> wrote:

So my thoughts:

Rather than deprecate urllib, we refactor it a bit (and maybe deprecate parts of it), so that it:

1) contains the core building blocks: e.g. urllib.parse with which to build "better" libraries,

2) make the "easy stuff easy" -- e.g. a basic http: request. - For instance, I'd like to see an API that's kind of "requests-lite"

And much better docs explain when you should use it, and when you might want to look for another library (even if it's the stdlib http.client)

This sounds like a decent plan. I'd like to add my voice to the appeal to keep urllib.parse; in fact, of all the places where I've used anything from urllib, only two of them are anything other than urllib.parse. (One is an old script that I specifically wanted to be as shareable as possible, so I restricted it to the stdlib; the other catches urllib.error.URLError thrown by a third-party library.) If there are security issues with urllib.request, I wouldn't shed many tears about its deprecation. A "requests-lite" module would certainly be handy, but it's hard to judge how much wants to be in the stdlib and how much can be pushed off to a pip-installable module:

the first thing I do for beginners is to point them to requests, as it's easier to use :-)

Exactly my thoughts :) But a very very simple HTTP/HTTPS GET request endpoint would be a great bootstrapping aid. Consider: with nothing but the stdlib, you could fetch a file from some server, unzip it (zipfile module), and import it. For building dirt-simple install scripts, this kind of thing is really REALLY handy, and I'd rather not have to use plain TCP sockets to do it :) ChrisA

Why do you think the stdlib *must *provide an example implementation for this specific scenario? Is there something unique to HTTP request handling that you feel is important to demonstrate?

*must* is too strong, but I would use a very strong *should*. I think the stdlib should provide simple source-included examples of most things. I think the case is even stronger when it is: (1) a fairly simple protocol (such as version 1 of http was) -- QUIC wouldn't count for a simple demonstration. (2) something new users are likely to find motivating. Short of "here is a way to do IO", and maybe "write a simple game", "get something from the web" is probably the most obvious case. (3) something where bootstrapping might be an issue (network protocols, particularly web downloads). Network access is not an always-available resource. Even when it is available, there is sometimes a barrier between "available in python" and "I could read it on my phone, but can't get it open in python". (4) something where a a beginner is likely to be overwhelmed by choices if we just say "use a 3rd party module". (5) something with a backwards-compatibility story in the stdlib already. As a side note, are there concerns about urllib.robotparser being broken or obsolete, or was that part of the deprecation proposal just contagion from urllib.request? -jJ