Proposing additions to the standard library
I'm interested in proposing several additions to the Python standard library, and I would like more information on the procedure for doing so. Are all additions done via a PEP? If not what is the procedure. If so, I've read that the first step was to email this board and get feedback. I have a library called `ubelt` that contains several tools that I think might be worthy of adding to the standard library. Here's my bullet point pitch: - Python is batteries included. Ubelt contains extra batteries. function are extra batteries. - Most function in ubelt are fast. All 222 tests takes 7.33 seconds. - Ubelt has 100% test coverage (sans `# nocover` locations). - I'm only championing a subset of the functions in ubelt. There are certainly functions in there that do not belong in the standard library. - I have a Jupyter notebook that give a demo of some select functions (not necessarily the same as the ones proposed here): https://github.com/Erotemic/ubelt/blob/master/docs/notebooks/Ubelt%20Demo.ip... - I do have documentation (mostly in docstrings) and in the docs folder, but I've been having trouble auto-updating read-the-docs. Here is the link anyway: https://ubelt.readthedocs.io/en/latest/ Here is a tentative list of interesting functions. Hopefully the names are descriptive (if not, see docstrings: https://github.com/Erotemic/ubelt) ub.cmd ub.compressuser ub.group_items ub.dict_hist ub.find_duplicates ub.AutoDict ub.import_module_from_path ub.import_module_from_name ub.modname_to_modpath, ub.modpath_to_modname ub.ProgIter ub.ensuredir ub.expandpath almost everything in util_list: allsame, argmax, argmin, argsort, argunique, chunks, flatten, iter_window, take, unique These functions might be worth modifying into dictionary methods: ub.dict_subset ub.dict_take ub.map_vals ub.map_keys ub.Timerit ub.Timer Because I built the library, I tend to like all the functions. Its difficult to decide if they are stdlib worthy, so there might be some false positives / negatives. I'm on the fence about: CacheStamp, Cacher, NoParam, argflag, argval, dzip, delete, hash_data, hash_file, memoize, memoize_method, NiceRepr, augpath, userhome, ensure_app_cache_dir, ensure_app_resource_dir, find_exe, find_path, get_app_cache_dir, get_app_resource_dir, platform_cache_dir, platform_resource_dir, CaptureStdout, codeblock, ensure_unicode, hzcat, indent, OrderedSet Its my hope that some of these are actually useful. Let me know any of the following: what you think, if there are any questions, if something else needs to be done, or what the next steps are. -- -Jon
On Sat, Nov 10, 2018 at 08:36:52PM -0500, Jonathan Crall wrote:
Not necessarily. Small, obvious enhancements can go straight to the bug tracker. The tricky part is deciding what is "obvious". Sometimes there's a good, useful function than doesn't get added because there's no reasonable place to put it. For example, a "flatten" function has been talked about since Python 1.x days, and we still don't have a standard solution for it, because (1) it isn't clear *precisely* what it should do, and (2) it isn't clear where it should go. Given that once something gets added to the std lib, it is hard to remove it or even rename it, its better to be conservative about adding things and leave it to third party libraries to cover the gaps.
If not what is the procedure. If so, I've read that the first step was to email this board and get feedback.
That's a good idea. If the enhancement request isn't both small and obvious, or is the least bit controversial, you'll usually be sent back here.
I have a library called `ubelt` that contains several tools that I think might be worthy of adding to the standard library.
Generally speaking, we don't typically add grab-bags of random utility functions. There is no "utilities" or "toolbox" module in the std lib. [...]
Here is a tentative list of interesting functions. Hopefully the names are descriptive (if not, see docstrings: https://github.com/Erotemic/ubelt)
Sorry, some of these aren't descriptive enough, and if you're trying to make a pitch for these features, you ought to give at least a one-sentence explanation of them here in the email. You will lose half your audience as soon as you ask them to click through to a link, and even if they do, that risks splitting the discussion across two places. My advice is to collate the functions you want to add into groups of related functionality, find the class or module in the std lib where you think they belong, and begin a new thread for each group. E.g. "New dict methods", "New importlib functions". -- Steve
@Steve, this is just the sort of feedback I was looking for. Small and conservative additions make sense. I definitely think that some functions do fit into existing stdlib modules. For instance, AutoDict might go in collections. Sorry, some of these aren't descriptive enough, and if you're trying to
make a pitch for these features.
... My advice is to collate the functions you want to add into groups of
related functionality.
Makes sense. I figured that my original list had too may entries to do that for, or else the email would explode. Separating each small group into its own thread will allow me to describe the specific function without writing a novel. Sometimes there's a good, useful function than doesn't get added because
The flatten example is good to know about. Is there a link to this discussion or a summary of it? I would think flatten could go in itertools, but clearly there must some reason why its not there. I imagine the duplication with it.chain.from_iter + "There should be one-- and preferably only one --obvious way to do it."? As for what it should do, I'm guessing the controversy was over flattening one level vs all levels? That makes sense and is good to know. I guess I won't pick `flatten` as one of my first functions to pick for a writeup. On a similar note, do you (or anyone else) have an intuition for which of these functions --- judging by name only (so you don't have to click any links) --- might be the least controversial? I'm not very good at judging controversy, which is one of the main reasons for this initial email. Maybe `expandpath` to os.path? Or perhaps start with ub.modname_to_modpath and ub.modpath_to_modname to importlib? Maybe some of the dict-methods? Perhaps I'm overestimating the clear usefulness of any of these functions to the stdlib? On Sat, Nov 10, 2018 at 9:14 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- -Jon
On Sat, Nov 10, 2018 at 6:56 PM Jonathan Crall <erotemic@gmail.com> wrote:
https://docs.python.org/3/library/itertools.html#itertools-recipes There's an example of ``flatten`` in the itertools recipes.
Not to derail the conversation, but I've always been curious why the itertools recipes are recipes and not ready-made goods (pre-baked?) that I can just consume. They're great examples to draw from, but that shouldn't preclude them from also being in the stdlib. On Mon, Nov 12, 2018 at 7:41 PM Michael Selik <mike@selik.org> wrote:
My intuition has always been that the recipes, taking 'flatten' as an excellent example, solve problems in a specific way that is not generally considered to be the "right" way. For example, should 'flatten' perform one-level flattening or deep recursive flattening? Should it handle strings as single entities, or should it treat them as iterables? What about byte strings, should they be treated differently than strings or the same? I could go on, but you probably get the point... On Mon, Nov 12, 2018 at 11:50 AM Nick Timkovich <prometheus235@gmail.com> wrote:
Are you aware of https://boltons.readthedocs.io/ (whose motto is "Functionality that should be in the standard library.") ? Or similar endeavours such as: - https://pypi.org/project/auxlib/ - https://pypi.org/project/omakase/ - (And probably many others on PyPI with similar descriptions such as "a library of stuff I'm using / we're using at company X for all my / our project(s)...") Or the functional libraries listed here: https://github.com/sfermigier/awesome-functional-python/blob/master/README.m... => IMHO there is room for a "semi-standard" library, stuff that's not included by default and has a release lifecycle more active that Python itself, but that can be considered the standard by a large group of users. Similar ideas can be found for instance in Java with Apache Commons ( https://commons.apache.org/ -> "an Apache project focused on all aspects of reusable Java components."). One could argue, though, that the Java standard library is much less developed than the Python standard library, so it's much easier to justify the existence of Apache Commons than a similar Python project. There is also the question of the porosity between such a project and the stdlib, which is the essence of the original question by the OP. Another interesting issue is the granularity of such a project. I sometimes, and somewhat foolishly, make libraries such as toolz or boltons a dependency of my projects, for just one or two function calls from my code. Regards, S. On Sun, Nov 11, 2018 at 2:37 AM Jonathan Crall <erotemic@gmail.com> wrote:
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - https://systematic-paris-region.org/fr/groupe-thematique-logiciel-libre/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
@Stéfane Bolton looks really neat! I'll take a look. Some of my stuff may fit better as a PR for this library. Also I don't think its foolish to depend on a package for one function, given that that (a) that function is really useful or (b) the size of the dependency itself is small. Given my initial impressions of boltons, I would guess that it doesn't have a large download size of runtime impact. Although if this case keeps reoccurring with a particular function, then perhaps that function might improve the stdlib? After really reviewing my stuff, I think a few functions I have would make the stdlib better, but its probably only a small few. I bet there are things in bolton that would improve the stdlib as well, but I do agree that "semi-standard" libraries (e.g. numpy / scipy / what-I-hope-ubelt-to-be) might be better place for these functions that would otherwise cause costly-clutter in the stdlib. On Tue, Nov 13, 2018 at 2:05 AM Stéfane Fermigier <sf@fermigier.com> wrote:
-- -Jon
On Sat, Nov 10, 2018 at 08:36:52PM -0500, Jonathan Crall wrote:
Not necessarily. Small, obvious enhancements can go straight to the bug tracker. The tricky part is deciding what is "obvious". Sometimes there's a good, useful function than doesn't get added because there's no reasonable place to put it. For example, a "flatten" function has been talked about since Python 1.x days, and we still don't have a standard solution for it, because (1) it isn't clear *precisely* what it should do, and (2) it isn't clear where it should go. Given that once something gets added to the std lib, it is hard to remove it or even rename it, its better to be conservative about adding things and leave it to third party libraries to cover the gaps.
If not what is the procedure. If so, I've read that the first step was to email this board and get feedback.
That's a good idea. If the enhancement request isn't both small and obvious, or is the least bit controversial, you'll usually be sent back here.
I have a library called `ubelt` that contains several tools that I think might be worthy of adding to the standard library.
Generally speaking, we don't typically add grab-bags of random utility functions. There is no "utilities" or "toolbox" module in the std lib. [...]
Here is a tentative list of interesting functions. Hopefully the names are descriptive (if not, see docstrings: https://github.com/Erotemic/ubelt)
Sorry, some of these aren't descriptive enough, and if you're trying to make a pitch for these features, you ought to give at least a one-sentence explanation of them here in the email. You will lose half your audience as soon as you ask them to click through to a link, and even if they do, that risks splitting the discussion across two places. My advice is to collate the functions you want to add into groups of related functionality, find the class or module in the std lib where you think they belong, and begin a new thread for each group. E.g. "New dict methods", "New importlib functions". -- Steve
@Steve, this is just the sort of feedback I was looking for. Small and conservative additions make sense. I definitely think that some functions do fit into existing stdlib modules. For instance, AutoDict might go in collections. Sorry, some of these aren't descriptive enough, and if you're trying to
make a pitch for these features.
... My advice is to collate the functions you want to add into groups of
related functionality.
Makes sense. I figured that my original list had too may entries to do that for, or else the email would explode. Separating each small group into its own thread will allow me to describe the specific function without writing a novel. Sometimes there's a good, useful function than doesn't get added because
The flatten example is good to know about. Is there a link to this discussion or a summary of it? I would think flatten could go in itertools, but clearly there must some reason why its not there. I imagine the duplication with it.chain.from_iter + "There should be one-- and preferably only one --obvious way to do it."? As for what it should do, I'm guessing the controversy was over flattening one level vs all levels? That makes sense and is good to know. I guess I won't pick `flatten` as one of my first functions to pick for a writeup. On a similar note, do you (or anyone else) have an intuition for which of these functions --- judging by name only (so you don't have to click any links) --- might be the least controversial? I'm not very good at judging controversy, which is one of the main reasons for this initial email. Maybe `expandpath` to os.path? Or perhaps start with ub.modname_to_modpath and ub.modpath_to_modname to importlib? Maybe some of the dict-methods? Perhaps I'm overestimating the clear usefulness of any of these functions to the stdlib? On Sat, Nov 10, 2018 at 9:14 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- -Jon
On Sat, Nov 10, 2018 at 6:56 PM Jonathan Crall <erotemic@gmail.com> wrote:
https://docs.python.org/3/library/itertools.html#itertools-recipes There's an example of ``flatten`` in the itertools recipes.
Not to derail the conversation, but I've always been curious why the itertools recipes are recipes and not ready-made goods (pre-baked?) that I can just consume. They're great examples to draw from, but that shouldn't preclude them from also being in the stdlib. On Mon, Nov 12, 2018 at 7:41 PM Michael Selik <mike@selik.org> wrote:
My intuition has always been that the recipes, taking 'flatten' as an excellent example, solve problems in a specific way that is not generally considered to be the "right" way. For example, should 'flatten' perform one-level flattening or deep recursive flattening? Should it handle strings as single entities, or should it treat them as iterables? What about byte strings, should they be treated differently than strings or the same? I could go on, but you probably get the point... On Mon, Nov 12, 2018 at 11:50 AM Nick Timkovich <prometheus235@gmail.com> wrote:
Are you aware of https://boltons.readthedocs.io/ (whose motto is "Functionality that should be in the standard library.") ? Or similar endeavours such as: - https://pypi.org/project/auxlib/ - https://pypi.org/project/omakase/ - (And probably many others on PyPI with similar descriptions such as "a library of stuff I'm using / we're using at company X for all my / our project(s)...") Or the functional libraries listed here: https://github.com/sfermigier/awesome-functional-python/blob/master/README.m... => IMHO there is room for a "semi-standard" library, stuff that's not included by default and has a release lifecycle more active that Python itself, but that can be considered the standard by a large group of users. Similar ideas can be found for instance in Java with Apache Commons ( https://commons.apache.org/ -> "an Apache project focused on all aspects of reusable Java components."). One could argue, though, that the Java standard library is much less developed than the Python standard library, so it's much easier to justify the existence of Apache Commons than a similar Python project. There is also the question of the porosity between such a project and the stdlib, which is the essence of the original question by the OP. Another interesting issue is the granularity of such a project. I sometimes, and somewhat foolishly, make libraries such as toolz or boltons a dependency of my projects, for just one or two function calls from my code. Regards, S. On Sun, Nov 11, 2018 at 2:37 AM Jonathan Crall <erotemic@gmail.com> wrote:
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - https://systematic-paris-region.org/fr/groupe-thematique-logiciel-libre/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/
@Stéfane Bolton looks really neat! I'll take a look. Some of my stuff may fit better as a PR for this library. Also I don't think its foolish to depend on a package for one function, given that that (a) that function is really useful or (b) the size of the dependency itself is small. Given my initial impressions of boltons, I would guess that it doesn't have a large download size of runtime impact. Although if this case keeps reoccurring with a particular function, then perhaps that function might improve the stdlib? After really reviewing my stuff, I think a few functions I have would make the stdlib better, but its probably only a small few. I bet there are things in bolton that would improve the stdlib as well, but I do agree that "semi-standard" libraries (e.g. numpy / scipy / what-I-hope-ubelt-to-be) might be better place for these functions that would otherwise cause costly-clutter in the stdlib. On Tue, Nov 13, 2018 at 2:05 AM Stéfane Fermigier <sf@fermigier.com> wrote:
-- -Jon
participants (6)
-
Eric Fahlgren -
Jonathan Crall -
Michael Selik -
Nick Timkovich -
Steven D'Aprano -
Stéfane Fermigier