itertools recipes: why not add them to the stdlib *somewhere*?
I just now ran into (since learning about the recipes) having use for a recipe (then I found out that I can't use it anyway, but that's a different thing) So... I consider it very strange and annoying that there's this code. Ready to use. But you have to copy-and-paste it into your code *somewhere* instead of easily importing it. As to the "it makes using Python harder to learn", I beg to differ: an addition to the *language* (like, say, metaclasses) *can* make it more complicated. But additions to the stdlib? What about "batteries included"? Not a motto anymore (I heard rumors, so maybe that's the case) Putting these in some official(!) iterutils (or name it what you want) package would be a solution for so many people. Yes, not everyone needs grouper's current functionality. But for the many who do, it'd be there already. And someone thought those recipes are useful, didn't you? So why not make them more easily available? And then there's Alex Martelli at http://stackoverflow.com/a/1641242/238406, and I agree with him wholeheartedly. Bye, J
On Jul 7, 4:14 am, Jürgen A. Erhard <jae+pyt...@jaerhard.com> wrote:
So... I consider it very strange and annoying that there's this code. Ready to use. But you have to copy-and-paste it into your code *somewhere* instead of easily importing it.
I find it strange and annoying when people expect the standard library to contain _everything_. What you gain in the convenience of readily imported code imposes a cost on the ongoing maintenance and support of the standard lib. Are you volunteering to write the PEP for its inclusion, provide the implementation and support it until the end of time?
As to the "it makes using Python harder to learn", I beg to differ: an addition to the *language* (like, say, metaclasses) *can* make it more complicated. But additions to the stdlib? What about "batteries included"? Not a motto anymore (I heard rumors, so maybe that's the case)
Every single function added to the library makes the library broader and thus more difficult to retain in memory. It _is_ possible to present a case without snide allusions based on hearsay: http://www.python.org/about/
Putting these in some official(!) iterutils (or name it what you want) package would be a solution for so many people. Yes, not everyone needs grouper's current functionality. But for the many who do, it'd be there already.
So in the extremely likely outcome that a piece of functionality has multiple implementations with different interpretations of edge cases, who gets to decide which one is "more standard" than the other? If such a decision has to be made, then it's not really "standard".
And someone thought those recipes are useful, didn't you? So why not make them more easily available?
Because not everything needs to be in the standard library. Because searching somewhere like Activestate's cookbook and copying the implementations you find useful really isn't an onerous task. Because if *you* feel there's an obvious lack in the stdlib then roll your own package that covers it and put it up on PyPI. If it becomes insanely popular, then you have a lot more leverage for insisting that something would be beneficial to be included other than gut feeling & convenience.
On Mon, Jul 9, 2012 at 2:16 PM, alex23 <wuwei23@gmail.com> wrote:
So in the extremely likely outcome that a piece of functionality has multiple implementations with different interpretations of edge cases, who gets to decide which one is "more standard" than the other? If such a decision has to be made, then it's not really "standard".
This is often what lies at the heart of "not every 3 line function needs to be <a builtin>/<in the standard library>". It's very easy to hit a point of diminishing returns where the number of possible alternatives mean that attempting to create an abstraction layer ends up creating a UI that is *more complicated* than just writing your own utility function that does exactly what you want. Grouping is one such API (fixed length? Variable length with a delimiter? Pad last group? Drop last group? Error early? Accept arbitrary iterables? Accept sequences only? Support overlapping windows?). Recursive descent into arbitrary collections is another (Error on reference loops? Treat strings/bytes/bytearray as atomic types? Treat any implementor of the buffer interface as an atomic type? Support only hashable objects?). There are a few key reasons why things end up in the standard library: 1. They're closely coupled to the language definition and implementation and should be updated in sync with it (e.g. sys, imp, gc, importlib, dis, inspect) 2. We want to use them in other parts of the standard library or its test suite (e.g. collections, many of the unittest enhancements) 3. They solve a common problem that is otherwise prone to being handled with incomplete solutions that lead to incorrect code (e.g. ipaddress instead of regex based recipes for processing IP addresses) 4. It's an old module that would probably be left to PyPI these days, but was added in earlier times when stdlib inclusion criteria were less strict (but isn't broken so there's no harm in keeping it around). Sometimes a problem is sufficiently common, and has few enough variations, that it's worth creating and providing a standard version. That's the general aim of the itertools module. Other times, the core problem is difficult enough that it's worth providing a standard solution, even if it does mean including a vast array of configuration options (e.g. subprocess. Popen). However, sometimes, the correct answer to "Hey, this is a really common pattern" is not "We should provide an API that uses that pattern internally" but "we should document this pattern, so people know it's a common idiom and can tailor it to their specific use case and preferences". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
It's very easy to hit a point of diminishing returns where the number of possible alternatives mean that attempting to create an abstraction layer ends up creating a UI that is *more complicated* than just writing your own utility function that does exactly what you want. [...]
+1 on everything Nick says here. One additional comment:
However, sometimes, the correct answer to "Hey, this is a really common pattern" is not "We should provide an API that uses that pattern internally" but "we should document this pattern, so people know it's a common idiom and can tailor it to their specific use case and preferences".
Well, maybe, but documenting software patterns are not necessarily "our" job (whoever "our" is -- the Python devs, or python.org, or the PSI, or whoever people think is responsible). Python has not been a one-man project for many, many years. There is a rich Python ecosystem now, not just Guido or the Python devs, but StackOverflow, comp.lang.python, dozens of books, more bloggers than you can poke a stick at, #python, etc etc etc. It's not "our" responsibility to teach every Python programmer every conceivable pattern. At some point we can just draw a line and say "Yeah, that's a useful pattern, but we're not going to document it. Somebody else can do it." Even if that line is somewhat arbitrary. -- Steven
On Jul 9, 2012 6:17 AM, "alex23" <wuwei23@gmail.com> wrote:
On Jul 7, 4:14 am, Jürgen A. Erhard <jae+pyt...@jaerhard.com> wrote:
So... I consider it very strange and annoying that there's this code. Ready to use. But you have to copy-and-paste it into your code *somewhere* instead of easily importing it.
I find it strange and annoying when people expect the standard library to contain _everything_. What you gain in the convenience of readily imported code imposes a cost on the ongoing maintenance and support of the standard lib. Are you volunteering to write the PEP for its inclusion, provide the implementation and support it until the end of time?
While I agree that these recipes are not suitable for inclusion in stdlib (for reasons made elsewhere), I don't agree with your (somewhat aggressive) reasoning. The implementation has already been provided, and is being maintained in the form of documentation. Simply copying the recipes into a module is hardly an onerous task. And fairly often people mailing to this list *are* volunteering to write a PEP, provide an implementation, etc.
As to the "it makes using Python harder to learn", I beg to differ: an addition to the *language* (like, say, metaclasses) *can* make it more complicated. But additions to the stdlib? What about "batteries included"? Not a motto anymore (I heard rumors, so maybe that's the case)
Every single function added to the library makes the library broader and thus more difficult to retain in memory.
It _is_ possible to present a case without snide allusions based on hearsay: http://www.python.org/about/
Putting these in some official(!) iterutils (or name it what you want) package would be a solution for so many people. Yes, not everyone needs grouper's current functionality. But for the many who do, it'd be there already.
So in the extremely likely outcome that a piece of functionality has multiple implementations with different interpretations of edge cases, who gets to decide which one is "more standard" than the other? If such a decision has to be made, then it's not really "standard".
And someone thought those recipes are useful, didn't you? So why not make them more easily available?
Because not everything needs to be in the standard library.
Because searching somewhere like Activestate's cookbook and copying the implementations you find useful really isn't an onerous task.
Because if *you* feel there's an obvious lack in the stdlib then roll your own package that covers it and put it up on PyPI. If it becomes insanely popular, then you have a lot more leverage for insisting that something would be beneficial to be included other than gut feeling & convenience. _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On Jul 9, 2012, at 8:18 AM, David Townshend wrote:
. Simply copying the recipes into a module is hardly an onerous task.
In fact, it has already been done :-) http://pypi.python.org/pypi?%3Aaction=search&term=itertools&submit=search Raymond
On 7/9/2012 12:18 PM, Raymond Hettinger wrote:
On Jul 9, 2012, at 8:18 AM, David Townshend wrote:
. Simply copying the recipes into a module is hardly an onerous task.
In fact, it has already been done :-)
Multiple times. Possible addition to the doc in 9.1.2 after first sentence: These and similar recipes have been incorporated in various third-party modules. See ...
http://pypi.python.org/pypi?%3Aaction=search&term=itertools&submit=search
-- Terry Jan Reedy
On Mon, Jul 9, 2012 at 12:16 AM, alex23 <wuwei23@gmail.com> wrote:
I find it strange and annoying when people expect the standard library to contain _everything_. What you gain in the convenience of readily imported code imposes a cost on the ongoing maintenance and support of the standard lib. Are you volunteering to write the PEP for its inclusion, provide the implementation and support it until the end of time?
Are you saying the recipes are unsupported? -- Devin
On Tue, Jul 10, 2012 at 6:12 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Are you saying the recipes are unsupported?
This is getting ridiculous. I'm not sure what is so controversial about not wanting to scrape up every piece of crap code off the web that some lazy coder can't be bothered to implement contextually and shove them into the _standard library_. Are _you_ seriously saying that every one of those random bits of code _is_ supported? That they will all be kept up to date as the language evolves? That they won't turn into another module that people complain is "stale"? Or another "why tkinter and not <x>" argument? Every inclusion to the library adds overhead to finding and using elements of the library.
On Tue, Jul 10, 2012 at 08:28:41AM +1000, wu wei <wuwei23@gmail.com> wrote:
Every inclusion to the library adds overhead to finding and using elements of the library.
And supporting it. It's much harder to remove bits from stdlib than to add to it because stdlib have to maintain backward compatibility. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Mon, Jul 9, 2012 at 3:36 PM, Oleg Broytman <phd@phdru.name> wrote:
On Tue, Jul 10, 2012 at 08:28:41AM +1000, wu wei <wuwei23@gmail.com> wrote:
Every inclusion to the library adds overhead to finding and using elements of the library.
And supporting it. It's much harder to remove bits from stdlib than to add to it because stdlib have to maintain backward compatibility.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN. _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
It is a good point that those recipes because they are in the official docs are already requiring support and maintenance. However, as somebody above mentioned, there are many possible variations on those recipes and so it makes a lot more sense to tell people to implement their own version of grouper that will match their specific needs rather than create an infinitely configurable version of grouper in the stdlib. Furthermore, if the maintenance of those recipes ever becomes undesirable, it is a simple matter to strike them from the docs. The same cannot be said if they were inducted in the stdlib. Alexandre Zani
On Mon, Jul 9, 2012 at 6:28 PM, wu wei <wuwei23@gmail.com> wrote:
On Tue, Jul 10, 2012 at 6:12 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Are you saying the recipes are unsupported?
This is getting ridiculous. I'm not sure what is so controversial about not wanting to scrape up every piece of crap code off the web that some lazy
This is code from the official documentation.
coder can't be bothered to implement contextually and shove them into the _standard library_. Are _you_ seriously saying that every one of those random bits of code _is_ supported?
No, I am suggesting that example code that exists in the documentation is supported. I think you are confused. -- Devin
On 7/9/2012 6:45 PM, Devin Jeanpierre wrote:
No, I am suggesting that example code that exists in the documentation is supported.
Well, in the sense that we try to make sure that they work as advertised *with the example data*, yes. However, that is the end of the support. Examples are just examples, put there to illustrate a didactic point. They are often simplified to serve the specific purpose. They are not intended to be production code, and they are not supported as that. Error handling is eliminated for clarity.# The api has not necessarily been thought through very well and is subject to change.# The multiple examples are not particularly meant to work together. #Whereas, for example, the new in 3.3 ipaddress module, while based on existing 3rd party code, has had its api reexamined to make sure it well defined, documented, and something we will (likely) stick with. Nick has added a lot of error checking code with messages that try to say what is wrong rather than just 'you goofed'. And oh yes, the test suite has been expanded and reviewed with an aim to 100% coverage. -- Terry Jan Reedy
On Mon, Jul 9, 2012 at 9:52 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Well, in the sense that we try to make sure that they work as advertised *with the example data*, yes. However, that is the end of the support. Examples are just examples, put there to illustrate a didactic point. They are often simplified to serve the specific purpose. They are not intended to be production code, and they are not supported as that.
I've never assumed that (obviously, I guess), and that isn't stated in the docs. If that's the level of support of those recipes, this should be made clear inside the itertools documentation, because they _are_ used in peoples' code. For example, see the third party module linked earlier in the thread. On the contrary, the docs state the recipes are for creating an "extended toolset", implying that they can be used freely the same as the rest of the code. -- Devin
Devin Jeanpierre writes:
I've never assumed that (obviously, I guess), and that isn't stated in the docs. If that's the level of support of those recipes, this should be made clear inside the itertools documentation,
It already is clear. The very name "recipe" indicates that it may not be to everyone's taste, that it's not carefully designed to handle corner cases, and that people should review the code for their own use. The (very sparse) documentation associated with the recipes, which is clearly oriented to advising you how you can (a) produce similar code for different functions yourself and (b) make it better further emphasizes what the name indicates. Of course, all that may not be obvious to non-native speakers, but that's a completely different issue from the starting point of this thread, which was that these recipes belong in the stdlib rather than in its documentation. I have nothing to add to Terry's post explaining why that is not appropriate for these recipes "as is."
because they _are_ used in peoples' code.
Of course they are. That's what they're there for.
participants (12)
-
alex23 -
Alexandre Zani -
David Townshend -
Devin Jeanpierre -
Jürgen A. Erhard -
Nick Coghlan -
Oleg Broytman -
Raymond Hettinger -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy -
wu wei