The strange contortions of the "fast sum for lists" discussions got me wondering about whether it was possible to rehabilitate reduce with a less error-prone API. It was banished to functools in 3.0 because it was so frequently used incorrectly, but now its disfavour seems to be causing people to propose ridiculous things. The 2.x reduce is modelled on map and filter: it accepts the combinator as the first argument, and then the iterable, and finally an optional initial value. The most common error was failing to handle the empty iterable case sensibly by leaving out the initial value, so you got a TypeError instead of returning a result. So, what if we instead added a new alternative API based on Haskell's "fold" [1] where the initial value is *mandatory*: def fold(op, start, iterable): ... Efficiently merging a collection of iterables into a list would then just be: data = fold(operator.iadd, [], iterables) I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write: data = fold("+=", [], iterables) This could also be introduced as an alternative API in functools. (Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd) Cheers, Nick. [1] https://en.wikipedia.org/wiki/Fold_%28higher-order_function%29 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12 July 2013 18:01, Nick Coghlan <ncoghlan@gmail.com> wrote:
The strange contortions of the "fast sum for lists" discussions got me wondering about whether it was possible to rehabilitate reduce with a less error-prone API. It was banished to functools in 3.0 because it was so frequently used incorrectly, but now its disfavour seems to be causing people to propose ridiculous things.
The 2.x reduce is modelled on map and filter: it accepts the combinator as the first argument, and then the iterable, and finally an optional initial value. The most common error was failing to handle the empty iterable case sensibly by leaving out the initial value, so you got a TypeError instead of returning a result.
So, what if we instead added a new alternative API based on Haskell's "fold" [1] where the initial value is *mandatory*:
+1
def fold(op, start, iterable): ...
It would be nice to spell it fold(op, iterable, start), so that a trivial sed can migrate reduce using code to fold. Or perhaps we could just fix reduce?
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
This seems like an unusual thing in Python, but I can certainly see it's convenience. -Rob -- Robert Collins <rbtcollins@hp.com> Distinguished Technologist HP Cloud Services
On Fri, 12 Jul 2013 16:01:07 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
The strange contortions of the "fast sum for lists" discussions got me wondering about whether it was possible to rehabilitate reduce with a less error-prone API. It was banished to functools in 3.0 because it was so frequently used incorrectly, but now its disfavour seems to be causing people to propose ridiculous things.
I would disagree with this interpretation. reduce() wasn't "banished" (what a strange qualification!) because its API was "error-prone", but because the whole concept isn't very useful - and, indeed, little used - in a language like Python.
So, what if we instead added a new alternative API based on Haskell's "fold" [1] where the initial value is *mandatory*:
def fold(op, start, iterable): ...
So fold(op, start, iterable) is the same as reduce(op, iterable, start)? This sounds silly and useless. Regards Antoine.
On 12 July 2013 07:01, Nick Coghlan <ncoghlan@gmail.com> wrote:
The strange contortions of the "fast sum for lists" discussions got me wondering about whether it was possible to rehabilitate reduce with a less error-prone API. It was banished to functools in 3.0 because it was so frequently used incorrectly, but now its disfavour seems to be causing people to propose ridiculous things.
The 2.x reduce is modelled on map and filter: it accepts the combinator as the first argument, and then the iterable, and finally an optional initial value. The most common error was failing to handle the empty iterable case sensibly by leaving out the initial value, so you got a TypeError instead of returning a result.
So, what if we instead added a new alternative API based on Haskell's "fold" [1] where the initial value is *mandatory*:
def fold(op, start, iterable): ...
Efficiently merging a collection of iterables into a list would then just be:
data = fold(operator.iadd, [], iterables)
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
This could also be introduced as an alternative API in functools.
(Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd)
This sounds like a good idea to me (although by_symbol could well have a more catchy name) -- there are so many places where "lambda a, b: a + b" is just ugly. This could work itself into a lot of APIs. What would be the equivalent for "operator.pos", "operator.neg" and "operator.{get|set|del}item"? ...But then I start suggesting extensions like having operator.* auto curry: "operator.iadd(left=foo)(bar)" === "operator.iadd(foo, bar)" (currying should require separate keyword-only syntax, and those keywords always make a curried function). And uncurry -- we need that if we want this to work with map.
From: Nick Coghlan <ncoghlan@gmail.com> Sent: Thursday, July 11, 2013 11:01 PM
So, what if we instead added a new alternative API based on Haskell's "fold" [1] where the initial value is *mandatory*:
def fold(op, start, iterable): ...
Note that Haskell, and many other functional languages, actually have both functions: def fold(op, start, iterable): def fold1(op, iterable): And really, they're only separate functions because a language with strict types and automatic currying can't handle variable arguments. Meanwhile, there are an awful lot of people who just don't like reduce/fold in any situation. The quote "Inside every reduce is a loop trying to get out" appears quite frequently, on this list and elsewhere. And I don't think it's because it's easy to get the fold/fold1 distinction wrong, but because they consider any use of reduce unreadable. I think the idea is that folding only makes immediate sense if you're thinking of your data structures recursively instead of iteratively, which you usually aren't in Python. But I'm probably not the best one to characterize the objection, since I don't share it. (Of course there are cases where reduce _is_ unreadable, and the only reason people use it is because in Haskell or OCaml or Scheme the explicit loop would be _more_ unreadable, even though that isn't even remotely true in Python… but there are also cases where it makes sense to me.) One more thing: The name "fold" to me really implies there's going to be "foldr" function as well, in a way that "reduce" doesn't. But I could probably get over that—after all, right-folding isn't nearly as important for code with arrays or iterators the same way it is for recursive code with cons lists or lazy lists.
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
I like this idea, but only if it's added to other functions in the stdlib where it makes sense, and easy to add to new functions of your own.
This could also be introduced as an alternative API in functools.
(Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd)
And that answers the "easy to add to new functions" bit! Except a helper function might be nice, something like operator.get_op (but with a better name): def get_op(func): if callable(func): return func else: return by_symbol[func]
On 12 July 2013 09:03, Andrew Barnert <abarnert@yahoo.com> wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
I like this idea, but only if it's added to other functions in the stdlib where it makes sense, and easy to add to new functions of your own.
Is there a use-case for that last part? It strikes me as equivalent to messing with builtins, which is largely unliked.
Sent from a random iPhone On Jul 12, 2013, at 1:49, Joshua Landau <joshua@landau.ws> wrote:
On 12 July 2013 09:03, Andrew Barnert <abarnert@yahoo.com> wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
I like this idea, but only if it's added to other functions in the stdlib where it makes sense, and easy to add to new functions of your own.
Is there a use-case for that last part? It strikes me as equivalent to messing with builtins, which is largely unliked.
I think you missed the word "to" in "add to". I don't want to create functions that can be passed to fold as if they were operators, I want to create functions that can take operators the same way fold does. The former is akin to messing with builtins, which is bad; the latter is akin to using them.
On 12 July 2013 17:01, Andrew Barnert <abarnert@yahoo.com> wrote:
On Jul 12, 2013, at 1:49, Joshua Landau <joshua@landau.ws> wrote:
On 12 July 2013 09:03, Andrew Barnert <abarnert@yahoo.com> wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
I like this idea, but only if it's added to other functions in the stdlib where it makes sense, and easy to add to new functions of your own.
Is there a use-case for that last part? It strikes me as equivalent to messing with builtins, which is largely unliked.
I think you missed the word "to" in "add to". I don't want to create functions that can be passed to fold as if they were operators, I want to create functions that can take operators the same way fold does.
The former is akin to messing with builtins, which is bad; the latter is akin to using them.
Apologies; in that case I agree completely.
On 12 July 2013 09:03, Andrew Barnert <abarnert@yahoo.com> wrote:
Meanwhile, there are an awful lot of people who just don't like reduce/fold in any situation. The quote "Inside every reduce is a loop trying to get out" appears quite frequently, on this list and elsewhere.
And yet we keep getting cases like the sum discussion which is a fold in essence, but people reject suggestions of "just use a loop". So it doesn't look like the loop is trying very hard to get out :-) Whether "inside every specialised function there is a fold trying to get out" is any more likely to gain traction, I don't know... Paul
I'd be all for increasing usage of fold and reduce, and higher order combinators in general, but wasn't it always a somewhat philosophical issue that kept lambdas intentionally verbose/crippled and discouraged usage of Higher-Order-Functions when direct imperative code (i.e. loops) works? I've always felt reduce() being banished was but a small facet of this overall philosophy, and not so much because it was individually difficult to use.
data = fold("+=", [], iterables)
Seems like a terrible hack to me =( it brings back memories of my PHP days where "first class functions" meant you passed in the functions name as a string which got concatted-around and eval-ed. We all laughed at how badly they designed the language to have it end up like that. Naturally, it is the path of least resistance, since it could be implemented with existing language features (i.e. `eval`, which can implement anything really) but it would leave a sour taste in my mouth every time i use it. I would much prefer the somewhat-more-difficult route of modifying the parser to let `a += b` be an expression, and then you could write data = fold(lambda a, b: a += b, [], iterables) or even groovy/scala/mathematica style data = fold(_ += _, [], iterables) Which is a lot further (implementation wise) from where we are now, and 2 characters more verbose, but it would be far more generally usable than a one-off "let's pass in operators as strings and concat/eval them" rule. -Haoyi On Fri, Jul 12, 2013 at 6:11 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 12 July 2013 09:03, Andrew Barnert <abarnert@yahoo.com> wrote:
Meanwhile, there are an awful lot of people who just don't like reduce/fold in any situation. The quote "Inside every reduce is a loop trying to get out" appears quite frequently, on this list and elsewhere.
And yet we keep getting cases like the sum discussion which is a fold in essence, but people reject suggestions of "just use a loop". So it doesn't look like the loop is trying very hard to get out :-)
Whether "inside every specialised function there is a fold trying to get out" is any more likely to gain traction, I don't know...
Paul
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
On 12 July 2013 11:36, Haoyi Li <haoyi.sg@gmail.com> wrote:
I'd be all for increasing usage of fold and reduce, and higher order combinators in general, but wasn't it always a somewhat philosophical issue that kept lambdas intentionally verbose/crippled and discouraged usage of Higher-Order-Functions when direct imperative code (i.e. loops) works? I've always felt reduce() being banished was but a small facet of this overall philosophy, and not so much because it was individually difficult to use.
data = fold("+=", [], iterables)
Seems like a terrible hack to me =( it brings back memories of my PHP days where "first class functions" meant you passed in the functions name as a string which got concatted-around and eval-ed. We all laughed at how badly they designed the language to have it end up like that. Naturally, it is the path of least resistance, since it could be implemented with existing language features (i.e. `eval`, which can implement anything really) but it would leave a sour taste in my mouth every time i use it.
I would much prefer the somewhat-more-difficult route of modifying the parser to let `a += b` be an expression, and then you could write
data = fold(lambda a, b: a += b, [], iterables)
or even groovy/scala/mathematica style
data = fold(_ += _, [], iterables)
Which is a lot further (implementation wise) from where we are now, and 2 characters more verbose, but it would be far more generally usable than a one-off "let's pass in operators as strings and concat/eval them" rule.
Yeah I'm the author of that =D My point wasn't so much that "use my cool macroz!!!" as "passing operator as string PHP-style makes me sad =(" and with examples of how other languages do it that doesn't make me sad. On Fri, Jul 12, 2013 at 7:58 PM, Joshua Landau <joshua@landau.ws> wrote:
I'd be all for increasing usage of fold and reduce, and higher order combinators in general, but wasn't it always a somewhat philosophical issue that kept lambdas intentionally verbose/crippled and discouraged usage of Higher-Order-Functions when direct imperative code (i.e. loops) works? I've always felt reduce() being banished was but a small facet of this overall philosophy, and not so much because it was individually difficult to use.
data = fold("+=", [], iterables)
Seems like a terrible hack to me =( it brings back memories of my PHP days where "first class functions" meant you passed in the functions name as a string which got concatted-around and eval-ed. We all laughed at how badly they designed the language to have it end up like that. Naturally, it is
On 12 July 2013 11:36, Haoyi Li <haoyi.sg@gmail.com> wrote: the
path of least resistance, since it could be implemented with existing language features (i.e. `eval`, which can implement anything really) but it would leave a sour taste in my mouth every time i use it.
I would much prefer the somewhat-more-difficult route of modifying the parser to let `a += b` be an expression, and then you could write
data = fold(lambda a, b: a += b, [], iterables)
or even groovy/scala/mathematica style
data = fold(_ += _, [], iterables)
Which is a lot further (implementation wise) from where we are now, and 2 characters more verbose, but it would be far more generally usable than a one-off "let's pass in operators as strings and concat/eval them" rule.
On Jul 12, 2013, at 3:36, Haoyi Li <haoyi.sg@gmail.com> wrote:
I would much prefer the somewhat-more-difficult route of modifying the parser to let `a += b` be an expression, and then you could write
data = fold(lambda a, b: a += b, [], iterables)
But that doesn't really _mean_ anything as an expression; it's pure side effect. Besides, if that returns the new a then it invites all the expression order problems from C and friends that Python has always escaped; if it returns None, the fold doesn't actually work. Meanwhile, the way you wrote it:
`a += b`
Reminds me. We took away backticks for repr; what about using them for quoting operators? `+=` says "quoting" in the lisp sense more loudly than the PHP sense... At least to me. And this means fold wouldn't need to accept a string; it's getting a function. Of course it's the exact opposite of Haskell, where you use backticks to turn a function into an operator and parens to turn an operator into a function.
or even groovy/scala/mathematica style
data = fold(_ += _, [], iterables)
I semi-suggested this elsewhere, but without the magic of guessing whether two underscores meant the same arg twice or two different args (so you'd have to write _1 == _2 or similar). You can actually write functions this way today using an expression template library, without macros or anything: class Expr: def __init__(self, f=identity) self.f = f def __iadd__(self, other): return Expr(compose(iadd, self.f)) def __call__(self, *args): return self.f(*args) _1, _2 = Expr(), Expr() Actually, this silly proof of concept _would_ work with _ for both args, but only because it won't work for much else (even adding literals). Anyway, while you can do this, I'm not sure you should. Boost.Lambda had this kind of magic, and nobody proposed it for C++11 when real lambdas were added, because it's a clumsy fit for a language that wasn't designed for it from the start--you end up needing const, var, and ref functions to paper over the gaps where overloading doesn't quite get you there, and horrible workarounds for the operators that can't be overloaded, ...
On 07/12/2013 08:01 AM, Nick Coghlan wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
I'd like to see a more Haskell like way to reference operators: data = fold((+=), [], iterables) (+=) would just be a short syntax for operator.iadd without the need to explicitly import any module. It should generate the same byte code. But I have the feeling that won't happen. :/ -panzi
On 12 July 2013 14:50, Mathias Panzenböck <grosser.meister.morti@gmx.net> wrote:
On 07/12/2013 08:01 AM, Nick Coghlan wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
I'd like to see a more Haskell like way to reference operators:
data = fold((+=), [], iterables)
(+=) would just be a short syntax for operator.iadd without the need to explicitly import any module. It should generate the same byte code.
But I have the feeling that won't happen. :/
Damn straight! Do you realise how much of an attractive nuisance that would be for people constantly begging "I have (+) now so why do I have to write lambda for <blah blah blah>" and then Guido gets upset because he's covered this so many times before and no-one will just agree goddamnit?
On 07/12/2013 04:17 PM, Joshua Landau wrote:
On 12 July 2013 14:50, Mathias Panzenböck <grosser.meister.morti@gmx.net> wrote:
On 07/12/2013 08:01 AM, Nick Coghlan wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
I'd like to see a more Haskell like way to reference operators:
data = fold((+=), [], iterables)
(+=) would just be a short syntax for operator.iadd without the need to explicitly import any module. It should generate the same byte code.
But I have the feeling that won't happen. :/
Damn straight! Do you realise how much of an attractive nuisance that would be for people constantly begging "I have (+) now so why do I have to write lambda for <blah blah blah>" and then Guido gets upset because he's covered this so many times before and no-one will just agree goddamnit?
I get your point, but (+) wouldn't be a lambda. It would just be a shorthand for operator.add. So you could write (+)(a, b) instead of a + b. Well, thinking of that maybe it's not such a good idea.
On Jul 12, 2013, at 04:01 PM, Nick Coghlan wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
You had me until here...
(Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd)
...but this is a neat idea. -Barry
On Fri, Jul 12, 2013 at 10:54 AM, Barry Warsaw <barry@python.org> wrote:
On Jul 12, 2013, at 04:01 PM, Nick Coghlan wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
You had me until here...
(Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd)
...but this is a neat idea.
+1 from me as well. The table already exists in the docs ( http://docs.python.org/3.4/library/operator.html#module-operator), it just needs to be codified. Maybe operator.map['+='] or operator.from_syntax['+=']. Go really nuts and support ['.attribute'] or ['[42]'] to auto-generate attrgetter or itemgetter instances, but that's probably just asking for support problems.
On Fri, Jul 12, 2013 at 2:32 PM, Brett Cannon <brett@python.org> wrote:
On Fri, Jul 12, 2013 at 10:54 AM, Barry Warsaw <barry@python.org> wrote:
On Jul 12, 2013, at 04:01 PM, Nick Coghlan wrote:
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
You had me until here...
(Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd)
...but this is a neat idea.
+1 from me as well. The table already exists in the docs (http://docs.python.org/3.4/library/operator.html#module-operator), it just needs to be codified. Maybe operator.map['+='] or operator.from_syntax['+=']. Go really nuts and support ['.attribute'] or ['[42]'] to auto-generate attrgetter or itemgetter instances, but that's probably just asking for support problems.
I decided to take a stab at this idea and thus created issue18436[1]. -- Zach [1]http://bugs.python.org/issue18436
On Fri, Jul 12, 2013 at 10:54 AM, Barry Warsaw <barry@python.org> wrote:
(Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd)
...but this is a neat idea.
-1 This is neat, but I don't really see much use beyond implementing things like fold("+=", ..) that you've just rejected.
12.07.13 22:53, Alexander Belopolsky написав(ла):
On Fri, Jul 12, 2013 at 10:54 AM, Barry Warsaw <barry@python.org <mailto:barry@python.org>> wrote: >(Independent of this idea, it would actually be nice if the operator module >had a dictionary mapping from op symbols to names, like >operator.by_symbol["+="] giving operator.iadd)
...but this is a neat idea.
-1
This is neat, but I don't really see much use beyond implementing things like fold("+=", ..) that you've just rejected.
Concur with Alexander. Why you want yet one alternative names for operators? What are alternative names for operator.neg() and operator.sub(). If any of them is not "-" then what is a benefit?
On Fri, Jul 12, 2013 at 2:53 PM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Fri, Jul 12, 2013 at 10:54 AM, Barry Warsaw <barry@python.org> wrote:
(Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd)
...but this is a neat idea.
-1
This is neat, but I don't really see much use beyond implementing things like fold("+=", ..) that you've just rejected.
I do somewhat agree that there may not be much place to use this in the standard library, but I think it could make use of the operator module a bit easier to read in some cases. For instance, it takes me a second thought to correctly parse `operator.irshift` as "right-shift in-place" instead of "IR shift" (which may or may not mean anything, but certainly doesn't in Python). On the other hand, `operator.get_op(">>=")` shows what the returned function is going to do. -- Zach
On 07/12/2013 01:01 AM, Nick Coghlan wrote:
Efficiently merging a collection of iterables into a list would then just be:
data = fold(operator.iadd, [], iterables)
I'd personally be in favour of the notion of also allowing strings as the first argument, so you could instead write:
data = fold("+=", [], iterables)
This could also be introduced as an alternative API in functools.
How about if start was first... and could take a class? data = fold(class, op, iterables) data = fold(instance, op, iterables)
(Independent of this idea, it would actually be nice if the operator module had a dictionary mapping from op symbols to names, like operator.by_symbol["+="] giving operator.iadd)
I'm guessing that this is how that might be used? name = operator.symbols['+='] op_method = start.__getattribute__(name) Looks good to me. +1 Cheers, Ron
participants (14)
-
Alexander Belopolsky -
Andrew Barnert -
Antoine Pitrou -
Barry Warsaw -
Brett Cannon -
Haoyi Li -
Joshua Landau -
Mathias Panzenböck -
Nick Coghlan -
Paul Moore -
Robert Collins -
Ron Adam -
Serhiy Storchaka -
Zachary Ware