[Python-ideas] (no subject)

[Andrew Barnert]

Apologies for the split replies; is everyone else seeing this as three separate threads spawned from two copies of the original mail, or is this just Yahoo sucking again?

On May 6, 2015, at 08:48, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>> On Wed, May 06, 2015 at 06:59:45AM -0700, Andrew Barnert via Python-ideas wrote:
>> 
>> Python functions don't just take 1 parameter, they take any number of 
>> parameters, possibly including optional parameters, keyword-only, 
>> *args, **kwargs, etc.
> 
> Maybe Haskell programmers are used to functions which all take one 
> argument, and f(a, b, c) is syntactic sugar for f(a)(b)(c), but I doubt 
> anyone else is.

But that's exactly the point. In Haskell, because f(a, b, c) is syntactic sugar for f(a)(b)(c), it's obvious (to a Haskell programmer, not to a human) what it means to compose f. In Python, it's not at all obvious.

Or, worse, it _is_ obvious that you just can't compose f with anything. There is no function whose return value can be passed to a function that takes 3 arguments. (Unless you add some extra rule, like auto-*-unpacking, or passing along *args[1:], **kw up the chain, or you do like Haskell and auto-curry everything.) Which makes composition far less useful in Python than in Haskell.

> When we Python programmers manually compose a function 
> today, by writing an expression or a new function, we have to deal with 
> the exact same problems.

Yes, but we can explicitly decide what to pass for the b and c arguments when writing an expression, and the obvious way to encode that decision is trivially readable. For example:

    f(g(a), b, c)
    f(*g(a, b, c))
    f(g(a)) # using default values for b and c

... etc.

I can't think of any syntax for compose that makes that true.

> There's nothing new about the programmer 
> needing to ensure that the function signatures are compatible:
> 
> def spam(a, b, c):
>    return a+b+c
> 
> def eggs(x, y, z):
>    return x*y/z
> 
> def composed(*args):
>    return eggs(spam(*args))  # doesn't work
> 
> It is the programmer's responsibility to compose compatible functions. 
> Why should it be a fatal flaw that the same limitation applies to a 
> composition operator?

Sure, just as bad for a compose function as for a compose operator. I'm not suggesting we should add composed to the stdlib instead of @, I'm suggesting we should add neither.

> Besides, with Argument Clinic, it's possible that the @ operator could 
> catch incompatible signatures ahead of time.
> 
> 
>> There are a dozen different compose 
>> implementations on PyPI and ActiveState that handle these differently.
> 
> That is good evidence that this is functionality that people want.

Not if nobody is using any of those implementations.

>> Which one is "right"?
> 
> Perhaps all of them? Perhaps none of them? There are lots of buggy or 
> badly designed functions and classes on the internet. Perhaps that 
> suggests that the std lib should solve it right once and for all.

It's not that they're buggy, it's that there are fundamental design choices that have to be made to fit compose into a language like Python, and none of the options are good enough to be standardized. One project may have good uses for a compose that passes along extra args, another for a compose that *-unpacks, and another for auto-currying. Providing one of those won't help the other two projects at all; all it'll do is collide with the name they wanted to use.

>> The design you describe can be easily implemented as a third-party 
>> library. Why not do so, put it on PyPI, see if you get any traction 
>> and any ideas for improvement, and then suggest it for the stdlib?
> 
> I agree that this idea needs to have some real use before it can be 
> added to the std lib, but see below for a counter-objection to the PyPI 
> objection.
> 
> 
>> The same thing is already doable today using a different 
>> operator--and, again, there are a dozen implementations. Why isn't 
>> anyone using them?
> 
> It takes a certain amount of effort for people to discover and use a 
> third-party library: one has to find a library, or libraries, determine 
> that it is mature, decide which competing library to use, determine if 
> the licence is suitable, download and install it. This "activiation 
> energy" is insignificant if the library does something big, say, like 
> numpy, or nltk, or even medium sized.
> 
> But for a library that provides effectively a single function, that 
> activation energy is a barrier to entry. It's not that the function 
> isn't useful, or that people wouldn't use it if it were already 
> available. It's just that the effort to get it is too much bother. 
> People will do without, or re-invent the wheel. (Re-inventing the wheel 
> is at least fun. Searching PyPI and reading licences is not.)
> 
> 
>> Thinking in terms of function composition requires a higher level of 
>> abstraction than thinking in terms of lambda expressions.
> 
> Do you think its harder than, say, the "async for" feature that's just 
> been approved by Guido?

That's not a fair comparison.

Writing proper c10k network code is hard. An extra layer of abstraction that you have to get your head around, but that makes it a lot easier once you do, is a clear win.

Calling functions with the result of other functions is easy. An extra layer of abstraction that you have to get your head around, but that makes it possible to write slightly more concise or elegant code once you do, is probably not a win.

When I first come back to Python after a bit of time with another language, I have to shift gears and stop reducing compositions and instead loop over explicitly composed expressions, but that means I write more Pythonic code. I don't want Python to enable me to write code that Guido can't understand (unless it's something inherently complex in the first place).

> Compared to asynchronous code, I would say function composition is 
> trivial. Anyone who can learn the correspondence
> 
>    (a @ b)(arg)  <=> a(b(arg))
> 
> can deal with it.
> 
> 
>> Python doesn't have a static optimizing compiler that can avoid 
>> building 4 temporary function objects to evaluate (plot @ sorted @ 
>> sqrt @ real) (data_array), so it will make your code significantly 
>> less efficient.
> 
> Why would it necessarily have to create 4 temporary function objects? 
> Besides, the rules for optimization apply here too: don't dismiss 
> something as too slow until you've measured it :-)

It's not so much creating the 4 temporary objects as having to call through them every time you want to call the composed function.

And, while I obviously haven't measured the vaporware implementation of the current proposal, I have played around with different ways of writing Haskelly code in Python and how they fare without a GHC-style optimizer, and the performance impact is definitely noticeable in almost everything you do.

Of course it's possible that the very nature of "playing" vs. writing production code means that I was pushing it a lot farther than anyone would do in real life (nobody's going to invent and apply new combinators in production code, even in Haskell...), so I'll concede that maybe this wouldn't be a problem. But I suspect it will be.

> We shouldn't care about the cost of the @ operator itself, only the cost 
> of calling the composed functions. Building the Composed object 
> generally happens only once, while calling it generally happens many 
> times.
> 
> 
>> Is @ for composition and () for application really sufficient to write 
>> point free code in general without auto-curried functions, operator 
>> sectioning, reverse compose, reverse apply, etc.? Most of the examples 
>> people use in describing the feature from Haskell have a (+ 1) or (== 
>> x) or take advantage of map-type functions being (a->b) -> ([a] -> 
>> [b]) instead of (a->b, [a]) -> [b].
> 
> See, now *that's* why people consider Haskell to be difficult:

Hold on. (+ 1) meaning lambda x: x + 1 doesn't require any abstruse graduate-level math.

Understanding auto-currying map functions might... But that's kind of my point: most of the best examples for compose involve exactly these kinds of things that you don't want to even try to understand. And notice that the author of this current proposal thinks we should add the same thing to Python. Doesn't that make you worry that maybe compose belongs to the wrong universe?

> it is 
> based on areas of mathematics which even maths graduates may never have 
> come across. But function composition is taught in high school. (At 
> least in Australia, and I expect Europe and Japan.) It's a nice, simple 
> and useful functional tool, like partial.
> 
> 
> -- 
> Steve
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/