It would be best to discuss this on comp.lang.python or python-ideas to get general support for the idea before trying to bring this to python-dev in hopes of changing people's minds. On Fri, Aug 14, 2009 at 11:39, Jason R. Coombs <jaraco@jaraco.com> wrote:

I’d like to express additional interest in python patch 1660179, discussed here:

http://mail.python.org/pipermail/patches/2007-February/021687.html

On several occasions, I’ve had the desire for something like this. I’ve made due with lambda functions, but as was mentioned, the lambda is clumsy and harder to read than functools.compose would be.

A potentially common use-case is when a library has a multi-decorator use case in which they want to compose a meta decorator out of one or more individual decorators.

Consider the hypothetical library.

# we have three decorators we use commonly

def dec_register_function_for_x(func):

# do something with func

return func

def dec_alter_docstring(func):

# do something to func.__doc__

return func

def inject_some_data(data):

def dec_inject_data(func):

func.data = data # this may not be legal, but assume it does something useful

return func

return dec_inject_data

# we could use these decorators explicitly throughout our project

@dec_register_function_for_x

@dec_alter_docstring

@dec_inject_some_data(‘foo data 1’)

def our_func_1(params):

pass

@dec_register_function_for_x

@dec_alter_docstring

@dec_inject_some_data(‘foo data 2’)

def our_func_2(params):

pass

For two functions, that’s not too onerous, but if it’s used throughout the application, it would be nice to abstract the collection of decorators. One could do this with lambdas.

def meta_decorator(data):

return lambda func: dec_register_function_for_x(dec_alter_docstring(dec_inject_some_data(data)(func)))

But to me, a compose function is much easier to read and much more consistent with the decorator usage syntax itself.

def meta_decorator(data):

return compose(dec_register_function_for_x, dec_alter_docstring, dec_inject_some_data(data))

The latter implementation seems much more readable and elegant. One doesn’t even need to know the decorator signature to effectively compose meta_decorators.

I’ve heard it said that Python is not a functional language, but if that were really the case, then functools would not exist. In addition to the example described above, I’ve had multiple occasions where having a general purpose function composition function would have simplified the implementation by providing a basic functional construct. While Python isn’t primarily a functional language, it does have some functional constructs, and this is one of the features that makes Python so versatile; one can program functionally, procedurally, or in an object-oriented way, all within the same language.

I admit, I may be a bit biased; my first formal programming course was taught in Scheme.

Nevertheless, I believe functools is the ideal location for a very basic and general capability such as composition.

I realize this patch was rejected, but I’d like to propose reviving the patch and incorporating it into functools.

Regards,

Jason

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

Steven D'Aprano wrote:

Sent: Sunday, 16 August, 2009 08:15

On Sat, 15 Aug 2009 04:39:03 am Jason R. Coombs wrote:

...

def meta_decorator(data): return compose(dec_register_function_for_x, dec_alter_docstring, dec_inject_some_data(data))

Surely that's better written as:

meta_decorator = compose(dec_register_function_for_x, dec_alter_docstring, dec_inject_some_data)

I agree. The former looks unnecessarily complicated. I purposely chose a non-trivial use case, one which involves a decorator that requires a parameter and thus must be called first before the actual decorator is returned. I think for this reason, the former syntax must be used so that the meta_decorator also takes the data parameter and constructs the proper "inject" decorator. Put another way, both dec_inject_some_data and meta_decorator are more like decorator factories. I suspect a simpler, and more common use-case would be like the one you described, where either data is global or the "inject" decorator is not used: meta_decorator = compose(dec_register_function_for_x, dec_alter_docstring)

Mine wasn't -- I've never even used Scheme, or Lisp, or any other functional language. But I've come to appreciate Python's functional tools, and would like to give a +0.5 to compose(). +1 if anyone can come up with additional use-cases.

Thanks for the interest. I decided to search through some of my active code for lambdas and see if there are areas where I would prefer to be using a compose function instead of an explicit lambda/reduce combination. I only found one such application; I attribute this limited finding to the fact that I probably elected for a procedural implementation when the functional implementation might have proven difficult to read, esp. with lambda. 1) Multiple string substitutions. You have a list of functions that operate on a string, but you want to collect them into a single operator that can be applied to a list of strings. sub_year = lambda s: s.replace("%Y", "2009") fix_strings_with_substituted_year = compose(str.strip, textwrap.dedent, sub_year) map(fix_strings_with_substituted_year, target_strings) Moreover, it would be great to be able to accept any number of substitutions. substitutions = [sub_year, sub_month, ...] fix_strings_with_substitutions = compose(str.strip, textwrap.dedent, *substitutions) I did conceive of another possibly interesting use case: vector translation. Consider an application that performs mathematical translations on n-dimensional vectors. While it would be optimal to use optimized matrix operations to perform these translations, for the sake of this example, all we have are basic Python programming constructs. At run-time, the user can compose an experiment to be conducted on his series of vectors. To do this, he selects from a list of provided translations and can provide his own. These translations can be tagged as named translations and thereafter used as translations themselves. The code might look something like: translations = selected_translations + custom_translations meta_translation = compose(*translations) save_translation(meta_translation, "My New Translation") def run_experiment(translation, vectors): result = map(translation, vectors) # do something with result Then, run_experiment can take a single translation or a meta-translation such as the one created above. This use-case highlights that a composed functions must take and return exactly one value, but that the value need not be a primitive scalar. I'm certain there are other, more obscure examples, but I feel these two use cases demonstrate some fairly common potential use cases for something like a composition function. Jason

[Antoine Pitrou]

I also think it would be a nice addition. (but someone has to propose a patch :-))

I agree with Martin's reasons for rejecting the feature request (see the bug report for his full explanation). IIRC, the compose() idea had come-up and been rejected in previous discussions as well. At best, it will be a little syntactic sugar (though somewhat odd because the traditional mathematical ordering of a composition operator is the opposite of what intuition would suggest). At worst, it will be slower and less flexible than our normal ways of linking functions together. IMO, its only virtue is that people coming from functional languages are used to having compose. Otherwise, it's a YAGNI. Raymond

Raymond Hettinger <python <at> rcn.com> writes:

IMO, its only virtue is that people coming from functional languages are used to having compose. Otherwise, it's a YAGNI.

Then I wonder how partial() ended up in the stdlib. It seems hardly more useful than compose(). Either we decide it is useful to have a set of basic "functional" tools in the stdlib, and both partial() and compose() have their place there, or we decide functools has no place in the stdlib at all. Providing a half-assed module is probably frustrating to its potential users. (not being particularly attached to functional tools, I still think compose() has its value, and Jason did a good job of presenting potential use cases) Regards Antoine.

Then I wonder how partial() ended up in the stdlib.

PEP 309 was written, discussed, approved, and implemented - that's how partial ended up in the stdlib. The feature itself might be debatable, that's what we have the PEP process for.

Either we decide it is useful to have a set of basic "functional" tools in the stdlib, and both partial() and compose() have their place there, or we decide functools has no place in the stdlib at all. Providing a half-assed module is probably frustrating to its potential users.

So write a PEP and propose to enhance the standard library.

(not being particularly attached to functional tools, I still think compose() has its value, and Jason did a good job of presenting potential use cases)

I don't think he did. Comparing it to the one obvious solution (use a lambda expression), his only reasoning was "it is much easier to read". I truly cannot believe that a compose function would be easier to read to the average Python programmer: if you have def foo(data): return compose(a, b(data), c) what would you expect that to mean? Please rewrite it as a regular Python expression, preferably without looking at the patch that has been proposed first. I bet there is a 50% chance that you get it wrong (because there are two possible interpretations). Regards, Martin

PEP 309 was written, discussed, approved, and implemented - that's how partial ended up in the stdlib.

Ok, I'm surprised that a single addition to a module needed a PEP in order to be approved. Interestingly, here's what the summary section in PEP 309 says: « A standard library module functional should contain an implementation of partial, /and any other higher-order functions the community want/. » (emphasis mine)

I truly cannot believe that a compose function would be easier to read to the average Python programmer: if you have

def foo(data): return compose(a, b(data), c)

what would you expect that to mean? Please rewrite it as a regular Python expression, preferably without looking at the patch that has been proposed first.

Ok, here's my attempt without looking at the patch: def foo(data): def bar(*args, **kwargs): return a(b(data)(c(*args, **kwargs))) return bar Whether or not it is easier to read to the "average Python programmer" is not that important I think. We have lots of things that certainly aren't, and yet still exist (all of the functions in the operator module, for example; or `partial` itself for that matter). They are there for advanced programmers. Regards Antoine.

...

PEP 309 was written, discussed, approved, and implemented - that's how partial ended up in the stdlib.

Ok, I'm surprised that a single addition to a module needed a PEP in order to be approved.

A PEP is generally needed if there is no easy consent achievable. It's not (primarily) the size of a feature that determines the need for a formal process, but but whether the community considers a certain change "obviously" correct and desirable.

...

def foo(data): return compose(a, b(data), c)

Ok, here's my attempt without looking at the patch:

def foo(data): def bar(*args, **kwargs): return a(b(data)(c(*args, **kwargs))) return bar

Ok, that's also what the patch has proposed. I was puzzled when I read l.sort(key=compose(itemgetter(1), itemgetter(0)))) because I expected it to mean l.sort(key=lambda x:x[1][0]) when it would really mean l.sort(key=lambda x:x[0][1])

Whether or not it is easier to read to the "average Python programmer" is not that important I think.

I completely disagree. It is one of Python's strength that it is "executable pseudo-code", which originates from the code being easy to read, and meaning the obvious thing even to a reader not familiar with the language. The proposed compose function breaks this important property, in a way that allows misinterpretation (i.e. you think you know what it does, and it actually does something different). I, personally, was not able to understand the compose function correctly, so I remain opposed.

We have lots of things that certainly aren't, and yet still exist (all of the functions in the operator module, for example; or `partial` itself for that matter). They are there for advanced programmers.

It's quite ok if only advanced programmers know that they are there, and know how to write them. However, I still think it is desirable that "lesser" programmers are then able to read them, or atleast notice that they mean something that they will need to learn first (such as a keyword they had never seen before). Regards, Martin

Greg Ewing wrote:

Jason R. Coombs wrote:

...

I had a use case that was compelling enough that I thought there should be something in functools to do what I wanted.

I think this is one of those things that a small minority of people would use frequently, but everyone else would use very rarely or never. The decision on whether to include something in the stdlib needs to be based on the wider picture.

In this case, it's trivial to write your own if you want it. As they say, "not every one-line function needs to be in the stdlib".

I have never found these arguments compelling. They are obviously not true (e.g., itertools.compress()[1] added in 2.7/3.1), and so what I really hear is: "I don't like it and I outrank you." I can't help invoke part of PEP309's justification for functools.partial()[2]: """ I agree that lambda is usually good enough, just not always. And I want the possibility of useful introspection and subclassing. """ The same reasoning would seem to apply here. In the OP's example, the meta-decorator becomes opaque due to the use of a lambda. If one could introspect a compose(), then introspection tools could actually know the set of decorators being applied. As it is, the "preferred" method of using a lambda actually makes it quite hard to know anything. class compose(): def __init__(self, *funcs): if not funcs: raise ValueError(funcs) self.funcs = funcs def __call__(self, *args, **kwargs): v = self.funcs[-1](*args, **kwargs) for func in reversed(self.funcs[:-1]): v = func(v) return v meta = functools.compose(decorator_a, decorator_b) print meta.funcs meta = lambda f: decorator_a(decorator_b(f)) # impossible, short of disassembling the lambda -Scott [1] http://docs.python.org/3.1/library/itertools.html#itertools.compress """ def compress(data, selectors): # compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F return (d for d, s in zip(data, selectors) if s) """

On 17 Aug 2009, at 09:43 , Steven D'Aprano wrote:

On Mon, 17 Aug 2009 08:10:16 am Martin v. Löwis wrote:

...

I don't think he did. Comparing it to the one obvious solution (use a lambda expression), his only reasoning was "it is much easier to read". I truly cannot believe that a compose function would be easier to read to the average Python programmer: if you have

def foo(data): return compose(a, b(data), c)

what would you expect that to mean?

foo is a factory function that, given an argument `data`, generates a function b(data), then composes it with two other functions a and c, and returns the result, also a function.

From his messages, I think Martin's issue with `compose` is with the composition order rather than the fact that it "pipes" functions: compose uses the mathematical order, (f ∘ g)(x) = f(g(x)) (so g, the last function of the composition, is applied first), rather than a "shell pipe" order of `(f >>> g)(x) = g(f(x))` (where g, the last function of the composition, is applied last).

For the record, Haskell makes compose a built-in operator:

http://www.haskell.org/haskellwiki/Function_composition

Yes, but Haskell also has a left-to-right composition, the (>>>) operator: http://haskell.org/ghc/docs/latest/html/libraries/base/Control-Arrow.html#v :>>>

Le lundi 17 août 2009 à 09:07 +0200, "Martin v. Löwis" a écrit :

Ok, that's also what the patch has proposed. I was puzzled when I read

l.sort(key=compose(itemgetter(1), itemgetter(0))))

because I expected it to mean

l.sort(key=lambda x:x[1][0])

But that's itemgetter's fault, not compose's. Because itemgetter's obvious equivalent (the [] operator) uses postfix notation, combining several itemgetters reverses the lexical order of appearance. Besides, the argument order is similar to the one in the function composition notation in mathematics (which isn't really advanced stuff and should have been taught to every former scientific/technical student out there). Regards Antoine.

and I don't see any reason why a compose() function shouldn't do the same.

I was tricked into reading it different when used with getters, i.e. l.sort(key=compose(attrgetter('name'),attrgetter('age'))) is too easy (IMO) to read as applying foo.name.age on all elements of the list.

(Aside: how do I look at the patch? The only link I have is here: http://mail.python.org/pipermail/patches/2007-February/021687.html but I can't see how to get to the patch from there.)

It's best to search for "compose" in the bug tracker: http://bugs.python.org/issue1660179 Regards, Martin

On Mon, 17 Aug 2009 07:14:05 pm Stefan Behnel wrote:

Antoine Pitrou wrote:

...

Raymond Hettinger <python <at> rcn.com> writes:

...

IMO, its only virtue is that people coming from functional languages are used to having compose. Otherwise, it's a YAGNI.

Then I wonder how partial() ended up in the stdlib. It seems hardly more useful than compose().

I would certainly consider it more useful, but that aside, it's also a lot simpler to understand and use than the proposed compose() function. I think the main difference is that compose() requires functional/math skills to be used and read correctly (and might still be surprising in some corner cases), whereas partial() only requires you to understand how to set a function argument. Totally different level of mental complexity, IMHO.

I find the opposite -- compose() seems completely simple and straight-forward to me, while partial() is still a mystery no matter how many times I use it. I always have to look it up to see which way it binds. Putting that aside, partial() too is easy enough to implement with lambda: partial(f, 2) is the same as lambda *args: f(2, *args). To my mind, there are two important reasons for preferring named functions like partial() and compose() over lambda solutions: * performance: a good C implementation should be better than a pure-Python lambda; and * specificity: there's only one thing compose() or partial() could do, whereas a lambda is so general it could do anything. Contrast: compose(f, g, h) lambda x: f(g(h(x))) You need to read virtually the entire lambda before you can distinguish it from some other arbitrary lambda: lambda x: f(g(h))(x) lambda x: f(g(x) or h(x)) lambda x: f(g(x)) + h(x) etc. -- Steven D'Aprano