Mailman 3 May 2015 - Python-ideas

Re: [Python-ideas] Function composition
by Ivan Levkivskyi 06 May '15

06 May '15

Dear Steve, Thank you for the feedback and for the links! I think that both (f(a)g).__name__ and str(f@g) should be f.__name__ + ' @ ' + g.__name__ and str(f) + ' @ ' +str(g) Concerning the performance, I think that it could be poor, and I don't know yet how to improve this. > > Dear all, > > > > The matrix multiplication operator @ is going to be introduced in Python > > 3.5 and I am thinking about the following idea: > > > > The semantics of matrix multiplication is the composition of the > > corresponding linear transformations. > > A linear transformation is a particular example of a more general concept - > > functions. > > The latter are frequently composed with ("wrap") each other. For example: > > > > plot(real(sqrt(data))) > > > > However, it is not very readable in case of many wrapping layers. > > Therefore, it could be useful to employ > > the matrix multiplication operator @ for indication of function > > composition. This could be done by such (simplified) decorator: > > I like the idea of @ as a function compose operator. > > There have been many requests and attempts at support for function > composition: > > http://code.activestate.com/recipes/574458-composable-functions/ > > http://code.activestate.com/recipes/52902-function-composition/ > > http://code.activestate.com/recipes/528929-dynamic-function-composition-dec… > > http://blog.o1iver.net/2011/08/09/python-function-composition.html > > https://mail.python.org/pipermail/python-dev/2009-August/091161.html > > http://stackoverflow.com/questions/2281693/is-it-a-good-idea-to-have-a-synt… > > > The last one is notable, as it floundered in part on the lack of a good > operator. I think @ makes a good operator for function composition. > > > I think that there are some questions that would need to be answered. > For instance, given some composition: > > f = math.sin @ (lambda x: x**2) > > what would f.__name__ return? What about str(f)? > > > Do the composed functions: > > (spam @ eggs @ cheese)(x) > > perform acceptibly compared to the traditional syntax? > > spam(eggs(cheese(x)) > > > > -- > Steve

1 0

Re: [Python-ideas] async/await and synchronous code (and PEP492 ?)
by Koos Zevenhoven 06 May '15

06 May '15

Hi all, I noticed a typo in my first email (had written__aenter__ instead of __enter__). I fixed the typo below. -- Koos On 5.5.2015 16:55, Koos Zevenhoven wrote: > > Relation to PEP492: > > This of course still needs more thinking, but I wanted to post it here > now in case there is desire to prepare for something like this already > in PEP492. It is not completely clear if/how this would need to affect > PEP492, but some things come to mind. For example, this could > potentially remove the need for __aenter__, __aiter__, etc. or even > "async for" and "async with". If __enter__ is defined as "async def", > then a with statement would do an "await" on it, and the context > manager would have __async__.framework (or whatever it would be > called) available, for determining what behavior is appropriate. >

1 0

async/await and synchronous code (and PEP492 ?)
by Koos Zevenhoven 06 May '15

06 May '15

Hi all! I am excited about seeing what's going on with asyncio and PEP492 etc. I really like that Python is becoming more suitable for the increasing amount of async code and that the distinction between async functions and generators is increasing. In addition, however, I would also like to see the async functions and methods come even closer to regular functions and methods. This is something that is keeping me from using asyncio at the moment even if I would like to. Below I'll try to explain what and why, and a little bit of how. If it is not clear, please ask :) Motivation: One of the best things about asyncio and coroutines/async functions is that you can write asynchronous code as if it were synchronous, the difference in many places being just the use of "await" ("yield from") when calling something that may end up doing IO (somewhere down the function call chain) and that the code is run from an event loop. When writing a package that does IO, you have the option to make it either synchronous or asynchronous. Regardless of the choice, the code will look roughly the same. But what if you want to be able to do both? Should you maintain two versions, one with "async" and "await" everywhere and one without? Besides the keywords "async" and "await", async code of course differs from synchronous code by the functions/coroutines that are used for IO at the end of the function call chain. Here, I mean the end (close to) where the "yield" expressions are hidden in the async versions. At the other end of the calling chain, async code needs the event loop and associated framework (almost always asyncio?) which hides all the async scheduling fanciness etc. I'm not sure about the terminology, but I will use "L end" and "Y end" to refer to the two ends here. (L for event Loop; Y for Yield) The Y and L ends need to be compatible with each other for the code to work. While asyncio and the standard library might provide both ends in many cases, there can also be situations where a package would want to work with different combinations of L and Y end, or completely without an event loop, i.e. synchronously. In a very simple example, one might want to wrap different implementations of sleep() in a function that would pick the right one depending on the context. Perhaps something like this: async def any_sleep(seconds): if __async__.framework is None: time.sleep(1) elif __async__.framework is asyncio: await asyncio.sleep(1) else: raise RuntimeError("Was called with an unsupported async framework.") [You could of course replace sleep() with socket IO or whatever, but sleep is nice and simple. Also, a larger library would probably have a whole chain of async functions and methods before calling something like this] But if await is only allowed inside "async def", then how can any_sleep() be conveniently run in non-async code? Also, there is nothing like __async__.framework. Below, I describe what I think a potential solution might look like. Potential solution: This is simplified version; for instance, as "awaitables", I consider only async function objects here. I describe the idea in three parts: (1) next(...): Add a keyword argument "async_framework" (or whatever) to next(...) with a default value of None. When an async framework, typically asyncio, starts an async function object (coroutine) with a call to next(...), it would do something like next(coro, async_framework = asyncio). Here, asyncio could of course be replaced with any object that identifies the framework. This information would then be somehow attached to the async function object. (2) __async__.framework or something similar: Add something like __async__ that has an attribute such as .framework that allows the code inside the async function to access the information passed to next(...) by the framework (L end) using the keyword argument of next [see (1)]. (3) Generalized "await": [When the world is ready:] Allow using "await" anywhere, not just within async functions. Inside async functions, the behavior of "await" would be the same as in PEP492, with the addition that it would somehow propagate the __async__.framework value to the awaited coroutine. Outside async functions, "await" would do roughly the same as this function: def await(async_func_obj): try: next(async_func_obj) # same as next(async_func_obj, async_framework = None) except StopIteration as si: return si.value raise RuntimeError("The function does not support synchronous execution") (This function would, of course, work in Python 3.4, but it would be mostly useless because the async functions would not know that they are being called in a 'synchronous program'. IIUC, this *function* would be valid even with PEP492, but having this as a function would be ugly in the long run.) Some random thoughts: With this addition to Python, one could write libraries that work both async and non-async. When await is not inside async def, one would expect it to potentially do blocking IO, just like an await inside async def would suggest that there is a yield/suspend somewhere in there. For testing, I tried to see if there is a reasonable way to make a hack with __async__.framework that could be set by next(), but did not find an obvious way. For instance, coro.gi_frame.f_locals is read-only, I believe. An alternative to this approach could be that await would implicitly start a temporary event loop for running the coroutine, but how would it know which event loop? This might also have a huge performance overhead. Relation to PEP492: This of course still needs more thinking, but I wanted to post it here now in case there is desire to prepare for something like this already in PEP492. It is not completely clear if/how this would need to affect PEP492, but some things come to mind. For example, this could potentially remove the need for __aenter__, __aiter__, etc. or even "async for" and "async with". If __aenter__ is defined as "async def", then a with statement would do an "await" on it, and the context manager would have __async__.framework (or whatever it would be called) available, for determining what behavior is appropriate. Was this clear enough to understand which problem(s) this would be solving and how? I'd be happy to hear about any thoughts on this :). Best regards, Koos

5 7

Support 1.x notation in version specifiers
by anatoly techtonik 05 May '15

05 May '15

pip team said they won't support setting limit for major version of package being installed in the way below until it is supported by PEP 440. pip install patch==1.x The current way ==1.* conflicts with system shell expansion and the other way is not known / not intuitive. https://github.com/pypa/pip/issues/2737#issuecomment-97621684 -- anatoly t.

4 3

More general "for" loop handling
by Todd 03 May '15

03 May '15

Looking at pep 492, it seems to me the handling of "for" loops has use outside of just asyncio. The primary use-case I can think of is multiprocessing and multithreading. For example, you could create a multiprocessing pool, and let the pool handle the items in a "for" loop, like so: from multiprocessing import Pool mypool = Pool(10, maxtasksperchild=2) mypool for item in items: do_something_here do_something_else do_yet_another_thing Or something similar with third-party modules: from greenlet import greenlet greenlet for item in items: do_something_here do_something_else do_yet_another_thing Of course this sort of thing is possible with iterators and maps today, but I think a lot of the same advantages that apply to asyncio also apply to these sorts of cases. So I think that, rather than having a special keyword just for asyncio, I think it would be better to have a more flexible approach. Perhaps something like a "__for__" magic method that lets a class implement "for" loop handling, along with the corresponding changes in how the language processes the "for" loop.

13 30

awaiting iterables
by Ron Adam 03 May '15

03 May '15

(I had posted this in the "more general 'for' loop" thread, but this really is a different idea from that.) Initialising several coroutines at once still doesn't seem clear/clean to me. Here is what I would like. values = awaiting [awaitable, awaitable, ...] a, b, ... = awaiting (awaitable, awaitable, ...) This doesn't have the issues of order because a list of values is returned with the same order of the awaitables. But the awaitables are scheduled in parallel. A regular for loop could still do these in order, but would pause when it gets to a values that hasn't returned/resolved yet. That would probably be expected. for x in awaiting [awaitable, awaitable, ...]: print(x) X is printed in the order of the awaitables. Awaiting sets would be different... they are unordered. So we could use a set and get the items that become available as they become available... for x in awaiting {awaitable, awaitable, ...}: print(x) x would print in an arbitrary order, but that would be what I would expect here. The body could have await calls in it, and so it could cooperate along with the awaiting set of awaitablers. Of course if the for body is only a few statements, that probably wouldn't make much difference. This seems like it's both explicit and simple to think about. It also seems like it might not be that hard to do, I think most of the parts are already worked out. One option is to allow await to work with iterables in this way. But the awaiting keyword would make the code clearer and error messages nicer. The last piece of the puzzle is how to specify the current coroutine manager/runner. import asyncio with asyncio.coroutine_loop(): main() That seems simple enough. It prettty much abstracts out all the coroutine specific stuff to three keyword. async, await, and awaiting. Are async for and async with needed if we have awaiting? Can they be impelented in terms of awaiting? import asyncio async def factorial(name, number): f = 1 for i in range(2, number+1): print("Task %s: Compute factorial(%s)..." % (name, i)) await yielding() f *= i print("Task %s: factorial(%s) = %s" % (name, number, f)) with asyncio.coroutine_loop(): awaiting [ factorial("A", 2), factorial("B", 3), factorial("C", 4)] Compared to the example in asyncio docs... import asyncio @asyncio.coroutine def factorial(name, number): f = 1 for i in range(2, number+1): print("Task %s: Compute factorial(%s)..." % (name, i)) yield from asyncio.sleep(1) f *= i print("Task %s: factorial(%s) = %s" % (name, number, f)) loop = asyncio.get_event_loop() tasks = [ asyncio.async(factorial("A", 2)), asyncio.async(factorial("B", 3)), asyncio.async(factorial("C", 4))] loop.run_until_complete(asyncio.wait(tasks)) loop.close() Cheers, Ron

3 4

PEP 492 terminology - (native) coroutine objects
by Oscar Benjamin 02 May '15

02 May '15

Looking through PEP 492 I dislike the terminology. With generators I can do: >>> def f(): yield ... >>> g = f() >>> g <generator object f at 0x7fb81dadc798> So f is a generator function and g is generator. That's all pretty clear and "generator" is IMO a great name. The term "generator function" is also unambiguous for f. With PEP 492 it seems that I would get something like: >>> async def af(): pass >>> ag = af() >>> ag <coroutine_object object af at 0x7fb81dadc828> According to the terminology in the PEP af is a coroutine but since the word coroutine also refers to some generators in a looser sense I should use "native coroutine" to disambiguate. ag is a "coroutine object" but again I should use "native coroutine object" to explicitly name the type because without the word "native" it also refers to other things. I think that if the terminology is already ambiguous with related language constructs before it's introduced then it would be better to change it now. The word coroutine (without "native" modifier) is used by the PEP to refer to both generator based coroutines and the new async def functions. I think it's reasonable to use the word as a generic term in this sense. Python's idea of coroutines (both types) doesn't seem to match with the pre-existing general definitions but they are at least generalisations of functions so it can be reasonable to describe them that way loosely. Greg's suggestion to call an async def function (af above) an "async function" seems like a big improvement. It clearly labels the purpose: a function for use in a asynchronous execution probably with the asyncio module. It also matches directly with the syntax: a function prefixed with the word "async". There would be no ambiguity between which of af or ag is referred to by the term. It seems harder to think of a good name for ag though. ag is a wrapper around a suspended call stack with methods to resume the stack so describing what is is doesn't lead to anything helpful. OTOH the purpose of ag is often described as implementing a "minithread" so the word "minithread" makes intuitive sense to me. That way async code is done by writing async functions that return minithreads. An event loop schedules the minthreads etc. (I'm trying to imagine explaining this to someone...) I'm not sure what the best word is for a coroutine object but the current terminology clearly has room for improvement. For a start using the word "object" in the name of a type is a bit rubbish in a language where everything is an object. Worse the PEP is reusing words that have already been used with different meanings so that it's already ambiguous. A concrete builtin language type such as the coroutine object deserves a good, short, one-word name that will not be confused with other things. -- Oscar

5 7

Re: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib
by Alexander Belopolsky 01 May '15

01 May '15

On Mon, Mar 23, 2015 at 2:08 AM, anatoly techtonik <techtonik(a)gmail.com> wrote: > > That's nice to know, but IIRC datetime is from the top 10 Python > modules that need a redesign. Things contained therein doesn't pass > human usability check, and are not used as a result. Where have you been when PEP 3108 was discussed? I have not seen any other list of Python modules that needed a redesign, so I cannot tell what's on your top ten list. Speaking of the datetime module, in what sense does it not "pass human usability check"? It does have a few quirks, for example I would rather see date accept a single argument in the constructor which may be a string, another date or a tuple, but I am not even sure this desire is shared by many other humans. It would be nice if datetime classes were named in CamelCase according to PEP 8 conventions, but again this is a very minor quirk. In my view, if anyone is to blame for the "human usability" of the datetime module, it would be Pope Gregory XIII, Benjamin Franklin and scores of unnamed astronomers who made modern timekeeping such a mess.

13 27