Mailman 3 June 2016 - Python-ideas

Match statement brainstorm
by Guido van Rossum June 1, 2016

June 1, 2016

On Wed, May 18, 2016 at 8:17 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 19 May 2016 at 08:53, Guido van Rossum <guido(a)python.org> wrote: >> The one thing that Python doesn't have (and mypy doesn't add) would be >> a match statement. The design of a Pythonic match statement would be >> an interesting exercise; perhaps we should see how far we can get with >> that for Python 3.7. > > If it's syntactic sugar for a particular variety of if/elif/else > statement, I think that may be feasible, but you'd presumably want to > avoid the "Can we precompute a lookup table?" quagmire that doomed PEP > 3103's switch statement. > > That said, for the pre-computed lookup table case, whatever signature > deconstruction design you came up with for a match statement might > also be usable as the basis for a functools.multidispatch() decorator > design. Let's give up on the pre-computed lookup table. Neither PEP 3103 nor PEP 275 (to which it compares itself) even gets into the unpacking part, which would be the interesting thing from the perspective of learning from Sum types and matching in other languages. Agreed on the idea of trying to reuse this for multidispatch! A few things that might be interesting to explore: - match by value or set of values (like those PEPs) - match by type (isinstance() checks) - match on tuple structure, including nesting and * unpacking (essentially, try a series of destructuring assignments until one works) - match on dict structure? (extension of destructuring to dicts) - match on instance variables or attributes by name? - match on generalized condition (predicate)? The idea is that many of these by themselves are better off using a classic if/elif/else structure, but a powerful matching should be allowed to alternate between e.g. destructuring matches and value or predicate matches. IIUC Haskell allows pattern matches as well as conditions, which they seem to call guards or where-clauses (see https://www.haskell.org/tutorial/patterns.html, or http://learnyouahaskell.com/syntax-in-functions if you like your pages more colorful). Or maybe we should be able to combine structure matches with guards. I guess the tuple structure matching idea is fairly easy to grasp. The attribute idea would be similar to a duck-type check, though more emphasizing data attributes. It would be nice if we could write a match that said "if it has attributes x and y, assign those to local variables p and q, and ignore other attributes". Strawman syntax could be like this: def demo(arg): switch arg: case (x=p, y=q): print('x=', p, 'y=', q) case (a, b, *_): print('a=', a, 'b=', b) else: print('Too bad') Now suppose we had a Point defined like this: Point = namedtuple('Point', 'x y z') and some variables like this: a = Point(x=1, y=2, z=3) b = (1, 2, 3, 4) c = 'hola' d = 42 then we could call demo with these variables: >>> demo(a) x= 1 y= 2 >>> demo(b) a= 1 b= 2 >>> demo(c) a= h b= o >>> demo(d) Too bad >>> (Note the slightly unfortunate outcome for 'hola', but that's what a destructuring assignment would do. Water under the bridge.) Someone else can try to fit simple value equality, set membership, isinstance, and guards into that same syntax. -- --Guido van Rossum (python.org/~guido)

24 78

Declaring immutability (was Re: Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects)
by Nick Coghlan June 1, 2016

June 1, 2016

On 31 May 2016 07:17, "Steven D'Aprano" <steve(a)pearwood.info> wrote: > > On Tue, May 31, 2016 at 01:36:31PM +0000, Joseph Martinot-Lagarde wrote: > > > I can only agree here. Even today, despite knowing the fact, it's > > > causing some headaches in some cases. > > > > How about raising an exception if mutable objects are in the list ? > > -1 > > It's a gratuitous breakage that cannot solve the problem, because you > cannot tell in advance which objects are mutable and which are not. Steven raised the iterator windowing example I was going to point to as the canonical use case that means the current behaviour won't change. However, if folks want a somewhat-related genuinely interesting language design problem to chew on: try to come up with a way a way to formally define-and-enforce instance immutability in Python. As a pragmatic matter, you'll probably still need to allow mutation at least until __new__ finishes running, and some use cases, like weak referencing and some caching techniques, involve storing mutable state on semantically immutable instances, so this is likely to be better approached as a type hinting problem rather than as a runtime feature. The closest related concept I'm personally familiar with is const pointers and references in C and C++, so there may be some insights to be gained there (e.g. being able to declare __new__ as returning "-> typing.Immutable" or "-> typing.Mutable", and being able to explicitly annotate function parameters as either Immutable parameters, which definitely won't be modified, and Mutable parameters, which definitely will be modified. Pure and impure methods could then use those annotations on their "self" parameter to indicate whether or not they had side effects) Cheers, Nick.

2 1

async ProcessPoolExecutor
by Christoph Groth June 1, 2016

June 1, 2016

Hi, While working on a parallel numerical quadrature routine that features speculative evaluation, I got interested in async programming in Python. Most async programming seems to be done for network applications, but it seems to me that controlling complex parallel computations could be also a promising application. The asyncio package has the run_in_executor function that seemed to be just what I need. When looking at how it works I noticed that dispatching tasks from an asyncio event loop to a concurrent.futures.ProcessPoolExecutor involves a lot of wrapping, locking and hence complexity and inefficiency. Shouldn't it be possible to write an async version of ProcessPoolExecutor? I did just this and the results seem quite promising. Consider the following program: **************************************************************** from concurrent import futures import time def work(x): return x**2 def main(): executor = futures.ProcessPoolExecutor(48) now = time.time() fs = [executor.submit(work, x) for x in range(10000)] print(sum(f.result() for f in futures.as_completed(fs))) print(time.time() - now) if __name__ == "__main__": main() **************************************************************** It takes more than 5 seconds on my machine. It becomes even worse (7 seconds) when asyncio with run_in_executor is used. Using the async executor, the following variant of the above script runs in 2 seconds: **************************************************************** import asyncio import time import aexecutor def work(x): return x**2 async def main_coro(): async with aexecutor.ProcessPoolExecutor(48) as executor: now = time.time() fs = [executor.submit(work, x) for x in range(10000)] acc = 0 for f in asyncio.as_completed(fs): acc += await f print(acc) print(time.time() - now) def main(): loop = asyncio.get_event_loop() loop.run_until_complete(main_coro()) if __name__ == "__main__": main() **************************************************************** That difference becomes less when the size of the worker pool gets reduced, but it's significant even for a few workers. And I am actually interested in controlling a huge number of workers (I do computations on a cluster with 20 nodes of 48 cores each). What's more, the source code of the executor got reduced from around 1100 lines to 600. Please see the attachment for the source of the "aexecutor" package as well as the three test programs. (All to be run with Python 3.5.) I expect that the async executor will be useful for me. Not just for the gain in efficiency, but also for having a simple executor to play with. (I think I that I will need futures with a priority and I might also want to write a MPI version of it.) Perhaps this is something that could be interesting for the wider python community? Bear in mind that this is just a quick hack and my first project that uses asyncio. It's most likely buggy and non-portable (it does work on Unix). It can be probably simplified further and there must be a way to improve the inner loop of the scheduler coroutine which currently looks like this: **************************************************************** reader = result_queue._reader event = asyncio.Event(loop=loop) loop.add_reader(reader._handle, event.set) while True: (...) while True: await event.wait() if reader.poll(0): break else: event.clear() result_item = reader.recv() event.clear() (...) **************************************************************** Any ideas? Christoph

1 1

Re: [Python-ideas] Quick idea: defining variables from functions that take the variable name
by Sjoerd Job Postmus June 1, 2016

June 1, 2016

I agree that cleaning up is not really needed. I think the problem is that people feel bad about having to type things twice and immediately think "It's violating DRY". And some are quite adamant about not violating DRY. Fact of the matter: on nearly all relevant scales, this "DRY violation" does not matter, in fact, I would not even want to call it a violation. DRY is about having 1 representative location for knowledge. The name of a variable (1) hardly counts as knowledge and (2) is also used at other locations (use-sites). Is repeating the variable name that much of a deal? Granted, I mostly only use namedtuple of all the given examples. And I use that sparingly. Are `TypeVar` and `Symbol` used more often? At least for namedtuple, we could do some meta-classy logic to allow for class Point(namedtuple): _fields = ('x', 'y') And for namedtuple it makes sense. For TypeVar it might too? class T(TypeVar): pass Symbol? Not so much, I think. I'm not eager to see "improvements" in this area, as I do not see it as a problem, except for those with an overly strict view on DRY. Any new syntax/semantics for this would need to bring a real obvious improvement for the general case, and I doubt that will happen. The one suggestion I have heard that remotely makes sense (and it was not even in this thread!) is the concept of decorators for variables @Something x = 5 would become x = Something('x', 5) And even that idea is bad: what if you want multiple arguments? Would you need a tuple? What if you don't need any arguments? (e.g. Symbol). After seeing all the suggestions so far, I'm not convinced by any of them that they are an improvement. So unless there's a suggestion for syntax/semantics that is reasonable to become the only **obvious** way to do it, I'd say nay. > On 1 Jun 2016, at 02:52, Michael Selik <michael.selik(a)gmail.com> wrote: > >> On Mon, May 30, 2016 at 11:08 PM Steven D'Aprano <steve(a)pearwood.info> wrote: >> On Mon, May 30, 2016 at 06:16:26PM -0700, Guido van Rossum wrote: >> >> [...] >> > T = TypeVar('T') >> > x = Symbol('x') >> > >> > I'm sure this is not a new idea, but so far I've always thought that >> > this is pretty esoteric and the approach here is good enough. > >> This comes up a lot and it would be nice to clean it up. >> >> T = type('T', bases, ns) >> Record = namedtuple('Record', fields) > > I'm not enthusiastic about cleaning it up. It feels appropriately awkward. I don't want my colleagues having too easy a time dynamically creating types or functions. > > You brought up decorators as an example, but I think that was more about how decorators tend to be used essentially as part of the function definition. They might dramatically alter the behavior of the function and yet appear as an afterthought without the decorator syntax. From the PEP 318, "This becomes less readable with longer [functions]". The motivation said nothing about avoiding typing the function name twice. > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/

2 1

Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects
by Matthew Tanous June 1, 2016

June 1, 2016

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Currently, the use of the (*) operator on a list is to duplicate a list by creating multiple references to the same object. While this works intuitively for immutable objects (like [True] * 5) as these immutable references are replaced when the list is assigned to, it makes the operator nigh unusable for mutable objects. The most obvious case is when the operator is duplicated in a sequence like this: arr = [[True] * 5] * 5 This does not create a matrix-like arrangement of the immutable truth variable, but instead creates a list of 5 references to the same list, such that a following assignment like arr[2][3] = False will not change just that one index, but every 4th element of each list in the outer list. This also makes the sequence construction using a mutable type a problem. For example, assume a class Foo: class Foo: def __init__(self): self.val = True def set(self): self.val = False def __repr__(self): return str(self.val) If I then use sequence repetition to create a list of these like so: arr = [Foo()] * 5 This will create a list of references to the same Foo instance, making the list construction itself effectively meaningless. Running the set() method on any of the instances in the list is the same as running it on all the instances in the list. It is my opinion that the sequence repetition operator should be modified to make copies of the objects it is repeating, rather than copying references alone. I believe this would both be more intuitive from a semantic point of view and more useful for the developer. This would change the operator in a way that is mostly unseen in current usage ([5] * 3 would still result in [5, 5, 5]) while treating mutable nesting in a way that is more understandable from the apparent intent of the syntax construction. Reference regarding previous discussion: https://bugs.python.org/issue27135 -----BEGIN PGP SIGNATURE----- iQEcBAEBCAAGBQJXTSDZAAoJEF14rZEhZ/cMd24H/1p24+EYIALc7pBR5qbGpW20 oxHUWGfVaERizkhvuDAbO/n5sXUB5QHbh6MMwe2tn3TCWLstnvRhvJDR9ahKx7gm EChB4sIAKM/npUQge6ljLqP61m88p7LpnIVV6gF4PC0Wkyz8g2iSMjVwFv4XEBYZ /PNWXa4QLlNmqksrcQ7pYKZObYSjU8lNAEsCmtRy8PbBTvWq2f+YB9kcc79byFIs W0bhSI7x2iaicU24UC7FJAo4bSFKNZ8LDSEMZhu7gWhFxJ7wVsyxk6/RrZkptdCx z/DNqo9/Ggs4UJ9vo4cfCoX0723bejT0VG1K/EuYxWAXYOlNuICYIMQVNiZhkoo= =2/BD -----END PGP SIGNATURE-----

11 13

Re: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects
by Matthew Tanous May 31, 2016

May 31, 2016

I could see the use of the operator '@' to construct a matrix, perhaps. It is more limited though (essentially a 2D version of sequence repetition). Semantically, it would seem to me intuitive that [Foo()] * N would create a list of N Foos, not a list of N references to the same Foo. I can see that people who are used to how it actually works now would have issues with such a change in the semantics, however. Didn't consider the zip(*[it]*3) case being broken, I admit.

1 0