Mailman 3 June 2016 - Python-ideas

Allow __bytes__ to return NotImplemented
by Émanuel Barry June 7, 2016

June 7, 2016

I'm currently writing a class that can take either a str of bytes object, and is meant to work with either type. For my purpose, it makes sense to return the underlying bytes object when calling bytes() on it. However, defining __bytes__ on the class means that I also have to return bytes even if the underlying object is a str. The obvious way to fix this is to define two different classes, one which handles str and one which handles bytes. However, I was wondering if returning NotImplemented … [View More]

4 6

1 + True = 2
by Giampaolo Rodola' June 6, 2016

June 6, 2016

>>> 1 + True 2 >>> 1 + False 1 >>> I bumped into this today by accident and I can't recall ever being aware of this. Why isn't this a TypeError in Python 3? -- Giampaolo - http://grodola.blogspot.com

7 8

Re: [Python-ideas] Exposing regular expression bytecode
by Jonathan Goble June 5, 2016

June 5, 2016

OK, thanks for the comments, everyone. I'm glad to hear that people generally think this is a useful idea. Some specific replies: On Tue, Feb 16, 2016 at 4:22 AM, Chris Angelico <rosuav(a)gmail.com> wrote: > For what it's worth, I read your post with interest, but didn't have > anything substantive to reply - mainly because I don't use regexes > much. But it would be rather cool to be able to decompile a regex. > Imagine a regex pretty-printer: compile an arbitrary string, … [View More]and if > it's a valid regex, decompile it to a valid source code form, using > re.VERBOSE. That could help _hugely_ with debugging, if the trick can > be pulled off. > > ChrisA That's exactly the type of tools I envision being made available by third parties. Depending on how much I get invested into this project, I may even write such a tool myself (though that's not guaranteed). On Tue, Feb 16, 2016 at 4:55 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > Sorry. I don't personally have any issue with the proposal, and it > sounds like a reasonable idea. I don't think it's likely to be > *hugely* controversial - although it will likely need a little care in > documenting the feature to ensure that we are clear that there's no > guarantees of backward compatibility that we don't want to commit to > on the newly - exposed data. And we should also ensure that by > exposing this information, we don't preclude changes such as the > incorporation of the regex module (I don't know if the regex module > has a bytecode implementation like the re module does). The regex implementation is indeed something I would need to investigate here, and will do so before I go too far. > The next step is probably simply to raise a tracker issue for this. I > know you said you have little C experience, but the big problem is > that it's unlikely that any of the core devs with C experience will > have the time or motivation to code up your idea. So without a working > patch, and someone willing and able to respond to comments on the > patch, it's not likely to progress. Tracker issue is already filed: http://bugs.python.org/issue26336 I actually filed the issue before I realized that the mailing lists were a better place to discuss it. > But if you are willing to dig into Python's C API yourself (and it > sounds like you are) there are definitely people who will help you. > You might want to join the core mentorship list (see > http://pythonmentors.com/) where you should get plenty of assistance. > This proposal sounds like a great "beginner" task, as well - so even > if you don't want to implement it yourself, still put it on the > tracker, and mark it as an "easy" change, and maybe some other > newcomer who wants a task to help them learn the C API will pick it > up. I'll look into the mentorship list; thanks for the link. As for marking it "easy", I don't seem to have the necessary permissions to change the Keywords field; perhaps you or someone else can set that flag for me? If so, I'd appreciate it. :-) > Hope that helps - thanks for the suggestion and sorry if it seems like > no-one was interested at first. It's an unfortunate fact of life > around here that things *do* take time to get people's interest. You > mention patience in one of your messages - that's definitely something > you'll need to cultivate, I'm afraid... :-) Patience is something I've been working on since I was a little kid. I'm 29 years old now, and it still eludes me from time to time. But yes, it's something I'll have to work on. :-P Also, I received a small patch off-list from Petr Viktorin implementing a getter for the code list (thanks, Petr). I'll need to test it, but from the little I know of the C API it looks like it will get me started in the right direction. Assuming that works, what's left is a public constructor for the regex type (to enable optimizers), a dis-like module, and docs and tests. I don't think this would be major enough to require a PEP, but of course being new here, I'm open to being told I'm wrong. :-) [View Less]

4 7

solving multi-core Python
by Eric Snow June 3, 2016

June 3, 2016

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them. This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources. Feedback appreciated! Offers to help even more so! :) -eric -------- Python's multi-core story is murky at best. … [View More]Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (i.e. parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic). Currently we have several concurrency models represented via threading, multiprocessing, asyncio, concurrent.futures (plus others in the cheeseshop). However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. (See Dave Beazley's talk at PyCon US 2015.) This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code. This is not a new topic. For a long time many have clamored for death to the GIL. Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance. Furthermore, removing the GIL is perhaps an obvious solution but not the only one. Others include Trent Nelson's PyParallels, STM, and other Python implementations.. Proposal ======= In some personal correspondence Nick Coghlan, he summarized my preferred approach as "the data storage separation of multiprocessing, with the low message passing overhead of threading". For Python 3.6: * expose subinterpreters to Python in a new stdlib module: "subinterpreters" * add a new SubinterpreterExecutor to concurrent.futures * add a queue.Queue-like type that will be used to explicitly share objects between subinterpreters This is less simple than it might sound, but presents what I consider the best option for getting a meaningful improvement into Python 3.6. Also, I'm not convinced that the word "subinterpreter" properly conveys the intent, for which subinterpreters is only part of the picture. So I'm open to a better name. Influences ======== Note that I'm drawing quite a bit of inspiration from elsewhere. The idea of using subinterpreters to get this (more) efficient isolated execution is not my own (I heard it from Nick). I have also spent quite a bit of time and effort researching for this proposal. As part of that, a number of people have provided invaluable insight and encouragement as I've prepared, including Guido, Nick, Brett Cannon, Barry Warsaw, and Larry Hastings. Additionally, Hoare's "Communicating Sequential Processes" (CSP) has been a big influence on this proposal. FYI, CSP is also the inspiration for Go's concurrency model (e.g. goroutines, channels, select). Dr. Sarah Mount, who has expertise in this area, has been kind enough to agree to collaborate and even co-author the PEP that I hope comes out of this proposal. My interest in this improvement has been building for several years. Recent events, including this year's language summit, have driven me to push for something concrete in Python 3.6. The subinterpreter Module ===================== The subinterpreters module would look something like this (a la threading/multiprocessing): settrace() setprofile() stack_size() active_count() enumerate() get_ident() current_subinterpreter() Subinterpreter(...) id is_alive() running() -> Task or None run(...) -> Task # wrapper around PyRun_*, auto-calls Task.start() destroy() Task(...) # analogous to a CSP process id exception() # other stuff? # for compatibility with threading.Thread: name ident is_alive() start() run() join() Channel(...) # shared by passing as an arg to the subinterpreter-running func # this API is a bit uncooked still... pop() push() poison() # maybe select() # maybe Note that Channel objects will necessarily be shared in common between subinterpreters (where bound). This sharing will happen when the one or more of the parameters to the function passed to Task() is a Channel. Thus the channel would be open to the (sub)interpreter calling Task() (or Subinterpreter.run()) and to the new subinterpreter. Also, other channels could be fed into such a shared channel, whereby those channels would then likewise be shared between the interpreters. I don't know yet if this module should include *all* the essential pieces to implement a complete CSP library. Given the inspiration that CSP is providing, it may make sense to support it fully. It would be interesting then if the implementation here allowed the (complete?) formalisms provided by CSP (thus, e.g. rigorous proofs of concurrent system models). I expect there will also be a _subinterpreters module with low-level implementation-specific details. Related Ideas and Details Under Consideration ==================================== Some of these are details that need to be sorted out. Some are secondary ideas that may be appropriate to address in this proposal or may need to be tabled. I have some others but these should be sufficient to demonstrate the range of points to consider. * further coalesce the (concurrency/parallelism) abstractions between threading, multiprocessing, asyncio, and this proposal * only allow one running Task at a time per subinterpreter * disallow threading within subinterpreters (with legacy support in C) + ignore/remove the GIL within subinterpreters (since they would be single-threaded) * use the GIL only in the main interpreter and for interaction between subinterpreters (and a "Local Interpreter Lock" for within a subinterpreter) * disallow forking within subinterpreters * only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables) * object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters + only allow 1 subinterpreter to have any refcounts to an object (except for channels) * only allow immutable objects to be shared between subinterpreters * for better immutability, move object ref counts into a separate table * freeze (new machinery or memcopy or something) objects to make them (at least temporarily) immutable * expose a more complete CSP implementation in the stdlib (or make the subinterpreters module more compliant) * treat the main interpreter differently than subinterpreters (or treat it exactly the same) * add subinterpreter support to asyncio (the interplay between them could be interesting) Key Dependencies ================ There are a few related tasks/projects that will likely need to be resolved before subinterpreters in CPython can be used in the proposed manner. The proposal could implemented either way, but it will help the multi-core effort if these are addressed first. * fixes to subinterpreter support (there are a couple individuals who should be able to provide the necessary insight) * PEP 432 (will simplify several key implementation details) * improvements to isolation between subinterpreters (file descriptors, env vars, others) Beyond those, the scale and technical scope of this project means that I am unlikely to be able to do all the work myself to land this in Python 3.6 (though I'd still give it my best shot). That will require the involvement of various experts. I expect that the project is divisible into multiple mostly independent pieces, so that will help. Python Implementations =================== They can correct me if I'm wrong, but from what I understand both Jython and IronPython already have subinterpreter support. I'll be soliciting feedback from the different Python implementors about subinterpreter support. C Extension Modules ================= Subinterpreters already isolate extension modules (and built-in modules, including sys). PEP 384 provides some help too. However, global state in C can easily leak data between subinterpreters, breaking the desired data isolation. This is something that will need to be addressed as part of the effort. [View Less]

26 130

pattern matching proof-of-concept
by Michael Selik June 3, 2016

June 3, 2016

My original dict unpacking proposal was very short and lacked a motivating usage. Toy examples made my proposal look unnecessarily verbose and suggested obvious alternatives with easy current syntax. Nested/recursive unpacking is much more troublesome, especially when combined with name-binding. I wrote an example to compare my proposal with current syntax. Example usage. https://github.com/selik/destructure/blob/master/examples/fips.py Implementation. https://github.com/selik/destructure/… [View More]

4 7

Match statement brainstorm
by Guido van Rossum June 2, 2016

June 2, 2016

On Wed, May 18, 2016 at 8:17 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 19 May 2016 at 08:53, Guido van Rossum <guido(a)python.org> wrote: >> The one thing that Python doesn't have (and mypy doesn't add) would be >> a match statement. The design of a Pythonic match statement would be >> an interesting exercise; perhaps we should see how far we can get with >> that for Python 3.7. > > If it's syntactic sugar for a particular variety of if/… [View More]elif/else > statement, I think that may be feasible, but you'd presumably want to > avoid the "Can we precompute a lookup table?" quagmire that doomed PEP > 3103's switch statement. > > That said, for the pre-computed lookup table case, whatever signature > deconstruction design you came up with for a match statement might > also be usable as the basis for a functools.multidispatch() decorator > design. Let's give up on the pre-computed lookup table. Neither PEP 3103 nor PEP 275 (to which it compares itself) even gets into the unpacking part, which would be the interesting thing from the perspective of learning from Sum types and matching in other languages. Agreed on the idea of trying to reuse this for multidispatch! A few things that might be interesting to explore: - match by value or set of values (like those PEPs) - match by type (isinstance() checks) - match on tuple structure, including nesting and * unpacking (essentially, try a series of destructuring assignments until one works) - match on dict structure? (extension of destructuring to dicts) - match on instance variables or attributes by name? - match on generalized condition (predicate)? The idea is that many of these by themselves are better off using a classic if/elif/else structure, but a powerful matching should be allowed to alternate between e.g. destructuring matches and value or predicate matches. IIUC Haskell allows pattern matches as well as conditions, which they seem to call guards or where-clauses (see https://www.haskell.org/tutorial/patterns.html, or http://learnyouahaskell.com/syntax-in-functions if you like your pages more colorful). Or maybe we should be able to combine structure matches with guards. I guess the tuple structure matching idea is fairly easy to grasp. The attribute idea would be similar to a duck-type check, though more emphasizing data attributes. It would be nice if we could write a match that said "if it has attributes x and y, assign those to local variables p and q, and ignore other attributes". Strawman syntax could be like this: def demo(arg): switch arg: case (x=p, y=q): print('x=', p, 'y=', q) case (a, b, *_): print('a=', a, 'b=', b) else: print('Too bad') Now suppose we had a Point defined like this: Point = namedtuple('Point', 'x y z') and some variables like this: a = Point(x=1, y=2, z=3) b = (1, 2, 3, 4) c = 'hola' d = 42 then we could call demo with these variables: >>> demo(a) x= 1 y= 2 >>> demo(b) a= 1 b= 2 >>> demo(c) a= h b= o >>> demo(d) Too bad >>> (Note the slightly unfortunate outcome for 'hola', but that's what a destructuring assignment would do. Water under the bridge.) Someone else can try to fit simple value equality, set membership, isinstance, and guards into that same syntax. -- --Guido van Rossum (python.org/~guido) [View Less]

24 78

Declaring immutability (was Re: Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects)
by Nick Coghlan June 1, 2016

June 1, 2016

On 31 May 2016 07:17, "Steven D'Aprano" <steve(a)pearwood.info> wrote: > > On Tue, May 31, 2016 at 01:36:31PM +0000, Joseph Martinot-Lagarde wrote: > > > I can only agree here. Even today, despite knowing the fact, it's > > > causing some headaches in some cases. > > > > How about raising an exception if mutable objects are in the list ? > > -1 > > It's a gratuitous breakage that cannot solve the problem, because you > cannot tell in … [View More]

2 1

async ProcessPoolExecutor
by Christoph Groth June 1, 2016

June 1, 2016

Hi, While working on a parallel numerical quadrature routine that features speculative evaluation, I got interested in async programming in Python. Most async programming seems to be done for network applications, but it seems to me that controlling complex parallel computations could be also a promising application. The asyncio package has the run_in_executor function that seemed to be just what I need. When looking at how it works I noticed that dispatching tasks from an asyncio event loop … [View More]to a concurrent.futures.ProcessPoolExecutor involves a lot of wrapping, locking and hence complexity and inefficiency. Shouldn't it be possible to write an async version of ProcessPoolExecutor? I did just this and the results seem quite promising. Consider the following program: **************************************************************** from concurrent import futures import time def work(x): return x**2 def main(): executor = futures.ProcessPoolExecutor(48) now = time.time() fs = [executor.submit(work, x) for x in range(10000)] print(sum(f.result() for f in futures.as_completed(fs))) print(time.time() - now) if __name__ == "__main__": main() **************************************************************** It takes more than 5 seconds on my machine. It becomes even worse (7 seconds) when asyncio with run_in_executor is used. Using the async executor, the following variant of the above script runs in 2 seconds: **************************************************************** import asyncio import time import aexecutor def work(x): return x**2 async def main_coro(): async with aexecutor.ProcessPoolExecutor(48) as executor: now = time.time() fs = [executor.submit(work, x) for x in range(10000)] acc = 0 for f in asyncio.as_completed(fs): acc += await f print(acc) print(time.time() - now) def main(): loop = asyncio.get_event_loop() loop.run_until_complete(main_coro()) if __name__ == "__main__": main() **************************************************************** That difference becomes less when the size of the worker pool gets reduced, but it's significant even for a few workers. And I am actually interested in controlling a huge number of workers (I do computations on a cluster with 20 nodes of 48 cores each). What's more, the source code of the executor got reduced from around 1100 lines to 600. Please see the attachment for the source of the "aexecutor" package as well as the three test programs. (All to be run with Python 3.5.) I expect that the async executor will be useful for me. Not just for the gain in efficiency, but also for having a simple executor to play with. (I think I that I will need futures with a priority and I might also want to write a MPI version of it.) Perhaps this is something that could be interesting for the wider python community? Bear in mind that this is just a quick hack and my first project that uses asyncio. It's most likely buggy and non-portable (it does work on Unix). It can be probably simplified further and there must be a way to improve the inner loop of the scheduler coroutine which currently looks like this: **************************************************************** reader = result_queue._reader event = asyncio.Event(loop=loop) loop.add_reader(reader._handle, event.set) while True: (...) while True: await event.wait() if reader.poll(0): break else: event.clear() result_item = reader.recv() event.clear() (...) **************************************************************** Any ideas? Christoph [View Less]

1 1

Re: [Python-ideas] Quick idea: defining variables from functions that take the variable name
by Sjoerd Job Postmus June 1, 2016

June 1, 2016

I agree that cleaning up is not really needed. I think the problem is that people feel bad about having to type things twice and immediately think "It's violating DRY". And some are quite adamant about not violating DRY. Fact of the matter: on nearly all relevant scales, this "DRY violation" does not matter, in fact, I would not even want to call it a violation. DRY is about having 1 representative location for knowledge. The name of a variable (1) hardly counts as knowledge and (2) is also … [View More]used at other locations (use-sites). Is repeating the variable name that much of a deal? Granted, I mostly only use namedtuple of all the given examples. And I use that sparingly. Are `TypeVar` and `Symbol` used more often? At least for namedtuple, we could do some meta-classy logic to allow for class Point(namedtuple): _fields = ('x', 'y') And for namedtuple it makes sense. For TypeVar it might too? class T(TypeVar): pass Symbol? Not so much, I think. I'm not eager to see "improvements" in this area, as I do not see it as a problem, except for those with an overly strict view on DRY. Any new syntax/semantics for this would need to bring a real obvious improvement for the general case, and I doubt that will happen. The one suggestion I have heard that remotely makes sense (and it was not even in this thread!) is the concept of decorators for variables @Something x = 5 would become x = Something('x', 5) And even that idea is bad: what if you want multiple arguments? Would you need a tuple? What if you don't need any arguments? (e.g. Symbol). After seeing all the suggestions so far, I'm not convinced by any of them that they are an improvement. So unless there's a suggestion for syntax/semantics that is reasonable to become the only **obvious** way to do it, I'd say nay. > On 1 Jun 2016, at 02:52, Michael Selik <michael.selik(a)gmail.com> wrote: > >> On Mon, May 30, 2016 at 11:08 PM Steven D'Aprano <steve(a)pearwood.info> wrote: >> On Mon, May 30, 2016 at 06:16:26PM -0700, Guido van Rossum wrote: >> >> [...] >> > T = TypeVar('T') >> > x = Symbol('x') >> > >> > I'm sure this is not a new idea, but so far I've always thought that >> > this is pretty esoteric and the approach here is good enough. > >> This comes up a lot and it would be nice to clean it up. >> >> T = type('T', bases, ns) >> Record = namedtuple('Record', fields) > > I'm not enthusiastic about cleaning it up. It feels appropriately awkward. I don't want my colleagues having too easy a time dynamically creating types or functions. > > You brought up decorators as an example, but I think that was more about how decorators tend to be used essentially as part of the function definition. They might dramatically alter the behavior of the function and yet appear as an afterthought without the decorator syntax. From the PEP 318, "This becomes less readable with longer [functions]". The motivation said nothing about avoiding typing the function name twice. > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

2 1

Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects
by Matthew Tanous June 1, 2016

June 1, 2016

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Currently, the use of the (*) operator on a list is to duplicate a list by creating multiple references to the same object. While this works intuitively for immutable objects (like [True] * 5) as these immutable references are replaced when the list is assigned to, it makes the operator nigh unusable for mutable objects. The most obvious case is when the operator is duplicated in a sequence like this: arr = [[True] * 5] * 5 This does not … [View More]

11 13

Re: [Python-ideas] Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects
by Matthew Tanous June 1, 2016

June 1, 2016

I could see the use of the operator '@' to construct a matrix, perhaps. It is more limited though (essentially a 2D version of sequence repetition). Semantically, it would seem to me intuitive that [Foo()] * N would create a list of N Foos, not a list of N references to the same Foo. I can see that people who are used to how it actually works now would have issues with such a change in the semantics, however. Didn't consider the zip(*[it]*3) case being broken, I admit.

1 0