Mailman 3 June 2016 - Python-ideas

Chaining asyncio futures?
by Mark E. Haase 30 Jun '16

30 Jun '16

Other async frameworks have constructs similar to asyncio.Future, e.g. JavaScript has Promise[1] and Dart has a Completer/Future pair[2][3]. (For the purpose of this e-mail, I'll call the generic concept a "Future".) JavaScript and Dart allow futures to be chained, such that one Future's callback is invoked only when the other Future completes. For example, consider an API that fetches a string from a URL and returns it, but also has an MRU cache. This API returns a Future because the caller does not know in advance if the string to be retrieved is cached or not. If the string is in cache, then the Future can complete immediately, but if the string it not in cache, then the string needs to be fetched over the network, perhaps by calling some API that returns a Future string. In this case, it would be nice to return a Future string that is chained to the HTTP request's future string.[4] I tried doing this with asyncio.Future: import asyncio f1 = asyncio.Future() f2 = asyncio.Future() f2.add_done_callback(lambda f: print('done: {}'.format(f.result()))) f2.set_result(f1) f1.set_result('hello world!') loop = asyncio.get_event_loop() loop.run_until_complete(f2) Output: done: <Future finished result='hello world!'> This does not make f2 dependent on f1; it immediately sets f2's result to repr(f1). It's easy to modify the example so that f1's callback explicitly sets f2's result. Abstracting a bit, we can write a chain_futures() function. def chain_futures(a, b): a.add_done_callback(lambda _: b.set_result(a.result())) f3 = asyncio.Future() f4 = asyncio.Future() f4.add_done_callback(lambda f: print('done: {}'.format(f.result()))) chain_futures(f3, f4) f3.set_result('hello world!') loop.run_until_complete(f4) Output: done: hello world! Wouldn't this be a nice feature in asyncio.Future? E.g. instead of `chain_futures(f3, f4)`, either `f3.set_result(f4)` or something more explicit like `f3.set_result_from(f4)`. [1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… [2] https://api.dartlang.org/stable/1.17.1/dart-async/Completer-class.html [3] https://api.dartlang.org/stable/1.17.1/dart-async/Future-class.html [4] Why not return the HTTP request Future directly? That feels hacky -- it's a leaky abstraction.

2 1

Re: [Python-ideas] Submitting a job to an asyncio event loop
by Vincent Michel 27 Jun '16

27 Jun '16

I did remove it, because the feature has been implemented after the discussion: https://docs.python.org/3/library/asyncio-task.html?highlight=run_coroutine… https://docs.python.org/3/library/asyncio-dev.html#asyncio-multithreading https://github.com/python/asyncio/pull/273 2016-06-27 22:53 GMT+02:00 David Ford <firefighterblu3(a)gmail.com>: > Vincent, your github repo referenced here is 404. did you rename/move it? > > > > On Saturday, September 26, 2015 at 2:29:43 PM UTC, Vincent Michel wrote: >> >> Hi, >> >> I noticed there is currently no standard solution to submit a job from a >> thread to an asyncio event loop. >> >> Here's what the asyncio documentation says about concurrency and >> multithreading: >> >> > To schedule a callback from a different thread, the >> > BaseEventLoop.call_soon_threadsafe() method should be used. >> > Example to schedule a coroutine from a different thread: >> > loop.call_soon_threadsafe(asyncio.async, coro_func()) >> >> The issue with this method is the loss of the coroutine result. >> >> One way to deal with this issue is to connect the asyncio.Future returned >> by async (or ensure_future) to a concurrent.futures.Future. It is then >> possible to use a subclass of concurrent.futures.Executor to submit a >> callback to an asyncio event loop. Such an executor can also be used to set >> up communication between two event loops using run_in_executor. >> >> I posted an implementation called LoopExecutor on GitHub: >> https://github.com/vxgmichel/asyncio-loopexecutor >> The repo contains the loopexecutor module along with tests for several use >> cases. The README describes the whole thing (context, examples, issues, >> implementation). > > >> >> [...]

1 0

Re: [Python-ideas] Statically checking purity of functions
by CFK 22 Jun '16

22 Jun '16

Why does the checking have to be static? How about a 'trust but verify' approach? If a function is marked pure, python trusts that it is, but turns on code to check for side effects at runtime. If impure code is executed in a pure function, then an exception is raised. That will deal with the weird corner cases brought up earlier, at the cost of not being able to statically type check code. It will also deal with old code that wasn't marked, but happens to be pure. Not sure how to write the checks though; maybe with Perl's data tainting approach? Thanks, Cem Karan On Jun 22, 2016 8:39 AM, "Joonas Liik" <liik.joonas(a)gmail.com> wrote: On 22 June 2016 at 13:23, Jukka Lehtosalo <jlehtosalo(a)gmail.com> wrote: > On Fri, Jun 10, 2016 at 11:57 AM, Christoph Groth < christoph(a)grothesque.org> > wrote: >>> >>> This means that any pure function would have to accept as parameters >>> everything it operates on, and you'd need to error on any call to a pure >>> function with impure arguments. >> >> >> You mean because one can never be sure about the purity of global >> variables? That's true, but exactly to the same extent that one cannot be >> sure about the typing annotations of global variables. >> >> I believe that such "weak" checking for purity would still be useful, just >> as the equally weak verification of typing by mypy is useful. > > > I'm late to the discussion, but I agree with Christoph. I doubt purity > checking can reliably enforce anything without making it impractically > restrictive, but even best-effort, partial purity checking could be useful, > similar to how mypy is useful. A checker could catch many (but not all) > purity violations and purity annotations would make programs easier to > reason about for humans and tools. I've thought about this a bit in the past > -- mostly in the context of mypy. I'm not sure whether purity checking would > be more than marginally useful, but at least in some use cases it might be a > valuable a tool. For example, purity checking might help with reasoning > about concurrency and parallelism. However, writing a purity checking tool > for Python that is flexible enough that many programmers would actually want > to use it sounds like a difficult (research?) project. > > Here are some practical considerations: > > 1) We'd need to annotate library functions with respect to purity, as Python > derives much of its value from the available libraries, and we'd want pure > functions to be able to use (pure) libraries. Perhaps stubs in typeshed > would have purity annotations for individual functions. Alternatively, there > could be a typeshed-like repository for purity information only, and a tool > could use information from both repositories. The safest thing would be to > default to impure, unless a function is explicitly annotated as pure. > > 2) Type annotations would likely be needed for practical checking of purity, > as otherwise things like method calls would be hard to check for purity. > Thus purity checking would be a stricter form of static type checking. > > 3) For consistency with PEP 484 style gradual typing, we may want to allow > functions without annotations to be called from pure functions. Tools might > make this user-configurable, as this is clearly a big hole in a purity > checking system. > > 4) It may be necessary to allow impure operations to be used within a pure > function as long the impurity won't be visible outside the function. For > example, perhaps this function should be fine, as an external observer can't > tell whether the function uses pure operations only: > > def squares(n: int) -> List[int]: > a = [] > for i in range(n): > a.append(i * i) > return a > > I`m not sure if I know how to implement this in a useful way, though. I > believe that there is some prior work in this area (not specific to Python). > > Jukka > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ we could have eg: def parallel_map(f, data): if is_pure(f): # <impl that does special optimisations to take advantage of the fact f is truly pure> # special case that uses light weight threads or sth (maybe can run bits of it GIL free or sth, less overhead or sth) # its ok to share immutable data else: #<impl that has no issue with impure f> # eg: multiprocessing process pools.. #its ok to mutate everything becauuse we no longer truly share anything. in other words, even if only a small subset of code can be proven pure it can be useful to help optimise code (in this example to avoid locking by knowing that no 2 parrallel instances of the callback can mock with mutually shared state) for some scenarios it - might be - ok even if we leave the proving to the programmer. (eg a C extension could implement the optimised parallel map in this example and a user defined decorator is enough to shove a flag on to the function f to trigger the optimised code path) if this it is not truly pure tho despite the assertion it might cause issues so in that case it probably is not something that should ship with python by default? _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Statically checking purity of functions
by Christoph Groth 22 Jun '16

22 Jun '16

Hello, As far as I can tell the following has not been discussed yet on the searchable internet. It may seem that my idea concerns only mypy, but since "typing" is part of the stdlib, I believe it is appropriate for this mailing list. Even though Python is a very dynamic language, in practice many routines (when called with arguments of the intended type) are pure in the functional programming sense, i.e. they have no observable side effects and the result reproducibly depends on the arguments. I think there could be many interesting applications of knowing whether a function is pure or not. Example: Long-running computations often need to be parallelized. There are plenty of ways to parallelize Python code (think of concurrent.futures or ipyparallel) and most of them make implicit assumptions that some functions are pure. Another example: if the purity of functions would be known, Jupyter notebooks could automatically deduce the side effects of code cells and thus calculate their dependencies on each other. Changes to a cell could thus invalidate other (dependent) cells. With plain Python (without type annotations) there is no way to verify the purity of a function as has been well explained here: http://stackoverflow.com/a/31664102 But couldn't this be done with a static type checker like mypy? If mypy would track functions as pure or not pure, it should be able to verify the claimed purity of any new function. For example, the following function would pass the test, @pure def add(a: int, b: int) -> int: return a + b while the next one would produce an error @pure def add_random(a: int) -> int: return a + random.randrange(10) because the static checker would know that random.randrange() is not pure (it modifies the state of the random module). Am I overseeing any problems with the above? Christoph

11 14

Re: [Python-ideas] package management is not python-dev's problem :-(
by Victor Stinner 21 Jun '16

21 Jun '16

Le 8 juin 2016 1:02 PM, "Thomas Güttler" <guettliml(a)thomas-guettler.de> a écrit : > This statement make me sad. Why is this not python-dev's problem? It was the case in the past, but most discussions ended with bikeshedding, trolls, etc. instead of trying to discuss solutions to fix issues. Since distutils-sig was created, many PEP were aproved and pip evolved faster than before. Victor

1 0

Make Python front end to the C math module
by Steven D'Aprano 21 Jun '16

21 Jun '16

Currently the math module is written in C, not Python. This greatly increases the barrier to contributions to math. I suggest that we treat the existing math module as an accelerator module: - rename math.so to _math.so; (or whatever platform-specific name it uses) - add math.py with something like the following content: from _math import * and we're done! That will make it easier for people to contribute pure Python functions in the future. Currently all math functions, regardless of how trivial or difficult, must be written in C. With this change, they can be written in Python, and only written in C if necessary. For the avoidance of doubt, there is NO requirement for pure-Python implementations of the existing math functions, or even future ones. Future functions can go straight into the C implementation is desired. This is to allow future functions the opportunity to be written in Python. -- Steve

5 5

collections.abc.Stream
by Ram Rachum 18 Jun '16

18 Jun '16

Is there a way to check in Python whether an object is file-like? (Like `open`, `StringIO`, etc.) I would think that an abc in collections.abc will be the standard solution, like we have for so many other data types, but I can't find one. Thanks, Ram.

7 13

Why aren't slices hashable?
by Ed Minnix 18 Jun '16

18 Jun '16

Hello, I was experimenting in the repl and I noticed something odd: slices are not hashable. I was wondering why this would be since, I would think slices are immutable, and most immutable objects are hashable. Thanks, Ed

2 1

Object for accessing identifiers/names
by Koos Zevenhoven 15 Jun '16

15 Jun '16

Inspired by the discussions regarding Guido's "M atch statement brainstorm " and " Quick idea: defining variables from functions..." threads, here's an idea regarding names/identifiers. Currently, there is no direct way to work with variable names from within Python. Yes, you can fiddle with __dict__s and locals() and globals(), but there is no convenient general way. To solve this, there could be a way (probably new syntax) for creating an object that can examine and manipulate a name binding conveniently. The object could be created for instance as follows: name_object = identifier some_name However, 'identifier' is so long that I'll use the keyword 'def' instead, regardless of whether it is optimal or not: name_obj = def some_name Now this name_obj thing could provide functionality like assigning to the name some_name, getting the assigned object, and determining whether something has been assigned to the name or not . The object would also be aware of the name 'some_name'. T he functionality might work as follows: bool( name_obj ) # True if something is assigned to some_name name_obj.name() # == 'some_name' name_obj .set(value) # equivalent to `some_name = value` name_obj.get() # equivalent to just `some_name` name_obj.unset() # like `del some_name` Then you could pass this to a function: func(name_obj) Now func will be able to assign to the variable some_name, but with some_name referring to the scope where `name_obj = def some_name` was executed. This is similar to things that can be done with closures. This would also allow things like: if def some_name: #do stuff if the name some_name is bound to something or if not def some_name: some_name = something() - Koos

4 4

Wild idea about mutability
by Rob Cliffe 15 Jun '16

15 Jun '16

I was prompted to post this after seeing comments on "Proposal to change List Sequence Repetition (*) so it is not useless for Mutable Objects" that you cannot in general tell whether an object is mutable or not. Well, why shouldn't you be able to tell? Python is superb at introspection, but ISTM there's a gap here. Here's an idea for Python N (N>=4): Every object has a boolean __mutable__ attribute. It would e.g. be False for ints and strings, True by default for objects that can be mutable, such as lists and dicts. It could be an optional extra argument to mutable object constructors (so you have the option of making them immutable): L = list(somesequence, mutable=False) There could also be a syntax for list/set literals and dictionary displays to indicate that they should be immutable (just as we preface literal strings with 'r' to indicate raw strings). I'm not sure what; I thought of f[1,2,3] for a literal immutable ("frozen") list, but that's already valid syntax. You can "freeze" a mutable object by changing its __mutable__ attribute from True to False. You are not allowed to change it from False to True, nor to mutate an object that has it False. I am sure there must be advantages in being able to tell if an object is mutable or not. (Perhaps "hashable" equals "not mutable" ?) Now: You don't need the frozenset class. You just have a set with __mutable__ = False. And you can freeze dicts (classes? modules?) etc., to make sure users of your code don't mess with them. And (fasten your safety belts): You don't need tuples. You just have a list with __mutable__ = False. And you don't have to explain to beginners one of the most FAQ "Why do we need lists and tuples?" Best wishes (and sorry I can't seem to get my e-mail out of double vertical spacing mode), Rob Cliffe

8 18