Mailman 3 July 2019 - Python-ideas

Please consider adding of functions file system operations to pathlib
by George Fischhof 16 Nov '19

16 Nov '19

Good day all, as a continuation of thread "OS related file operations (copy, move, delete, rename...) should be placed into one module" https://mail.python.org/pipermail/python-ideas/2017-January/044217.html please consider making pathlib to a central file system module with putting file operations (copy, move, delete, rmtree etc) into pathlib. BR, George

3 3

Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
by Raymond Hettinger 23 Oct '19

23 Oct '19

> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote: > On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor <peter.ed...(a)gmail.com> wrote: >> So some more humble proposals would be: >> >> 1) An initializer to itertools.accumulate >> functools.reduce already has an initializer, I can't see any controversy to adding an initializer to itertools.accumulate > > See if that's accepted in the bug tracker. It did come-up once but was closed for a number reasons including lack of use cases. However, Peter's signal processing example does sound interesting, so we could re-open the discussion. For those who want to think through the pluses and minuses, I've put together a Q&A as food for thought (see below). Everybody's design instincts are different -- I'm curious what you all think think about the proposal. Raymond --------------------------------------------- Q. Can it be done? A. Yes, it wouldn't be hard. _sentinel = object() def accumulate(iterable, func=operator.add, start=_sentinel): it = iter(iterable) if start is _sentinel: try: total = next(it) except StopIteration: return else: total = start yield total for element in it: total = func(total, element) yield total Q. Do other languages do it? A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes. * http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulate.… * https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html * http://microapl.com/apl/apl_concepts_chapter5.html \+ 1 2 3 4 5 1 3 6 10 15 * https://reference.wolfram.com/language/ref/Accumulate.html * https://www.haskell.org/hoogle/?hoogle=mapAccumL Q. How much work for a person to do it currently? A. Almost zero effort to write a simple helper function: myaccum = lambda it, func, start: accumulate(chain([start], it), func) Q. How common is the need? A. Rare. Q. Which would be better, a simple for-loop or a customized itertool? A. The itertool is shorter but more opaque (especially with respect to the argument order for the function call): result = [start] for x in iterable: y = func(result[-1], x) result.append(y) versus: result = list(accumulate(iterable, func, start=start)) Q. How readable is the proposed code? A. Look at the following code and ask yourself what it does: accumulate(range(4, 6), operator.mul, start=6) Now test your understanding: How many values are emitted? What is the first value emitted? Are the two sixes related? What is this code trying to accomplish? Q. Are there potential surprises or oddities? A. Is it readily apparent which of assertions will succeed? a1 = sum(range(10)) a2 = sum(range(10), 0) assert a1 == a2 a3 = functools.reduce(operator.add, range(10)) a4 = functools.reduce(operator.add, range(10), 0) assert a3 == a4 a4 = list(accumulate(range(10), operator.add)) a5 = list(accumulate(range(10), operator.add, start=0)) assert a5 == a6 Q. What did the Python 3.0 Whatsnew document have to say about reduce()? A. "Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable." Q. What would this look like in real code? A. We have almost no real-world examples, but here is one from a StackExchange post: def wsieve(): # wheel-sieve, by Will Ness. ideone.com/mqO25A->0hIE89 wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2, 4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10] cs = accumulate(cycle(wh11), start=11) yield( next( cs)) # cf. ideone.com/WFv4f ps = wsieve() # codereview.stackexchange.com/q/92365/9064 p = next(ps) # 11 psq = p*p # 121 D = dict( zip( accumulate(wh11, start=0), count(0))) # start from sieve = {} for c in cs: if c in sieve: wheel = sieve.pop(c) for m in wheel: if not m in sieve: break sieve[m] = wheel # sieve[143] = wheel@187 elif c < psq: yield c else: # (c==psq) # map (p*) (roll wh from p) = roll (wh*p) from (p*p) x = [p*d for d in wh11] i = D[ (p-11) % 210] wheel = accumulate(cycle(x[i:] + x[:i]), start=psq) p = next(ps) ; psq = p*p next(wheel) ; m = next(wheel) sieve[m] = wheel

17 49

Why not accept lists or arbitrary iterables in str.endswith?
by Soni L. 14 Oct '19

14 Oct '19

I'm parsing configs for domain filtering rules, and they come as a list. However, str.endswith requires a tuple. So I need to use str.endswith(tuple(list)). I don't know the reasoning for this, but why not just accept a list as well?

3 6

Skip modules by default in star-import
by Serhiy Storchaka 19 Sep '19

19 Sep '19

From the description of the import statement in the language reference [1]: > The *public names* defined by a module are determined by checking the module's > namespace for a variable named ``__all__``; if defined, it must be a sequence > of strings which are names defined or imported by that module. The names > given in ``__all__`` are all considered public and are required to exist. If > ``__all__`` is not defined, the set of public names includes all names found > in the module's namespace which do not begin with an underscore character > (``'_'``). ``__all__`` should contain the entire public API. It is intended > to avoid accidentally exporting items that are not part of the API (such as > library modules which were imported and used within the module). As an example of names which should be excluded from the public API the reference mentions names of modules which were imported and used within the module. And they are perhaps the most common cases. It is often imported module names are the only non-underscored names which are imported by a star-import by accident. For example, modules re and sys were used in Lib/idlelib/editor.py but there were no imports for them. This worked because they were imported implicitly by `from tkinter import *`. [1] There is other case when a module name can be occurred in the module namespace: if it is a submodule. For example: >>> from xml.etree import * >>> ElementTree Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementTree' is not defined >>> ElementPath Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementPath' is not defined >>> import xml.etree.ElementTree >>> from xml.etree import * >>> ElementTree <module 'xml.etree.ElementTree' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementTree.py'> >>> ElementPath <module 'xml.etree.ElementPath' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementPath.py'> Names imported by a `from xml.etree import *` are depended on importing a xml.etree's submodule ElementTree. I do not think this is good. I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules. In these rare cases where modules should be imported in a star import it is not difficult to introduce `__all__`. [1] https://docs.python.org/3/reference/simple_stmts.html#the-import-statement [2] https://bugs.python.org/issue29446

9 16

Namespace context managers
by Batuhan Taskaya 10 Aug '19

10 Aug '19

I am proposing namespace context managers with implementing `__enter__` and `__exit__` on dict objects. It would make closures possible in python with a pythonic syntax. a = 4 namespace = {} with namespace: a = 3 assert a == 4 assert namespace["a"] == 3

15 32

Universal parsing library in the stdlib to alleviate security issues
by Nam Nguyen 09 Aug '19

09 Aug '19

Hello list, I sent an email to this list two or three months ago about the same idea. In that discussion, there were both skepticism and support. Since I had some time during the previous long weekend, I have made my idea more concrete and I thought I would try with the list again, after having run it through some of you privately. GOAL: To have some parsing primitives in the stdlib so that other modules in the stdlib itself can make use of. This would alleviate various security issues we have seen throughout the years. With that goal in mind, I opine that any parsing library for this purpose should have the following characteristics: #. Can be expressed in code. My opinion is that it is hard to review generated code. Code review is even more crucial in security contexts. #. Small and verifiable. This helps build trust in the code that is meant to plug security holes. #. Less evolving. Being in the stdlib has its drawback that is development velocity. The library should be theoretically sound and stable from the beginning. #. Universal. Most of the times we'll parse left-factored context-free grammars, but sometimes we'll also want to parse context-sensitive grammars such as short XML fragments in which end tags must match start tags. I have implemented a tiny (~200 SLOCs) package at https://gitlab.com/nam-nguyen/parser_compynator that demonstrates something like this is possible. There are several examples for you to have a feel of it, as well as some early benchmark numbers to consider. This is far smaller than any of the Python parsing libraries I have looked at, yet more universal than many of them. I hope that it would convert the skeptics ;). Finally, my request to the list is: Please debate on: 1) whether we want a small (even private, underscore prefixed) parsing library in the stdlib to help with tasks that are a little too complex for regexes, and 2) if yes, how should it look like? I also welcome comments (naming, uses of operator overloading, features, bikeshedding, etc.) on the above package ;). Thanks! Nam

12 31

For-expression/throwaway comprehension
by Eli Berkowitz 07 Aug '19

07 Aug '19

(This is my first time posting on any Python list; I've tried to search for this idea and didn't find it but if I looked in the wrong places/this has already been discussed I apologize and feel free to tell me!) Say you have a list and you want to perform some operation on each item in the list - but you don't need to store the result in a list. There are three simple ways of doing this, at least as far as I know: ([print(item)] could be any expression, just using it as an example) ``` lst = [1, 2, 3, 4] #1 for item in lst: print(item) # 2 [print(item) for item in lst] # 3 for item in lst: print(item) ``` #1 - In my opinion, this should be a one line operation so #1 is not ideal. #2 - It also shouldn't require storing results in array, to save time/memory, so #2 is out. #3 - I think #3 is just not good syntax, it seems very unpythonic to me - it breaks the norm that blocks go on their own lines. It does seem the best of the three though and I know my assessment is kind of subjective. I'm wondering if a possible alternative syntax could be a for-expression, like there's if-expressions, which always evaluates to None: ``` print(item) for item in lst ``` A more practical example of when this would be useful is extending list-2 with a modified version of list-1 - this syntax would avoid creating an intermediate list (not sure if the way lists are implemented in python removes this advantage by the way it resizes lists though). ``` lst1 = [1, 2, 3] lst2 = [4, 5, 6] lst1.append(item * 2) for item in lst1 ```

23 76

Non-standard evaluation for Python
by Nima Hamidi 05 Aug '19

05 Aug '19

Hello all, I would like to briefly share my thoughts on non-standard evaluation (NSE), why this is useful, and how, potentially, it can be added to Python. In most languages, functions have access only to the *value* of their arguments, and not to the expressions that yielded those values. However, sometimes it is useful to have access to the expressions as well. For example, let plot(x, y, ...) be a function that draws the scatter plot of y against x and adds labels for the x- and y-axes. Currently, there is no way to distinguish plot(x, *z*, ...) from plot(x, *y*, ...) from within the function. As such, one needs to pass the names of the variables to the plot function to be used as the axes' labels. Using NSE, though, the function can look at the expressions in the method call and use those as the default value for labels. In R, this idea is used widely to make the syntax simpler. In the following, I sketch how I think this can be implemented: 1. Let BoundExpression be a class containing an ast.Expression as well as locals and globals dictionaries. BoundExpression can also have an eval method that evaluates its expression using its locals and globals dictionaries. 2. Let `x` be the short form for BoundExpression("x", locals(), globals()). Then, the plot function can be implemented in a way that plot(`x`, `y`) draws scatter plot of x and y *and also,* labels the axes correctly. No need to provide labels explicitly anymore. 3. A more challenging idea is to let developers decide whether their functions need NSE or not. For example, when a function is defined as def f(`x`), for any method call like f(x), the first argument should be wrapped with a BoundExpression instance. I would appreciate any feedback on these ideas. Best, Nima

5 18

asyncio: futures and tasks with synchronous callbacks
by aurelien.lambert.89＠gmail.com 01 Aug '19

01 Aug '19

In asyncio, when a task awaits for another task (or future), it can be cancelled right after the awaited task finished (before the callback have been processed). Thus, if the awaited task has consumed data, the data is lost. For instance, with the following code: import asyncio available_data = [] data_ready = asyncio.Future() def feed_data(data): global data_ready available_data.append(data) data_ready.set_result(None) data_ready = asyncio.Future() async def consume_data(): while not available_data: await asyncio.shield(data_ready) return available_data.pop() async def wrapped_consumer(): task = asyncio.ensure_future(consume_data()) return await task If I perform those exact steps: async def test(): task = asyncio.ensure_future(wrapped_consumer()) await asyncio.sleep(0) feed_data('data') await asyncio.sleep(0) task.cancel() await asyncio.sleep(0) print ('task', task) print ('available_data', available_data) loop = asyncio.get_event_loop() loop.run_until_complete(test()) Then I can see that the task has been cancelled despite the data being consumed. Since the result of `wrapped_consumer` cannot be retrieved, the data is forever lost. task <Task cancelled coro=<wrapped_consumer() done, defined at <ipython-input-1-de4ad193b1d0>:17>> available_data [] This side effect does not happen when awaiting a coroutine, but coroutine are not as flexible as tasks. It happens when awaiting a `Future`, a `Task`, or any function like `asyncio.wait`, `asyncio.wait_for` or `asyncio.gather` (which all inherit from or use `Future`). There is then no way to do anything equivalent to: stop_future = asyncio.Future() async def wrapped_consumer2(): task = asyncio.ensure_future(consume_data()) try: await asyncio.wait([task, stop_future]) finally: task.cancel() await asyncio.wait([task]) if not task.cancelled(): return task.result() else: raise RuntimeError('stopped') This is due to the Future calling the callback asynchronously: https://github.com/python/cpython/blob/3.6/Lib/asyncio/futures.py#L214 for callback in callbacks: self._loop.call_soon(callback, self) I propose to create synchronous versions of those, or a `synchronous_callback` parameter, that turns the callbacks of `Future` synchronous. I've experimented a simple librairy `syncio` with CPython 3.6 to do this (it is harder to patch later versions due to the massive use of private methods). Basically, needs to: 1) replace the `Future._schedule_callbacks` method by a synchronous version 2) fix `Task._step` to not fail when cleaning `_current_tasks` (https://github.com/python/cpython/blob/3.6/Lib/asyncio/tasks.py#L245) 3) rewrite all the functions to use synchronous futures instead of normal ones With that librairy, the previous functions are possible and intuitive import syncio async def wrapped_consumer(): task = syncio.ensure_sync_future(consume_data()) return await task stop_future = asyncio.Future() async def wrapped_consumer2(): task = syncio.ensure_sync_future(consume_data()) try: await syncio.sync_wait([task, stop_future]) finally: task.cancel() await asyncio.wait([task]) if not task.cancelled(): return task.result() else: raise RuntimeError('stopped') No need to use `syncio` anywhere else in the code, which makes it totally transparent for the end user. This "library" can be found here: https://github.com/aure-olli/aiokafka/blob/216b4ad3b4115bc9fa463e44fe23636b… It implements `SyncFuture`, `SyncTask`, `ensure_sync_future`, `sync_wait`, `sync_wait_for`, `sync_gather` and `sync_shield`. It works with CPython 3.6 only. To conclude: - asynchronous callbacks are preferrable in most case, but do not provide a coherent cancelled status in specific cases - implementing a version with synchronous callback (or a `synchronous_callback` parameter) is rather easy (however step 2 need to be clarified, probably a cleaner way to fix this) - it is totally transparent for the end user, as synchronous callback are totally compatible with asynchronous ones

4 6

Re: Entrypoint function for modules (AKA if __name__ == '__main__' ) with built-in argument parsing
by Agustín Scaramuzza 31 Jul '19

31 Jul '19

Ouch! That's what happens when you don't proof read your emails :S Replace --verbose with --print_operation, the idea of the snippet was to show an example for keyword parameters.

2 1