Mailman 3 September 2016 - Python-ideas

Three ways of paths canonization
by Serhiy Storchaka Sept. 8, 2016

Sept. 8, 2016

Before removing provisional state from the pathlib module, we should resolve the issue with Path.resolve(). It corresponds to os.path.realpath(), but behaves differently in case of non-existent path. Actually we can't say that any of these functions is wrong. Both behaviors make sense in different situations. The readlink utility from GNU coreutils has three mode for resolving file path: -f, --canonicalize canonicalize by following every symlink in every component of the given name recursively; all but the last component must exist -e, --canonicalize-existing canonicalize by following every symlink in every component of the given name recursively, all components must exist -m, --canonicalize-missing canonicalize by following every symlink in every component of the given name recursively, without requirements on components existence Current behavior of posixpath.realpath() is matches (besides one minor detail) to `readlink -m`. The behavior of Path.resolve() matches `readlink -e`. I have proposed a patch that adds three-state optional parameter to posixpath.realpath() and I'm going to provide similar patch for Path.resolve(). But I'm not sure this is good API. Are there better variants? [1] http://bugs.python.org/issue19717 [2] http://bugs.python.org/issue27002

5 4

Re: [Python-ideas] Shuffled
by Arek Bulski Sept. 8, 2016

Sept. 8, 2016

Hey, I just did put my thoughts on the discussion floor. Dont yell at me for it not being the most interesting idea ever. There were few interesting responses after all. Lets move on. pozdrawiam, Arkadiusz Bulski

1 0

Typecheckers: there can be only one
by Hugh Fisher Sept. 8, 2016

Sept. 8, 2016

There's been discussion here and on python-dev regarding PEP 526 that assumes there will be multiple type checkers for Python. I really can't see this happening. If typing annotations become common or maybe considered best practice, the Zen of Python "there should be one-- and preferably only one --obvious way to do it" will take effect. Firstly, the interpreter will need to have type checking built in. Just about every intro book and tutorial for Python says how great it is that you don't have an edit-save-compile cycle, just fire up the Python interpreter and start typing. Having to run a separate type checker will be considered as ridiculous as a C compiler that didn't run the preprocessor itself. Secondly, PyPI will collapse if there isn't just one. How can we express dependencies between packages that use different type checkers? When type checkers themselves have versions? When a dev team uses one type checker for 1.x and then switches to another for 2.x? That's a special circle of hell. -- cheers, Hugh Fisher

7 13

Re: [Python-ideas] Shuffled
by Arek Bulski Sept. 8, 2016

Sept. 8, 2016

> So, why can't you call random.shuffle(all_tests) if you want to run your tests in random order? I dont randomize test order. People should stop assuming that they know better. I need to randomize some arguments for one particular test and I cannot call shuffle between tests. Its a continous list of declarative tests. Needs to be shuffled(). https://github.com/construct/construct/blob/master/tests/test_all.py See? No way to put imperative code between tests. > sample(container, len(container)) That wont work because I would have to type the expression that is used as argument twice in a test. I need shuffled. Enough said. > I'll reiterate that I don't have a use case for this myself... I dont have a use case for half of what the std library offers. Or for type annotations. Asynchronous comprehesions, what is that? Do you see me rejecting those? > (sample having default of entire list size) That would work but would not be pretty. shuffled() is self explanatory and has a nice ring to it. Randomized list is not a sample by definition. pozdrawiam, Arkadiusz Bulski

7 6

Re: [Python-ideas] Shuffled
by Arek Bulski Sept. 8, 2016

Sept. 8, 2016

> You've probably spent way more time debating shuffled here than you would have needed to add a shuffled function to your test file. Another presupposition. I did add it to my utility. I am proposing to add this to the std library so that others dont have to. pozdrawiam, Arkadiusz Bulski

2 1

Re: [Python-ideas] Shuffled
by Arek Bulski Sept. 8, 2016

Sept. 8, 2016

If you want to see the declarative tests, here it is. https://github.com/construct/construct/blob/master/tests/test_all.py pozdrawiam, Arkadiusz Bulski 2016-09-08 2:13 GMT+02:00 Arek Bulski <arek.bulski(a)gmail.com>: > See for yourself. There is a long list of declarative tests. > https://github.com/construct/construct/blob/master/tests/test_all.py > > pozdrawiam, > Arkadiusz Bulski > > > 2016-09-08 1:28 GMT+02:00 Alexander Belopolsky < > alexander.belopolsky(a)gmail.com>: > >> >> On Wed, Sep 7, 2016 at 6:56 PM, Arek Bulski <arek.bulski(a)gmail.com> >> wrote: >> >>> In the project I maintain (construct) there are declarative testcases >>> that look like a long list of (func, args, excepted output, error type) >>> tuples. There is no way for me to call shuffle in there. >> >> >> Can you explain why? Something like this can be easily done with pytest: >> >> In [1]: def foo(x): >> ...: return x + 1 >> ...: >> >> In [2]: import pytest >> >> In [3]: @pytest.mark.parametrize('x, y', [ >> ...: (100, 101), >> ...: (200, 201), >> ...: (300, 301),]) >> ...: def test_foo(x, y): >> ...: assert foo(x) == y >> ...: >> >> In [4]: test_foo.parametrize.args >> Out[4]: ('x, y', [(100, 101), (200, 201), (300, 301)]) >> >> In [5]: import random >> >> In [6]: random.shuffle(test_foo.parametrize.args[1]) >> >> In [7]: test_foo.parametrize.args >> Out[7]: ('x, y', [(200, 201), (100, 101), (300, 301)]) >> > >

2 2

Re: [Python-ideas] Shuffled
by Arek Bulski Sept. 7, 2016

Sept. 7, 2016

Thank you all for the meta discussion. I know we sidetracked a bit but please everyone remember, this is an open discussion floor. I can present you a use case, just didnt think I even needed one. This seems so obvious to me that it should just go in without much saying. In the project I maintain (construct) there are declarative testcases that look like a long list of (func, args, excepted output, error type) tuples. There is no way for me to call shuffle in there. I can only use shuffled. So here it is, the use case you requested. The argument that it can be easily implemented does not stick. In interactive sessions, indeed writing own functions is a pain in the ass. Expecting batteries included is rightful. Builtins like min() max() sum() are sooo easy to implement on your own. So are some methods in itertools, you can even find their sources in the docs. That doesnt mean batteries should not be included. shuffled = sorted(list, key=lambda x: randint) This is wrong because sorting is comparison based and must be nlogn bound. Shuffling is cheaper computationally. Also someone would have to come up with a formal proof that it provides a uniform distribution which probably was done for the shuffle implementaion. Another argument still remains: there is a missing analogy between sort-sorted and shuffle-shuffled. I did NOT suggest making it a builtin. It should be added to random module. > "Also shuffle() should return self so mutating methods could be chained." That was a side suggestion. I withdraw that part. Chaining methods is sometimes useful but we can go without it. pozdrawiam, Arkadiusz Bulski

2 1

Re: [Python-ideas] PEP 530: Asynchronous Comprehensions
by Adam Bartoš Sept. 7, 2016

Sept. 7, 2016

Hello, will await be allowed also in the "if" part of comprehensions? And what about the "in" part? (e.g. if I'm not mistaken, we may have an asynchronous function returning an asynchronous iterator.) Regards, Adam Bartoš

4 5

PEP 530: Asynchronous Comprehensions
by Yury Selivanov Sept. 7, 2016

Sept. 7, 2016

Hi, Below is a proposal to add support for asynchronous comprehensions and asynchronous generator expressions in Python 3.6. I have a half-working implementation of the proposal which fully implements all required grammar and AST changes. What's left is to update the compiler to emit correct opcodes for async comprehensions. I'm confident that we can have a fully working patch before the feature freeze. Thank you, Yury PEP: 530 Title: Asynchronous Comprehensions Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Discussions-To: <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2016 Python-Version: 3.6 Post-History: 03-Sep-2016 Abstract ======== PEP 492 and PEP 525 introduce support for native coroutines and asynchronous generators using ``async`` / ``await`` syntax. This PEP proposes to add asynchronous versions of list, set, dict comprehensions and generator expressions. Rationale and Goals =================== Python has extensive support for synchronous comprehensions, allowing to produce lists, dicts, and sets with a simple and concise syntax. We propose implementing similar syntactic constructions for the asynchronous code. To illustrate the readability improvement, consider the following example:: result = [] async for i in aiter(): if i % 2: result.append(i) With the proposed asynchronous comprehensions syntax, the above code becomes as short as:: result = [i async for i in aiter() if i % 2] The PEP also makes it possible to use the ``await`` expressions in all kinds of comprehensions:: result = [await fun() for fun in funcs] Specification ============= Asynchronous Comprehensions --------------------------- We propose to allow using ``async for`` inside list, set and dict comprehensions. Pending PEP 525 approval, we can also allow creation of asynchronous generator expressions. Examples: * set comprehension: ``{i async for i in agen()}``; * list comprehension: ``[i async for i in agen()]``; * dict comprehension: ``{i: i ** 2 async for i in agen()}``; * generator expression: ``(i ** 2 async for i in agen())``. It is allowed to use ``async for`` along with ``if`` and ``for`` clauses in asynchronous comprehensions and generator expressions:: dataset = {data for line in aiter() async for data in line if check(data)} Asynchronous comprehensions are only allowed inside an ``async def`` function. In principle, asynchronous generator expressions are allowed in any context. However, in Python 3.6, due to ``async`` and ``await`` soft-keyword status, asynchronous generator expressions are only allowed in an ``async def`` function. Once ``async`` and ``await`` become reserved keywords in Python 3.7 this restriction will be removed. ``await`` in Comprehensions --------------------------- We propose to allow the use of ``await`` expressions in both asynchronous and synchronous comprehensions:: result = [await fun() for fun in funcs] result = {await fun() for fun in funcs} result = {fun: await fun() for fun in funcs} result = [await fun() async for fun in funcs] result = {await fun() async for fun in funcs} result = {fun: await fun() async for fun in funcs} This is only valid in ``async def`` function body. Grammar Updates --------------- The proposal requires one change on the grammar level: adding the optional "async" keyword to ``comp_for``:: comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter] The ``comprehension`` AST node will have the new ``is_async`` argument. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

12 31

Atomic counter / atomic increment
by Ben Hoyt Sept. 7, 2016

Sept. 7, 2016

I had to implement a simple atomic counter the other day to count the total number of requests processed in a multi-threaded Python web server. I was doing a demo of "how cool Python is" to my colleagues, and they were generally wowed, but one of the things that made them do a double-take (coming mostly from Scala/Java) was that there was no atomic counter in the standard library. The fact that I and many other folks have implemented such things makes me wonder if it should be in the standard library. It's pretty simple to implement, basically the handful of lines of code below (full version on GitHub Gist at https://gist.github.com/benhoyt/8c8a8d62debe8e5aa5340373f9c509c7): import threading class AtomicCounter: def __init__(self, initial=0): self.value = initial self._lock = threading.Lock() def increment(self, num=1): with self._lock: self.value += num return self.value And if you just want a one-off and don't want to write a class, it's like so: import threading counter_lock = threading.Lock() counter = 0 with counter_lock: counter += 1 value = counter print(value) But it could be this much more obvious code: import threading counter = threading.AtomicCounter() value = counter.increment() print(value) Thoughts? Would such a class make a good candidate for the standard library? (API could probably be improved.) -Ben

4 5