Python-ideas January 2016

python-ideas@python.org

79 participants
43 discussions

Respectively and its unpacking sentence
by Mirmojtaba Gharibi 30 Jan '16

30 Jan '16

Hello, I'm thinking of this idea that we have a pseudo-operator called "Respectively" and shown maybe with ; Some examples first: a;b;c = x1;y1;z1 + x2;y2;z2 is equivalent to a=x1+x2 b=y1+y2 c=z1+z2 So it means for each position in the statement, do something like respectively. It's like what I call a vertical expansion, i.e. running statements one by one. Then there is another unpacking operator which maybe we can show with $ sign and it operates on lists and tuples and creates the "Respectively" version of them. So for instance, vec=[]*10 $vec = $u + $v will add two 10-dimensional vectors to each other and put the result in vec. I think this is a syntax that can make many things more concise plus it makes component wise operation on a list done one by one easy. For example, we can calculate the inner product between two vectors like follows (inner product is the sum of component wise multiplication of two vectors): innerProduct =0 innerProduct += $a * $b which is equivalent to innerProduct=0 for i in range(len(a)): ...innerProduct += a[i]+b[i] For example, let's say we want to apply a function to all element in a list, we can do: f($a) The $ and ; take precedence over anything except (). Also, an important thing is that whenever, we don't have the respectively operator, such as for example in the statement above on the left hand side, we basically use the same variable or value or operator for each statement or you can equivalently think we have repeated that whole thing with ;;;;. Such as: s=0 s;s;s += a;b;c; * d;e;f which result in s being a*d+b,c*e+d*f Also, I didn't spot (at least for now any ambiguity). For example one might think what if we do this recursively, such as in: x;y;z + (a;b;c);(d;e;f);(g;h;i) using the formula above this is equivalent to (x;x;x);(y;y;y);(z;z;z)+(a;b;c);(d;e;f);(g;h;i) if we apply print on the statement above, the result will be: x+a x+b x+c y+d y+e y+f z+g z+h z+i Beware that in all of these ; or $ does not create a new list. Rather, they are like creating new lines in the program and executing those lines one by one( in the case of $, to be more accurate, we create for loops). I'll appreciate your time and looking forward to hearing your thoughts. Cheers, Moj

10 22

PEP 511: Add a check function to decide if a "language extension" code transformer should be used or not
by Victor Stinner 28 Jan '16

28 Jan '16

Hi, Thank you for all feedback on my PEP 511. It looks like the current blocker point is the unclear status of "language extensions": code tranformers which deliberately changes the Python semantics. I would like to discuss how we should register them. I think that the PEP 511 must discuss "language extensions" even if it doesn't have to propose a solution to make their usage easier. It's an obvious usage of code transformers. If possible, I would like to find a compromise to support them, but make it explicit that they change the Python semantics. By the way, I discussed with Joseph Jevnik who wrote codetransformer (bytecode transformer) and lazy_python (AST transformer). He wrote me: "One concern that I have though is that transformers are registered globally. I think that the decorators in codetransformer do a good job of signalling to reader the scope of some new code generation." Currently, the PEP 511 doesn't provide a way to register a code transformer but only use it under some conditions. For example, if fatoptimizer is registered, all .pyc files will be called file.cpython-36.fat-0.pyc even if fatoptimizer was disabled. I propose to change the design of sys.set_code_transformers() to use it more like a registry similar to the codecs registry (codecs.register), but different (details below). A difference is that the codecs registry uses a mapping (codec name => codec functions), whereas sys.set_code_transformers() uses an ordered sequence (list) of code transformers. A sequence is used because multiple code transformers can be applied sequentially on a single .py file. Petr Viktorin wrote that language extensions "target specific modules, with which they're closely coupled: The modules won't run without the transformer. And with other modules, the transformer either does nothing (as with MacroPy, hopefully), or would fail altogether (as with Hy). So, they would benefit from specific packages opting in. The effects of enabling them globally range from inefficiency (MacroPy) to failures or needing workarounds (Hy)." Problem (A): solutions proposed below don't make code tranformers mandatory. If a code *requires* a code transformer and the code transformer is not registered, Python doesn't complain. Do you think that it is a real issue in practice? For MacroPy, it's not a problem in practice since functions must be decorated using a decorator from the macropy package. If importing macropy fails, the module cannot be imported. Problem (B): proposed solutions below adds markers to ask to enable a specific code transformer, but a code transformer can decide to always modify the Python semantics without using such marker. According to Nick Coghlan, code transformers changing the Python semantics *must* require a marker in the code using them. IMHO it's the responsability of the author of the code transformer to use markers, not the responsability of Python. Code transformers should maybe return a flag telling if they changed the code or not. I prefer a flag rather than comparing the output to the input, since the comparison can be expensive, especially for a deep AST tree. Example: class Optimizer: def ast_optimizer(self, tree, context): # ... return modified, tree *modified* must be True if tree was modified. There are several options to decide if a code transformer must be used on a specific source file. (1) Add a check_code() and check_ast() functions to code transformers. The code transformer is responsible to decide if it wants to transform the code or not. Python doesn't use the code transformer if the check method returns False. Examples: * MacroPy can search for the "import macropy" statement (of "from macropy import ...") in the AST tree * fatoptimizer can search for "__fatoptimizer__ = {'enabled': False}" in the code: if this variable is found, the optimizer is completly skipped (2) Petr proposed to extend importlib to pass a code transformer when importing a module. importlib.util.import_with_transformer( 'mypackage.specialmodule', MyTransformer()) IMHO this option is too specific: it's restricted to importlib (py_compile, compileall and interactive interpreter don't have the feature). I also dislike the API. (3) Petr also proposed "a special flag in packages": __transformers_for_submodules__ = [MyTransformer()] I don't like having to get access to MyTransformer. The PEP 511 mentions an use case where the transformed code is run *without* registering the transformer. But this issue can easily be fixed by using the string to identify the transformer in the registery (ex: "fat") rather than its class. I'm not sure that putting a flag on the package (package/__init__.py?) is a good idea. I would prefer to enable language extensions on individual files to restrict their scope. (4) Sjoerd Job Postmus proposed something similar but using a comment and not for packages, but any source file: #:Transformers modname.TransformerClassName, modname.OtherTransformerClassName The problem is that comments are not stored in the AST tree. I would prefer to use AST to decide if an AST transformer should be used or not. Note: I'm not really motived to extend the AST to start to include comments, or even code formatting (spaces, newlines, etc.). https://pypi.python.org/pypi/redbaron/ can be used if you want to transform a .py file without touching the format. But I don't think that AST must go to this direction. I prefer to keep AST simple. (5) Nick proposed (indirectly) to use a different filename (don't use ".py") for language extensions. This option works with my option (2): the context contains the filename which can be used to decide to enable or not the code transformer. I understand that the code transformer must also install an importlib hook to search for other filenames than only .py files. Am I right? (6) Nick proposed (indirectly) to use an encoding cookie "which are visible as a comment in the module header". Again, I dislike this option because comments are not stored in AST. Victor

6 7

Re: [Python-ideas] Respectively and its unpacking sentence
by Mirmojtaba Gharibi 28 Jan '16

28 Jan '16

On Thu, Jan 28, 2016 at 3:26 AM, Mirmojtaba Gharibi < mojtaba.gharibi(a)gmail.com> wrote: > > > On Wed, Jan 27, 2016 at 4:15 PM, Andrew Barnert <abarnert(a)yahoo.com> > wrote: > >> On Jan 27, 2016, at 09:55, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com> >> wrote: >> >> On Wed, Jan 27, 2016 at 2:29 AM, Andrew Barnert <abarnert(a)yahoo.com> >> wrote: >> >>> On Jan 26, 2016, at 22:19, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com> >>> wrote: >>> >>> Yes, I'm aware sequence unpacking. >>> There is an overlap like you mentioned, but there are things that can't >>> be done with sequence unpacking, but can be done here. >>> >>> For example, let's say you're given two lists that are not necessarily >>> numbers, so you can't use numpy, but you want to apply some component-wise >>> operator between each component. This is something you can't do with >>> sequence unpacking or with numpy. >>> >>> >>> Yes, you can do it with numpy. >>> >>> Obviously you don't get the performance benefits when you aren't using >>> "native" types (like int32) and operations that have vectorizes >>> implementations (like adding two arrays of int32 or taking the dot product >>> of float64 matrices), but you do still get the same elementwise operators, >>> and even a way to apply arbitrary callables over arrays, or even other >>> collections: >>> >>> >>> firsts = ['John', 'Jane'] >>> >>> lasts = ['Smith', 'Doe'] >>> >>> np.vectorize('{1}, {0}'.format)(firsts, lasts) >>> array(['Smith, John', 'Doe, Jane'], dtype='<U11) >>> >>> I think the form I am suggesting is simpler and more readable. >> >> >> But the form you're suggesting doesn't work for vectorizing arbitrary >> functions, only for operator expressions (including simple function calls, >> but that doesn't help for more general function calls). The fact that numpy >> is a little harder to read for cases that your syntax can't handle at all >> is hardly a strike against numpy. >> > > I don't need to vectorize the functions. It's already being done. > Consider the ; example below: > a;b = f(x;y) > it is equivalent to > a=f(x) > b=f(y) > So in effect, in your terminology, it is already vectorized. > Similar example only with $: > a=[0,0,0,0] > x=[1,2,3,4] > $a=f($x) > is equivalent to > a=[0,0,0,0] > x=[1,2,3,4] > for i in range(len(a)): > ...a[i]=f(x[i]) > > > >> >> And, as I already explained, for the cases where your form _does_ work, >> numpy already does it, without all the sigils: >> >> c = a + b >> >> c = a*a + 2*a*b + b*b >> >> c = (a * b).sum() >> >> It also works nicely over multiple dimensions. For example, if a and b >> are both arrays of N 3-vectors instead of just being 3-vectors, you can >> still elementwise-add them just with +; you can sum all of the results with >> sum(axis=1); etc. How would you write any of those things with your >> $-syntax? >> >> I'm happy you brought vectorize to my attention though. I think as soon >> you make the statement just a bit complex, it would become really >> complicated with vectorize. >> >> >> For example lets say you have >> x=[1,2,3,4,5,...] >> y=['A','BB','CCC',...] >> p=[2,3,4,6,6,...] >> r=[]*n >> >> $r = str(len($y*$p)+$x) >> >> >> As a side note, []*n is always just []. Maybe you meant [None for _ in >> range(n)] or [None]*n? Also, where does n come from? It doesn't seem to >> have anything to do with the lengths of x, y, and p. So, what happens if >> it's shorter than them? Or longer? With numpy, of course, that isn't a >> problem--there's no magic being attempted on the = operator (which is good, >> because = isn't an operator in Python, and I'm not sure how you'd even >> properly define your design, much less implement it); the operators just >> create arrays of the right length. >> >> n I just meant symbolically to be len(x). So please replace n with > len(x). I didn't mean to confuse you. sorry. > > >> Anyway, that's still mostly just operators. You _could_ wrap up an >> operator expression in a function to vectorize, but you almost never want >> to. Just use the operators directly on the arrays. >> >> So, let's try a case that has even some minimal amount of logic, where >> translating to operators would be clumsy at best: >> >> @np.vectorize >> def sillyslice(y, x, p): >> if x < p: return y[x:p] >> return y[p:x] >> >> r = sillyslice(y, x, p) >> >> Being a separate function provides all the usual benefits: sillyslice is >> reusable, debuggable, unit-testable, usable as a first-class object, etc. >> But forget that; how would you do this at all with your $-syntax? >> >> Since you didn't answer any of my other questions, I'll snip them and >> repost shorter versions: >> >> * what's wrong with using numpy? Nothing. What's wrong even with for loop >> or assembly for that matter? I didn't argue that it's not possible to >> achieve these things with assembly. >> * what's wrong with APL or J or MATLAB? Not sure how relevant it is to >> our core of conversation. Skipping this. >> * what's wrong with making the operators elementwise instead of wrapping >> the objects in some magic thing? The fact that whenever you >> * what is the type of that magic thing anyway? It has no type. I refer >> you to my very first email. In that email I exactly explained what it >> means. It's at best a psuedo macro or something like that. It exactly is >> equivalent when you write >> > a;b=f(x;y) > to > a=f(x) > b=f(y) > > In other words, if I could interpret my code before python interpreter > interpret it, I would convert the first to the latter. > >

2 1

several different needs [Explicit variable capture list]
by Jim J. Jewett 28 Jan '16

28 Jan '16

I think a small part of the confusion is that there are at least four separate (albeit related) use cases. They all use default arguments for the current workarounds, but they don't have the same ideal solution. (1) Auxiliary variables def f(x, _len=len): ... This is often a micro-optimization; the _len keyword really shouldn't be overridden. Partly because it shouldn't be overridden, having it in the signature is just ugly. This could be solved with another separator in the signature, such as ; or a second () or a new keyword ... def f(x, aux _len=len): ... def f(x, once _len=len): ... def f(x; _len=len):... def f(x)(_len=len): ... def f(x){_len=len}: ... But realistically, that _len isn't ugly *just* because it shouldn't be overridden; it is also inherently ugly. I would prefer that something like Victor's FAT optimizer just make this idiom obsolete. (2) immutable bindings once X final Y const Z This is pretty similar to the auxiliary variables case, except that it tends to be desired more outside of functions. The immutability can be worth documenting on its own, but it feels too much like a typing declaration, which raises questions of "why *this* distinction for *this* variable?" So again, I think something like Victor's FAT optimizer (plus comments when immutability really is important) is a better long-term solution, but I'm not as sure as I was for case 1. (3) Persistent storage def f(x, _cached_results={}): ... In the past, I've managed to convince myself that it is good to be able to pass in a different cache ... or to turn the function into a class, so that I could get to self, or even to attach attributes to the function itself (so that rebinding the name to another function in a naive manner would fail, rather than produces bugs). Those convincings don't stick very well, though. This was clearly at least one of the motivations of some people who asked about static variables. I still think it might be nice to just have a way of easily opening a new scope ... but then I have to explain why I can't just turn the function into a class... So in the end, I suspect this use case is better off ignored, but I am pretty sure it will lead to some extra confusion if any of the others are "solved" in a way that doesn't consider it. (4) Current Value Capture This is the loop variable case that some have thought was the only case under consideration. I don't have anything to add to Andrew Barnert's https://mail.python.org/pipermail/python-ideas/2016-January/037980.html but do see Steven D'Aprano's https://mail.python.org/pipermail/python-ideas/2016-January/038047.html for gotchas even within this use case. -jJ

5 10

intput()
by Marcel O'Neil 27 Jan '16

27 Jan '16

def intput(): return int(input()) Life would be just marginally easier, with a punny function name as a bonus.

12 19

PEP 484 change proposal: Allowing @overload outside stub files
by Guido van Rossum 26 Jan '16

26 Jan '16

Ben Darnell (Tornado lead) brought up a good use case for allowing @overload in regular Python files. There's some discussion (some old, some new) here: https://github.com/ambv/typehinting/issues/72 I now propose to allow @overload in non-stub (i.e. .py) files, but with the following rule: a series of @overload-decorated functions must be followed by an implementation function that's not @overload-decorated. Calling an @overload-decorated function is still an error (I propose NotImplemented). Due to the way repeated function definitions with the same name replace each other, leaving only the last one active, this should work. E.g. for Tornado's utf8() the full definition would look like this: @overloaddef utf8(value: None) -> None: ...@overloaddef utf8(value: bytes) -> bytes: ...@overloaddef utf8(value: str) -> bytes: ... # or (unicode)->bytes, in PY2def utf8(value): # Real implementation goes here. NOTE: If you are trying to understand why we can't use a stub file here or why we can't solve this with type variables or unions, please read the issue and comment there if things are not clear. Here on python-ideas I'd like to focus on seeing whether this amendment is non-controversial (apart from tea party members who just want to repeal PEP 484 entirely :-). I know that previously we wanted to come up with a complete solution for multi-dispatch based on type annotations first, and there are philosophical problems with using @overload (though it can be made to work using sys._getframe()). The proposal here is *not* that solution. If you call the @overload-decorated function, it will raise NotImplemented. (But if you follow the rule, the @overload-decorated function objects are inaccessible so this would only happen if you forgot or misspelled the final, undecorated implementation function). -- --Guido van Rossum (python.org/~guido)

7 11

Making Python great again
by Sven R. Kunze 25 Jan '16

25 Jan '16

Hi, for all those who felt that something is wrong with Python. Here's the solution: https://github.com/samshadwell/TrumpScript ;) Best, Sven

1 0

Documenting asyncio methods as returning awaitables
by Ian Kelly 25 Jan '16

25 Jan '16

The official asyncio documentation includes this note: """ Note: In this documentation, some methods are documented as coroutines, even if they are plain Python functions returning a Future. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be used in a callback-style code, wrap its result with ensure_future(). """ Despite the note, this still causes confusion. See for example https://mail.python.org/pipermail/python-list/2016-January/702342.html As of Python 3.5, "awaitable" is a thing, and as of Python 3.5.1, ensure_future is supposed to accept any awaitable. Would it be better then to document these methods as returning awaitables rather than as coroutines?

2 1

Cross link documentation translations
by Julien Palard 25 Jan '16

25 Jan '16

o/ While translating the Python Documentation in French [1][2], I discovered that we're not the only country doing it, there is also Japan [3][4], and Spain [5]. It's possible there's other but I didn't find them (and it's the problem). But there's only a few way for users to find the translations (hearing about them, or explicitly searching for them on a search engine, which they won't do, obviously expecting a link from the english version if they exists). So here is my idea: Why not linking translations from the main documentation? I know that's not directly supported by Sphinx doc [6], but separate sphinx build, blindly (with hardcoded links) linking themselves, may work (like readthedoc is probably doing). The downside of those links is that we'll sometime link to untranslated parts, but those parts may be marked as untranslated [7] to encourage new translators to help. Thoughts? [1] http://www.afpy.org/doc/python/3.5/ [2] https://github.com/afpy/python_doc_fr [3] http://docs.python.jp/3/ [4] https://github.com/python-doc-ja/python-doc-ja [5] http://docs.python.org.ar/tutorial/3/index.html [6] https://github.com/sphinx-doc/sphinx/issues/2252 [7] https://github.com/sphinx-doc/sphinx/issues/1246 -- Julien Palard

4 4

Multiple dispatch (was Re: PEP 484 change proposal: Allowing <at> overload outside stub files
by Nick Coghlan 24 Jan '16

24 Jan '16

On 24 January 2016 at 07:12, Stefan Krah <skrah.temporarily(a)gmail.com> wrote: > Nick Coghlan <ncoghlan@...> writes: >> You're already going to have to allow this for single lines to handle >> Py2 compatible annotations, so it seems reasonable to also extend it >> to handle overloading while you're still figuring out a native syntax >> for that. > > I find that https://pypi.python.org/pypi/multipledispatch looks quite > nice: > >>>> from multipledispatch import dispatch >>>> @dispatch(int, int) > ... def add(x, y): > ... return x + y > ... >>>> @dispatch(float, float) > ... def add(x, y): > ... return x + y > ... >>>> add(1, 2) > 3 >>>> add(1.0, 2.0) > 3.0 >>>> add(1.0, 2) > Traceback (most recent call last): > File > [cut because gmane.org is inflexible] > line 155, in __call__ > func = self._cache[types] > KeyError: (<class 'float'>, <class 'int'>) Right, the Blaze folks have been doing some very nice work in that area. One of the projects building on multipledispatch is the Odo network of data conversion operations: https://github.com/blaze/odo They do make the somewhat controversial design decision to make dispatch operations process global by default [1], rather than scoping by module. On the other hand, the design also makes it easy to define your own dispatch namespace, so the default orthogonality with the module system likely isn't a problem in practice, and the lack of essential boilerplate does make it very easy to use in contexts like an IPython notebook. There is one aspect that still requires runtime stack introspection [2], and that's getting access to the class scope in order to implicitly make method dispatch specific to the class defining the methods. It's the kind of thing that makes me wonder whether we should be exposing a thread-local variable somewhere with a "class namespace stack" that made it possible to: - tell that you're currently running in the context of a class definition - readily get access to the namespace of the innermost class currently being defined Cheers, Nick. [1] http://multiple-dispatch.readthedocs.org/en/latest/design.html#namespaces-a… [2] https://github.com/mrocklin/multipledispatch/blob/master/multipledispatch/c… -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas January 2016