Mailman 3 January 2016 - Python-ideas

Respectively and its unpacking sentence
by Mirmojtaba Gharibi Jan. 30, 2016

Jan. 30, 2016

Hello, I'm thinking of this idea that we have a pseudo-operator called "Respectively" and shown maybe with ; Some examples first: a;b;c = x1;y1;z1 + x2;y2;z2 is equivalent to a=x1+x2 b=y1+y2 c=z1+z2 So it means for each position in the statement, do something like respectively. It's like what I call a vertical expansion, i.e. running statements one by one. Then there is another unpacking operator which maybe we can show with $ sign and it operates on lists and tuples and creates the "… [View More]

10 22

PEP 511: Add a check function to decide if a "language extension" code transformer should be used or not
by Victor Stinner Jan. 28, 2016

Jan. 28, 2016

Hi, Thank you for all feedback on my PEP 511. It looks like the current blocker point is the unclear status of "language extensions": code tranformers which deliberately changes the Python semantics. I would like to discuss how we should register them. I think that the PEP 511 must discuss "language extensions" even if it doesn't have to propose a solution to make their usage easier. It's an obvious usage of code transformers. If possible, I would like to find a compromise to support them, but … [View More]make it explicit that they change the Python semantics. By the way, I discussed with Joseph Jevnik who wrote codetransformer (bytecode transformer) and lazy_python (AST transformer). He wrote me: "One concern that I have though is that transformers are registered globally. I think that the decorators in codetransformer do a good job of signalling to reader the scope of some new code generation." Currently, the PEP 511 doesn't provide a way to register a code transformer but only use it under some conditions. For example, if fatoptimizer is registered, all .pyc files will be called file.cpython-36.fat-0.pyc even if fatoptimizer was disabled. I propose to change the design of sys.set_code_transformers() to use it more like a registry similar to the codecs registry (codecs.register), but different (details below). A difference is that the codecs registry uses a mapping (codec name => codec functions), whereas sys.set_code_transformers() uses an ordered sequence (list) of code transformers. A sequence is used because multiple code transformers can be applied sequentially on a single .py file. Petr Viktorin wrote that language extensions "target specific modules, with which they're closely coupled: The modules won't run without the transformer. And with other modules, the transformer either does nothing (as with MacroPy, hopefully), or would fail altogether (as with Hy). So, they would benefit from specific packages opting in. The effects of enabling them globally range from inefficiency (MacroPy) to failures or needing workarounds (Hy)." Problem (A): solutions proposed below don't make code tranformers mandatory. If a code *requires* a code transformer and the code transformer is not registered, Python doesn't complain. Do you think that it is a real issue in practice? For MacroPy, it's not a problem in practice since functions must be decorated using a decorator from the macropy package. If importing macropy fails, the module cannot be imported. Problem (B): proposed solutions below adds markers to ask to enable a specific code transformer, but a code transformer can decide to always modify the Python semantics without using such marker. According to Nick Coghlan, code transformers changing the Python semantics *must* require a marker in the code using them. IMHO it's the responsability of the author of the code transformer to use markers, not the responsability of Python. Code transformers should maybe return a flag telling if they changed the code or not. I prefer a flag rather than comparing the output to the input, since the comparison can be expensive, especially for a deep AST tree. Example: class Optimizer: def ast_optimizer(self, tree, context): # ... return modified, tree *modified* must be True if tree was modified. There are several options to decide if a code transformer must be used on a specific source file. (1) Add a check_code() and check_ast() functions to code transformers. The code transformer is responsible to decide if it wants to transform the code or not. Python doesn't use the code transformer if the check method returns False. Examples: * MacroPy can search for the "import macropy" statement (of "from macropy import ...") in the AST tree * fatoptimizer can search for "__fatoptimizer__ = {'enabled': False}" in the code: if this variable is found, the optimizer is completly skipped (2) Petr proposed to extend importlib to pass a code transformer when importing a module. importlib.util.import_with_transformer( 'mypackage.specialmodule', MyTransformer()) IMHO this option is too specific: it's restricted to importlib (py_compile, compileall and interactive interpreter don't have the feature). I also dislike the API. (3) Petr also proposed "a special flag in packages": __transformers_for_submodules__ = [MyTransformer()] I don't like having to get access to MyTransformer. The PEP 511 mentions an use case where the transformed code is run *without* registering the transformer. But this issue can easily be fixed by using the string to identify the transformer in the registery (ex: "fat") rather than its class. I'm not sure that putting a flag on the package (package/__init__.py?) is a good idea. I would prefer to enable language extensions on individual files to restrict their scope. (4) Sjoerd Job Postmus proposed something similar but using a comment and not for packages, but any source file: #:Transformers modname.TransformerClassName, modname.OtherTransformerClassName The problem is that comments are not stored in the AST tree. I would prefer to use AST to decide if an AST transformer should be used or not. Note: I'm not really motived to extend the AST to start to include comments, or even code formatting (spaces, newlines, etc.). https://pypi.python.org/pypi/redbaron/ can be used if you want to transform a .py file without touching the format. But I don't think that AST must go to this direction. I prefer to keep AST simple. (5) Nick proposed (indirectly) to use a different filename (don't use ".py") for language extensions. This option works with my option (2): the context contains the filename which can be used to decide to enable or not the code transformer. I understand that the code transformer must also install an importlib hook to search for other filenames than only .py files. Am I right? (6) Nick proposed (indirectly) to use an encoding cookie "which are visible as a comment in the module header". Again, I dislike this option because comments are not stored in AST. Victor [View Less]

6 7

Re: [Python-ideas] Respectively and its unpacking sentence
by Mirmojtaba Gharibi Jan. 28, 2016

Jan. 28, 2016

On Thu, Jan 28, 2016 at 3:26 AM, Mirmojtaba Gharibi < mojtaba.gharibi(a)gmail.com> wrote: > > > On Wed, Jan 27, 2016 at 4:15 PM, Andrew Barnert <abarnert(a)yahoo.com> > wrote: > >> On Jan 27, 2016, at 09:55, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com> >> wrote: >> >> On Wed, Jan 27, 2016 at 2:29 AM, Andrew Barnert <abarnert(a)yahoo.com> >> wrote: >> >>> On Jan 26, 2016, at 22:19, Mirmojtaba Gharibi <… [View More]mojtaba.gharibi(a)gmail.com> >>> wrote: >>> >>> Yes, I'm aware sequence unpacking. >>> There is an overlap like you mentioned, but there are things that can't >>> be done with sequence unpacking, but can be done here. >>> >>> For example, let's say you're given two lists that are not necessarily >>> numbers, so you can't use numpy, but you want to apply some component-wise >>> operator between each component. This is something you can't do with >>> sequence unpacking or with numpy. >>> >>> >>> Yes, you can do it with numpy. >>> >>> Obviously you don't get the performance benefits when you aren't using >>> "native" types (like int32) and operations that have vectorizes >>> implementations (like adding two arrays of int32 or taking the dot product >>> of float64 matrices), but you do still get the same elementwise operators, >>> and even a way to apply arbitrary callables over arrays, or even other >>> collections: >>> >>> >>> firsts = ['John', 'Jane'] >>> >>> lasts = ['Smith', 'Doe'] >>> >>> np.vectorize('{1}, {0}'.format)(firsts, lasts) >>> array(['Smith, John', 'Doe, Jane'], dtype='<U11) >>> >>> I think the form I am suggesting is simpler and more readable. >> >> >> But the form you're suggesting doesn't work for vectorizing arbitrary >> functions, only for operator expressions (including simple function calls, >> but that doesn't help for more general function calls). The fact that numpy >> is a little harder to read for cases that your syntax can't handle at all >> is hardly a strike against numpy. >> > > I don't need to vectorize the functions. It's already being done. > Consider the ; example below: > a;b = f(x;y) > it is equivalent to > a=f(x) > b=f(y) > So in effect, in your terminology, it is already vectorized. > Similar example only with $: > a=[0,0,0,0] > x=[1,2,3,4] > $a=f($x) > is equivalent to > a=[0,0,0,0] > x=[1,2,3,4] > for i in range(len(a)): > ...a[i]=f(x[i]) > > > >> >> And, as I already explained, for the cases where your form _does_ work, >> numpy already does it, without all the sigils: >> >> c = a + b >> >> c = a*a + 2*a*b + b*b >> >> c = (a * b).sum() >> >> It also works nicely over multiple dimensions. For example, if a and b >> are both arrays of N 3-vectors instead of just being 3-vectors, you can >> still elementwise-add them just with +; you can sum all of the results with >> sum(axis=1); etc. How would you write any of those things with your >> $-syntax? >> >> I'm happy you brought vectorize to my attention though. I think as soon >> you make the statement just a bit complex, it would become really >> complicated with vectorize. >> >> >> For example lets say you have >> x=[1,2,3,4,5,...] >> y=['A','BB','CCC',...] >> p=[2,3,4,6,6,...] >> r=[]*n >> >> $r = str(len($y*$p)+$x) >> >> >> As a side note, []*n is always just []. Maybe you meant [None for _ in >> range(n)] or [None]*n? Also, where does n come from? It doesn't seem to >> have anything to do with the lengths of x, y, and p. So, what happens if >> it's shorter than them? Or longer? With numpy, of course, that isn't a >> problem--there's no magic being attempted on the = operator (which is good, >> because = isn't an operator in Python, and I'm not sure how you'd even >> properly define your design, much less implement it); the operators just >> create arrays of the right length. >> >> n I just meant symbolically to be len(x). So please replace n with > len(x). I didn't mean to confuse you. sorry. > > >> Anyway, that's still mostly just operators. You _could_ wrap up an >> operator expression in a function to vectorize, but you almost never want >> to. Just use the operators directly on the arrays. >> >> So, let's try a case that has even some minimal amount of logic, where >> translating to operators would be clumsy at best: >> >> @np.vectorize >> def sillyslice(y, x, p): >> if x < p: return y[x:p] >> return y[p:x] >> >> r = sillyslice(y, x, p) >> >> Being a separate function provides all the usual benefits: sillyslice is >> reusable, debuggable, unit-testable, usable as a first-class object, etc. >> But forget that; how would you do this at all with your $-syntax? >> >> Since you didn't answer any of my other questions, I'll snip them and >> repost shorter versions: >> >> * what's wrong with using numpy? Nothing. What's wrong even with for loop >> or assembly for that matter? I didn't argue that it's not possible to >> achieve these things with assembly. >> * what's wrong with APL or J or MATLAB? Not sure how relevant it is to >> our core of conversation. Skipping this. >> * what's wrong with making the operators elementwise instead of wrapping >> the objects in some magic thing? The fact that whenever you >> * what is the type of that magic thing anyway? It has no type. I refer >> you to my very first email. In that email I exactly explained what it >> means. It's at best a psuedo macro or something like that. It exactly is >> equivalent when you write >> > a;b=f(x;y) > to > a=f(x) > b=f(y) > > In other words, if I could interpret my code before python interpreter > interpret it, I would convert the first to the latter. > > [View Less]

2 1

several different needs [Explicit variable capture list]
by Jim J. Jewett Jan. 28, 2016

Jan. 28, 2016

I think a small part of the confusion is that there are at least four separate (albeit related) use cases. They all use default arguments for the current workarounds, but they don't have the same ideal solution. (1) Auxiliary variables def f(x, _len=len): ... This is often a micro-optimization; the _len keyword really shouldn't be overridden. Partly because it shouldn't be overridden, having it in the signature is just ugly. This could be solved with another separator in the … [View More]signature, such as ; or a second () or a new keyword ... def f(x, aux _len=len): ... def f(x, once _len=len): ... def f(x; _len=len):... def f(x)(_len=len): ... def f(x){_len=len}: ... But realistically, that _len isn't ugly *just* because it shouldn't be overridden; it is also inherently ugly. I would prefer that something like Victor's FAT optimizer just make this idiom obsolete. (2) immutable bindings once X final Y const Z This is pretty similar to the auxiliary variables case, except that it tends to be desired more outside of functions. The immutability can be worth documenting on its own, but it feels too much like a typing declaration, which raises questions of "why *this* distinction for *this* variable?" So again, I think something like Victor's FAT optimizer (plus comments when immutability really is important) is a better long-term solution, but I'm not as sure as I was for case 1. (3) Persistent storage def f(x, _cached_results={}): ... In the past, I've managed to convince myself that it is good to be able to pass in a different cache ... or to turn the function into a class, so that I could get to self, or even to attach attributes to the function itself (so that rebinding the name to another function in a naive manner would fail, rather than produces bugs). Those convincings don't stick very well, though. This was clearly at least one of the motivations of some people who asked about static variables. I still think it might be nice to just have a way of easily opening a new scope ... but then I have to explain why I can't just turn the function into a class... So in the end, I suspect this use case is better off ignored, but I am pretty sure it will lead to some extra confusion if any of the others are "solved" in a way that doesn't consider it. (4) Current Value Capture This is the loop variable case that some have thought was the only case under consideration. I don't have anything to add to Andrew Barnert's https://mail.python.org/pipermail/python-ideas/2016-January/037980.html but do see Steven D'Aprano's https://mail.python.org/pipermail/python-ideas/2016-January/038047.html for gotchas even within this use case. -jJ [View Less]

5 10

intput()
by Marcel O'Neil Jan. 27, 2016

Jan. 27, 2016

def intput(): return int(input()) Life would be just marginally easier, with a punny function name as a bonus.

12 19

PEP 484 change proposal: Allowing @overload outside stub files
by Guido van Rossum Jan. 26, 2016

Jan. 26, 2016

Ben Darnell (Tornado lead) brought up a good use case for allowing @overload in regular Python files. There's some discussion (some old, some new) here: https://github.com/ambv/typehinting/issues/72 I now propose to allow @overload in non-stub (i.e. .py) files, but with the following rule: a series of @overload-decorated functions must be followed by an implementation function that's not @overload-decorated. Calling an @overload-decorated function is still an error (I propose NotImplemented). … [View More]

7 11

Making Python great again
by Sven R. Kunze Jan. 25, 2016

Jan. 25, 2016

Hi, for all those who felt that something is wrong with Python. Here's the solution: https://github.com/samshadwell/TrumpScript ;) Best, Sven

1 0

Documenting asyncio methods as returning awaitables
by Ian Kelly Jan. 25, 2016

Jan. 25, 2016

The official asyncio documentation includes this note: """ Note: In this documentation, some methods are documented as coroutines, even if they are plain Python functions returning a Future. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be used in a callback-style code, wrap its result with ensure_future(). """ Despite the note, this still causes confusion. See for example https://mail.python.org/… [View More]

2 1

Cross link documentation translations
by Julien Palard Jan. 25, 2016

Jan. 25, 2016

o/ While translating the Python Documentation in French [1][2], I discovered that we're not the only country doing it, there is also Japan [3][4], and Spain [5]. It's possible there's other but I didn't find them (and it's the problem). But there's only a few way for users to find the translations (hearing about them, or explicitly searching for them on a search engine, which they won't do, obviously expecting a link from the english version if they exists). So here is my idea: Why not … [View More]

4 4

Multiple dispatch (was Re: PEP 484 change proposal: Allowing <at> overload outside stub files
by Nick Coghlan Jan. 24, 2016

Jan. 24, 2016

On 24 January 2016 at 07:12, Stefan Krah <skrah.temporarily(a)gmail.com> wrote: > Nick Coghlan <ncoghlan@...> writes: >> You're already going to have to allow this for single lines to handle >> Py2 compatible annotations, so it seems reasonable to also extend it >> to handle overloading while you're still figuring out a native syntax >> for that. > > I find that https://pypi.python.org/pypi/multipledispatch looks quite > nice: > >>>… [View More]

2 1