Python-ideas January 2016

python-ideas@python.org

79 participants
42 discussions

PEP 511: Add a check function to decide if a "language extension" code transformer should be used or not

by Victor Stinner

Hi, Thank you for all feedback on my PEP 511. It looks like the current blocker point is the unclear status of "language extensions": code tranformers which deliberately changes the Python semantics. I would like to discuss how we should register them. I think that the PEP 511 must discuss "language extensions" even if it doesn't have to propose a solution to make their usage easier. It's an obvious usage of code transformers. If possible, I would like to find a compromise to support them, but make it explicit that they change the Python semantics. By the way, I discussed with Joseph Jevnik who wrote codetransformer (bytecode transformer) and lazy_python (AST transformer). He wrote me: "One concern that I have though is that transformers are registered globally. I think that the decorators in codetransformer do a good job of signalling to reader the scope of some new code generation." Currently, the PEP 511 doesn't provide a way to register a code transformer but only use it under some conditions. For example, if fatoptimizer is registered, all .pyc files will be called file.cpython-36.fat-0.pyc even if fatoptimizer was disabled. I propose to change the design of sys.set_code_transformers() to use it more like a registry similar to the codecs registry (codecs.register), but different (details below). A difference is that the codecs registry uses a mapping (codec name => codec functions), whereas sys.set_code_transformers() uses an ordered sequence (list) of code transformers. A sequence is used because multiple code transformers can be applied sequentially on a single .py file. Petr Viktorin wrote that language extensions "target specific modules, with which they're closely coupled: The modules won't run without the transformer. And with other modules, the transformer either does nothing (as with MacroPy, hopefully), or would fail altogether (as with Hy). So, they would benefit from specific packages opting in. The effects of enabling them globally range from inefficiency (MacroPy) to failures or needing workarounds (Hy)." Problem (A): solutions proposed below don't make code tranformers mandatory. If a code *requires* a code transformer and the code transformer is not registered, Python doesn't complain. Do you think that it is a real issue in practice? For MacroPy, it's not a problem in practice since functions must be decorated using a decorator from the macropy package. If importing macropy fails, the module cannot be imported. Problem (B): proposed solutions below adds markers to ask to enable a specific code transformer, but a code transformer can decide to always modify the Python semantics without using such marker. According to Nick Coghlan, code transformers changing the Python semantics *must* require a marker in the code using them. IMHO it's the responsability of the author of the code transformer to use markers, not the responsability of Python. Code transformers should maybe return a flag telling if they changed the code or not. I prefer a flag rather than comparing the output to the input, since the comparison can be expensive, especially for a deep AST tree. Example: class Optimizer: def ast_optimizer(self, tree, context): # ... return modified, tree *modified* must be True if tree was modified. There are several options to decide if a code transformer must be used on a specific source file. (1) Add a check_code() and check_ast() functions to code transformers. The code transformer is responsible to decide if it wants to transform the code or not. Python doesn't use the code transformer if the check method returns False. Examples: * MacroPy can search for the "import macropy" statement (of "from macropy import ...") in the AST tree * fatoptimizer can search for "__fatoptimizer__ = {'enabled': False}" in the code: if this variable is found, the optimizer is completly skipped (2) Petr proposed to extend importlib to pass a code transformer when importing a module. importlib.util.import_with_transformer( 'mypackage.specialmodule', MyTransformer()) IMHO this option is too specific: it's restricted to importlib (py_compile, compileall and interactive interpreter don't have the feature). I also dislike the API. (3) Petr also proposed "a special flag in packages": __transformers_for_submodules__ = [MyTransformer()] I don't like having to get access to MyTransformer. The PEP 511 mentions an use case where the transformed code is run *without* registering the transformer. But this issue can easily be fixed by using the string to identify the transformer in the registery (ex: "fat") rather than its class. I'm not sure that putting a flag on the package (package/__init__.py?) is a good idea. I would prefer to enable language extensions on individual files to restrict their scope. (4) Sjoerd Job Postmus proposed something similar but using a comment and not for packages, but any source file: #:Transformers modname.TransformerClassName, modname.OtherTransformerClassName The problem is that comments are not stored in the AST tree. I would prefer to use AST to decide if an AST transformer should be used or not. Note: I'm not really motived to extend the AST to start to include comments, or even code formatting (spaces, newlines, etc.). https://pypi.python.org/pypi/redbaron/ can be used if you want to transform a .py file without touching the format. But I don't think that AST must go to this direction. I prefer to keep AST simple. (5) Nick proposed (indirectly) to use a different filename (don't use ".py") for language extensions. This option works with my option (2): the context contains the filename which can be used to decide to enable or not the code transformer. I understand that the code transformer must also install an importlib hook to search for other filenames than only .py files. Am I right? (6) Nick proposed (indirectly) to use an encoding cookie "which are visible as a comment in the module header". Again, I dislike this option because comments are not stored in AST. Victor

4 years, 10 months

Re: [Python-ideas] Respectively and its unpacking sentence

by Mirmojtaba Gharibi

On Thu, Jan 28, 2016 at 3:26 AM, Mirmojtaba Gharibi < mojtaba.gharibi(a)gmail.com> wrote: > > > On Wed, Jan 27, 2016 at 4:15 PM, Andrew Barnert <abarnert(a)yahoo.com> > wrote: > >> On Jan 27, 2016, at 09:55, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com> >> wrote: >> >> On Wed, Jan 27, 2016 at 2:29 AM, Andrew Barnert <abarnert(a)yahoo.com> >> wrote: >> >>> On Jan 26, 2016, at 22:19, Mirmojtaba Gharibi <mojtaba.gharibi(a)gmail.com> >>> wrote: >>> >>> Yes, I'm aware sequence unpacking. >>> There is an overlap like you mentioned, but there are things that can't >>> be done with sequence unpacking, but can be done here. >>> >>> For example, let's say you're given two lists that are not necessarily >>> numbers, so you can't use numpy, but you want to apply some component-wise >>> operator between each component. This is something you can't do with >>> sequence unpacking or with numpy. >>> >>> >>> Yes, you can do it with numpy. >>> >>> Obviously you don't get the performance benefits when you aren't using >>> "native" types (like int32) and operations that have vectorizes >>> implementations (like adding two arrays of int32 or taking the dot product >>> of float64 matrices), but you do still get the same elementwise operators, >>> and even a way to apply arbitrary callables over arrays, or even other >>> collections: >>> >>> >>> firsts = ['John', 'Jane'] >>> >>> lasts = ['Smith', 'Doe'] >>> >>> np.vectorize('{1}, {0}'.format)(firsts, lasts) >>> array(['Smith, John', 'Doe, Jane'], dtype='<U11) >>> >>> I think the form I am suggesting is simpler and more readable. >> >> >> But the form you're suggesting doesn't work for vectorizing arbitrary >> functions, only for operator expressions (including simple function calls, >> but that doesn't help for more general function calls). The fact that numpy >> is a little harder to read for cases that your syntax can't handle at all >> is hardly a strike against numpy. >> > > I don't need to vectorize the functions. It's already being done. > Consider the ; example below: > a;b = f(x;y) > it is equivalent to > a=f(x) > b=f(y) > So in effect, in your terminology, it is already vectorized. > Similar example only with $: > a=[0,0,0,0] > x=[1,2,3,4] > $a=f($x) > is equivalent to > a=[0,0,0,0] > x=[1,2,3,4] > for i in range(len(a)): > ...a[i]=f(x[i]) > > > >> >> And, as I already explained, for the cases where your form _does_ work, >> numpy already does it, without all the sigils: >> >> c = a + b >> >> c = a*a + 2*a*b + b*b >> >> c = (a * b).sum() >> >> It also works nicely over multiple dimensions. For example, if a and b >> are both arrays of N 3-vectors instead of just being 3-vectors, you can >> still elementwise-add them just with +; you can sum all of the results with >> sum(axis=1); etc. How would you write any of those things with your >> $-syntax? >> >> I'm happy you brought vectorize to my attention though. I think as soon >> you make the statement just a bit complex, it would become really >> complicated with vectorize. >> >> >> For example lets say you have >> x=[1,2,3,4,5,...] >> y=['A','BB','CCC',...] >> p=[2,3,4,6,6,...] >> r=[]*n >> >> $r = str(len($y*$p)+$x) >> >> >> As a side note, []*n is always just []. Maybe you meant [None for _ in >> range(n)] or [None]*n? Also, where does n come from? It doesn't seem to >> have anything to do with the lengths of x, y, and p. So, what happens if >> it's shorter than them? Or longer? With numpy, of course, that isn't a >> problem--there's no magic being attempted on the = operator (which is good, >> because = isn't an operator in Python, and I'm not sure how you'd even >> properly define your design, much less implement it); the operators just >> create arrays of the right length. >> >> n I just meant symbolically to be len(x). So please replace n with > len(x). I didn't mean to confuse you. sorry. > > >> Anyway, that's still mostly just operators. You _could_ wrap up an >> operator expression in a function to vectorize, but you almost never want >> to. Just use the operators directly on the arrays. >> >> So, let's try a case that has even some minimal amount of logic, where >> translating to operators would be clumsy at best: >> >> @np.vectorize >> def sillyslice(y, x, p): >> if x < p: return y[x:p] >> return y[p:x] >> >> r = sillyslice(y, x, p) >> >> Being a separate function provides all the usual benefits: sillyslice is >> reusable, debuggable, unit-testable, usable as a first-class object, etc. >> But forget that; how would you do this at all with your $-syntax? >> >> Since you didn't answer any of my other questions, I'll snip them and >> repost shorter versions: >> >> * what's wrong with using numpy? Nothing. What's wrong even with for loop >> or assembly for that matter? I didn't argue that it's not possible to >> achieve these things with assembly. >> * what's wrong with APL or J or MATLAB? Not sure how relevant it is to >> our core of conversation. Skipping this. >> * what's wrong with making the operators elementwise instead of wrapping >> the objects in some magic thing? The fact that whenever you >> * what is the type of that magic thing anyway? It has no type. I refer >> you to my very first email. In that email I exactly explained what it >> means. It's at best a psuedo macro or something like that. It exactly is >> equivalent when you write >> > a;b=f(x;y) > to > a=f(x) > b=f(y) > > In other words, if I could interpret my code before python interpreter > interpret it, I would convert the first to the latter. > >

4 years, 10 months

several different needs [Explicit variable capture list]

by Jim J. Jewett

I think a small part of the confusion is that there are at least four separate (albeit related) use cases. They all use default arguments for the current workarounds, but they don't have the same ideal solution. (1) Auxiliary variables def f(x, _len=len): ... This is often a micro-optimization; the _len keyword really shouldn't be overridden. Partly because it shouldn't be overridden, having it in the signature is just ugly. This could be solved with another separator in the signature, such as ; or a second () or a new keyword ... def f(x, aux _len=len): ... def f(x, once _len=len): ... def f(x; _len=len):... def f(x)(_len=len): ... def f(x){_len=len}: ... But realistically, that _len isn't ugly *just* because it shouldn't be overridden; it is also inherently ugly. I would prefer that something like Victor's FAT optimizer just make this idiom obsolete. (2) immutable bindings once X final Y const Z This is pretty similar to the auxiliary variables case, except that it tends to be desired more outside of functions. The immutability can be worth documenting on its own, but it feels too much like a typing declaration, which raises questions of "why *this* distinction for *this* variable?" So again, I think something like Victor's FAT optimizer (plus comments when immutability really is important) is a better long-term solution, but I'm not as sure as I was for case 1. (3) Persistent storage def f(x, _cached_results={}): ... In the past, I've managed to convince myself that it is good to be able to pass in a different cache ... or to turn the function into a class, so that I could get to self, or even to attach attributes to the function itself (so that rebinding the name to another function in a naive manner would fail, rather than produces bugs). Those convincings don't stick very well, though. This was clearly at least one of the motivations of some people who asked about static variables. I still think it might be nice to just have a way of easily opening a new scope ... but then I have to explain why I can't just turn the function into a class... So in the end, I suspect this use case is better off ignored, but I am pretty sure it will lead to some extra confusion if any of the others are "solved" in a way that doesn't consider it. (4) Current Value Capture This is the loop variable case that some have thought was the only case under consideration. I don't have anything to add to Andrew Barnert's https://mail.python.org/pipermail/python-ideas/2016-January/037980.html but do see Steven D'Aprano's https://mail.python.org/pipermail/python-ideas/2016-January/038047.html for gotchas even within this use case. -jJ

4 years, 10 months

intput()

by Marcel O'Neil

def intput(): return int(input()) Life would be just marginally easier, with a punny function name as a bonus.

4 years, 10 months

PEP 484 change proposal: Allowing @overload outside stub files

by Guido van Rossum

Ben Darnell (Tornado lead) brought up a good use case for allowing @overload in regular Python files. There's some discussion (some old, some new) here: https://github.com/ambv/typehinting/issues/72 I now propose to allow @overload in non-stub (i.e. .py) files, but with the following rule: a series of @overload-decorated functions must be followed by an implementation function that's not @overload-decorated. Calling an @overload-decorated function is still an error (I propose NotImplemented). Due to the way repeated function definitions with the same name replace each other, leaving only the last one active, this should work. E.g. for Tornado's utf8() the full definition would look like this: @overloaddef utf8(value: None) -> None: ...@overloaddef utf8(value: bytes) -> bytes: ...@overloaddef utf8(value: str) -> bytes: ... # or (unicode)->bytes, in PY2def utf8(value): # Real implementation goes here. NOTE: If you are trying to understand why we can't use a stub file here or why we can't solve this with type variables or unions, please read the issue and comment there if things are not clear. Here on python-ideas I'd like to focus on seeing whether this amendment is non-controversial (apart from tea party members who just want to repeal PEP 484 entirely :-). I know that previously we wanted to come up with a complete solution for multi-dispatch based on type annotations first, and there are philosophical problems with using @overload (though it can be made to work using sys._getframe()). The proposal here is *not* that solution. If you call the @overload-decorated function, it will raise NotImplemented. (But if you follow the rule, the @overload-decorated function objects are inaccessible so this would only happen if you forgot or misspelled the final, undecorated implementation function). -- --Guido van Rossum (python.org/~guido)

4 years, 10 months

Making Python great again

by Sven R. Kunze

Hi, for all those who felt that something is wrong with Python. Here's the solution: https://github.com/samshadwell/TrumpScript ;) Best, Sven

4 years, 10 months

Documenting asyncio methods as returning awaitables

by Ian Kelly

The official asyncio documentation includes this note: """ Note: In this documentation, some methods are documented as coroutines, even if they are plain Python functions returning a Future. This is intentional to have a freedom of tweaking the implementation of these functions in the future. If such a function is needed to be used in a callback-style code, wrap its result with ensure_future(). """ Despite the note, this still causes confusion. See for example https://mail.python.org/pipermail/python-list/2016-January/702342.html As of Python 3.5, "awaitable" is a thing, and as of Python 3.5.1, ensure_future is supposed to accept any awaitable. Would it be better then to document these methods as returning awaitables rather than as coroutines?

4 years, 10 months

Cross link documentation translations

by Julien Palard

o/ While translating the Python Documentation in French [1][2], I discovered that we're not the only country doing it, there is also Japan [3][4], and Spain [5]. It's possible there's other but I didn't find them (and it's the problem). But there's only a few way for users to find the translations (hearing about them, or explicitly searching for them on a search engine, which they won't do, obviously expecting a link from the english version if they exists). So here is my idea: Why not linking translations from the main documentation? I know that's not directly supported by Sphinx doc [6], but separate sphinx build, blindly (with hardcoded links) linking themselves, may work (like readthedoc is probably doing). The downside of those links is that we'll sometime link to untranslated parts, but those parts may be marked as untranslated [7] to encourage new translators to help. Thoughts? [1] http://www.afpy.org/doc/python/3.5/ [2] https://github.com/afpy/python_doc_fr [3] http://docs.python.jp/3/ [4] https://github.com/python-doc-ja/python-doc-ja [5] http://docs.python.org.ar/tutorial/3/index.html [6] https://github.com/sphinx-doc/sphinx/issues/2252 [7] https://github.com/sphinx-doc/sphinx/issues/1246 -- Julien Palard

4 years, 10 months

Multiple dispatch (was Re: PEP 484 change proposal: Allowing <at> overload outside stub files

by Nick Coghlan

On 24 January 2016 at 07:12, Stefan Krah <skrah.temporarily(a)gmail.com> wrote: > Nick Coghlan <ncoghlan@...> writes: >> You're already going to have to allow this for single lines to handle >> Py2 compatible annotations, so it seems reasonable to also extend it >> to handle overloading while you're still figuring out a native syntax >> for that. > > I find that https://pypi.python.org/pypi/multipledispatch looks quite > nice: > >>>> from multipledispatch import dispatch >>>> @dispatch(int, int) > ... def add(x, y): > ... return x + y > ... >>>> @dispatch(float, float) > ... def add(x, y): > ... return x + y > ... >>>> add(1, 2) > 3 >>>> add(1.0, 2.0) > 3.0 >>>> add(1.0, 2) > Traceback (most recent call last): > File > [cut because gmane.org is inflexible] > line 155, in __call__ > func = self._cache[types] > KeyError: (<class 'float'>, <class 'int'>) Right, the Blaze folks have been doing some very nice work in that area. One of the projects building on multipledispatch is the Odo network of data conversion operations: https://github.com/blaze/odo They do make the somewhat controversial design decision to make dispatch operations process global by default [1], rather than scoping by module. On the other hand, the design also makes it easy to define your own dispatch namespace, so the default orthogonality with the module system likely isn't a problem in practice, and the lack of essential boilerplate does make it very easy to use in contexts like an IPython notebook. There is one aspect that still requires runtime stack introspection [2], and that's getting access to the class scope in order to implicitly make method dispatch specific to the class defining the methods. It's the kind of thing that makes me wonder whether we should be exposing a thread-local variable somewhere with a "class namespace stack" that made it possible to: - tell that you're currently running in the context of a class definition - readily get access to the namespace of the innermost class currently being defined Cheers, Nick. [1] http://multiple-dispatch.readthedocs.org/en/latest/design.html#namespaces-a… [2] https://github.com/mrocklin/multipledispatch/blob/master/multipledispatch/c… -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 years, 10 months

Proposal to extend PEP 484 (gradual typing) to support Python 2.7

by Agustín Herranz Cecilia

Hi!, I'd come to this thread late and by coincidence, but after read the whole thread I want to share some thoughts: The main concern it's add a way to add some kind of gradual typing to python 2 code. Because it's working python2 code it can't use annotations and it can't be added in code, so a type comment is the way to go (independent of the convenience, or not, of having type annotation available on runtime). Someone points that using a comment with the python3 annotation signature it's good to educate people on how to use annotations, I feel that's not the point, the point is to people get used to type hints. For this the same syntax must be used across different python versions, so 'function type comments' must be available to use also in python 3 code, this also allow people who can't/won't use annotations to use type hinting. For this, I don't believe that the section "Suggested syntax for Python 2.7 and straddling code" added to PEP 484 is the correct way to go, the proper it's add type comments for functions, as a extension of the "Type comments" section or perhaps in a new PEP. Some concerns that must be take into account to add function type comments: - Using 'type comments' syntax of PEP484 a function signature should look like this: def func(arg1, arg2): # type: Callable[[int, int], int] """ Do something """ return arg1 + arg2 - This easily becomes a long line so breaks PEP8 and linters would complain. So it need to define a way to put the type comment in another line. The type comment will be put In the line after or in the line before? Put in another line will be only available for function type comments or for other type comments too? - With some kind of complex types the type comment surely become a long lint too, how the type comments will be break into different lines? - GVR proposal includes some kind of syntactic sugar for function type comments (" # type: (t_arg1, t_arg2) -> t_ret "). I think it's good but this must be an alternative over typing module syntax (PEP484), not the preferred way (for people get used to typehints). Is this syntactic sugar compatible with generators? The type analyzers could be differentiate between a Callable and a Generator? More concerns on type comments: - As this is intended to gradual type python2 code to port it to python 3 I think it's convenient to add some sort of import that only be used for type checking, and be only imported by the type analyzer, not the runtime. This could be achieve by prepending "#type: " to the normal import statement, something like: # type: import module # type: from package import module - Also there must be addressed how it work on a python2 to python3 environment as there are types with the same name, str for example, that works differently on each python version. If the code is for only one version uses the type names of that version. For 2/3 code types could be define with a "py2" prefix on a module that could be "py2types" having "py2str", for example, to mark things that be of python2 str type. Python 3 types will not have prefixes. I hope this reasoning/ideas will be useful. Also I hope that I have been expressed good enough, English is not my mother tongue. Agustín Herranz.

4 years, 10 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas January 2016