Mailman 3 January 2016 - Python-ideas

Proposal to extend PEP 484 (gradual typing) to support Python 2.7
by Agustín Herranz Cecilia Jan. 23, 2016

Jan. 23, 2016

Hi!, I'd come to this thread late and by coincidence, but after read the whole thread I want to share some thoughts: The main concern it's add a way to add some kind of gradual typing to python 2 code. Because it's working python2 code it can't use annotations and it can't be added in code, so a type comment is the way to go (independent of the convenience, or not, of having type annotation available on runtime). Someone points that using a comment with the python3 annotation signature … [View More]it's good to educate people on how to use annotations, I feel that's not the point, the point is to people get used to type hints. For this the same syntax must be used across different python versions, so 'function type comments' must be available to use also in python 3 code, this also allow people who can't/won't use annotations to use type hinting. For this, I don't believe that the section "Suggested syntax for Python 2.7 and straddling code" added to PEP 484 is the correct way to go, the proper it's add type comments for functions, as a extension of the "Type comments" section or perhaps in a new PEP. Some concerns that must be take into account to add function type comments: - Using 'type comments' syntax of PEP484 a function signature should look like this: def func(arg1, arg2): # type: Callable[[int, int], int] """ Do something """ return arg1 + arg2 - This easily becomes a long line so breaks PEP8 and linters would complain. So it need to define a way to put the type comment in another line. The type comment will be put In the line after or in the line before? Put in another line will be only available for function type comments or for other type comments too? - With some kind of complex types the type comment surely become a long lint too, how the type comments will be break into different lines? - GVR proposal includes some kind of syntactic sugar for function type comments (" # type: (t_arg1, t_arg2) -> t_ret "). I think it's good but this must be an alternative over typing module syntax (PEP484), not the preferred way (for people get used to typehints). Is this syntactic sugar compatible with generators? The type analyzers could be differentiate between a Callable and a Generator? More concerns on type comments: - As this is intended to gradual type python2 code to port it to python 3 I think it's convenient to add some sort of import that only be used for type checking, and be only imported by the type analyzer, not the runtime. This could be achieve by prepending "#type: " to the normal import statement, something like: # type: import module # type: from package import module - Also there must be addressed how it work on a python2 to python3 environment as there are types with the same name, str for example, that works differently on each python version. If the code is for only one version uses the type names of that version. For 2/3 code types could be define with a "py2" prefix on a module that could be "py2types" having "py2str", for example, to mark things that be of python2 str type. Python 3 types will not have prefixes. I hope this reasoning/ideas will be useful. Also I hope that I have been expressed good enough, English is not my mother tongue. Agustín Herranz. [View Less]

8 20

Typehinting repo moved to python/typing
by Guido van Rossum Jan. 23, 2016

Jan. 23, 2016

This is just a note that with Benjamin's help we've moved the ambv/typehinting repo on GitHub into the python org, so its URL is now https://github.com/python/typing . This repo was used most intensely for discussions during PEP 484's drafting period. It also contains the code for typing.py, repackaged for earlier releases on PyPI. The issue tracker is still open for proposals to change PEP 484, which is not unheard of given its provisional status. If you find a pointer to the original … [View More]

1 0

Fwd: Why do equality tests between OrderedDict keys/values views behave not as expected?
by Alexandre Figura Jan. 21, 2016

Jan. 21, 2016

Hi, I recently compared OrderedDict instances while writing unit tests, and discovered an interesting behavior. If I create two ordered dictionaries with the same keys/values in the same order, I observe that their values are not equal when I compare them. I recently asked a question about this on Stackoverflow: http://stackoverflow.com/questions/34312674/why-values-of-an-ordereddict-ar… Moreover, another user observed that keys of ordered dictionaries are compared in an order insensitive … [View More]

12 40

Re: [Python-ideas] Boolean value of an Enum member
by Ethan Furman Jan. 16, 2016

Jan. 16, 2016

[resending to lists -- sorry, Greg] On 01/15/2016 12:36 PM, Greg Ewing wrote: > Ethan Furman wrote: >> So the question now is: for a standard Enum (meaning no other type >> besides Enum is involved) should __bool__ look to the value of the >> Enum member to determine True/False, or should we always be True by >> default and make the Enum creator add their own __bool__ if they want >> something different? > > Can't you just specify a starting value of 0 … [View More]

1 0

Boolean value of an Enum member
by Ethan Furman Jan. 16, 2016

Jan. 16, 2016

When Enum was being designed one of the questions considered was where to start autonumbering: zero or one. As I remember the discussion we chose not to start with zero because we didn't want an enum member to be False by default, and having a member with value 0 be True was discordant. So the functional API starts with 1 unless overridden. In fact, according to the Enum docs: The reason for defaulting to ``1`` as the starting number and not ``0`` is that ``0`` is ``False`` in a … [View More]

4 4

(FAT Python) Convert keyword arguments to positional?
by Franklin? Lee Jan. 14, 2016

Jan. 14, 2016

(FAT Python: http://faster-cpython.readthedocs.org/fat_python.html) FAT Python uses guards to check whether a global name (for example, the name for a function) has changed its value. Idea: If you know exactly which function will be called, you can also optimize based on the properties of that function. According to Eli Bendersky's 2012 blog post[1] (which might be outdated), a function call with keyword arguments is potentially slower than one with only positional arguments. """ If … [View More]

2 2

Password masking for getpass.getpass
by Muhammad Ahmed Khalid Jan. 14, 2016

Jan. 14, 2016

Greetings, I am working on a project and I am using getpass.getpass() to grab passwords from the user. Some of the users wanted asterisks to be displayed when they were typing in the passwords for feedback i.e. how many characters were typed and how many to backspace. Therefore I have created a solution but I think the feature should come by default (and the programmer should have the option to use a blank or any other character). The code lives here: https://goo.gl/OeY3QI Please let me … [View More]

12 22

Higher leavel heapq
by Joao S. O. Bueno Jan. 14, 2016

Jan. 14, 2016

Hi, the heapq stdlib module is really handy, but a little low level - in that it accepts a sequence, possibly only a list, as the heap-object, and that object have to be handled independently, outside the functions provided in there. (One can't otherwise insert or delete elements of that list, without destroying the heap, for example). It would be simple to have a higher level class that would do just that, and simplify the use of an ordered container - what about having an extra class there? … [View More]

3 2

Fwd: Using functools.lru_cache only on some arguments of a function
by Bill Winslow Jan. 13, 2016

Jan. 13, 2016

Sorry for the late reply everyone. I think relying on closures, while a solution, is messy. I'd still much prefer a way to tell lru_cache to merely ignore certain arguments. I'll use some variant of the storing-the-partial-progress-as-an-attribute-on-the-cached-recursive-function method (either with the cached-recursive hidden from the top level via the try/except stuff, or with a more simple wrapper function). I've further considered my original proposal, and rather than naming it "… [View More]arg_filter", I realized that builtins like sorted(), min(), max(), etc all already have the exact same thing -- a "key" argument which transforms the elements to the user's purpose. (In the sorted/min/max case, it's called on the elements of the argument rather than the argument itself, but it's still the same concept.) So basically, my original proposal with renaming from arg_filter to key, is tantamount to extending the same functionality from sorted/min/max to lru_cache as well. As has been pointed out, my own use case is almost certainly *not* the only use case. The implementation and interface are both simple, and simpler than the alternatives which I'll rely on for now (wrappers and closures, or worse, global singletons etc). I would still like to see it in the stdlib in the future. I've appended a largely similar patch with the proposed additions (there's some internal variable renaming to avoid confusion, resulting in a longer diff). Thanks again for all the input. -Bill ----------------------------------------------------------------------------------------------------------------- https://hg.python.org/cpython/file/3.5/Lib/functools.py diff functools.py.orig functools.py 363c363 < def _make_key(args, kwds, typed, --- > def _make_key(args, kwds, typed, key, 377c377,379 < key = args --- > if key is not None: > args, kwds = key(args, kwds) > cache_key = args 380c382 < key += kwd_mark --- > cache_key += kwd_mark 382c384 < key += item --- > cache_key += item 384c386 < key += tuple(type(v) for v in args) --- > cache_key += tuple(type(v) for v in args) 386,389c388,391 < key += tuple(type(v) for k, v in sorted_items) < elif len(key) == 1 and type(key[0]) in fasttypes: < return key[0] < return _HashedSeq(key) --- > cache_key += tuple(type(v) for k, v in sorted_items) > elif len(cache_key) == 1 and type(cache_key[0]) in fasttypes: > return cache_key[0] > return _HashedSeq(cache_key) 391c393 < def lru_cache(maxsize=128, typed=False): --- > def lru_cache(maxsize=128, typed=False, key=None): 400a403,407 > If *key* is not None, it must be a callable which acts on the arguments > passed to the function. Its return value is used in place of the actual > arguments. It works analogously to the *key* argument to the builtins > sorted, max, and min. > 421a429,431 > if key is not None and not callable(key): > raise TypeErrpr('Expected key to be a callable') > 423c433,434 < wrapper = _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo) --- > wrapper = _lru_cache_wrapper(user_function, maxsize, typed, key, > _CacheInfo) 428c439 < def _lru_cache_wrapper(user_function, maxsize, typed, _CacheInfo): --- > def _lru_cache_wrapper(user_function, maxsize, typed, key, _CacheInfo): 456,457c467,468 < key = make_key(args, kwds, typed) < result = cache_get(key, sentinel) --- > cache_key = make_key(args, kwds, typed, key) > result = cache_get(cache_key, sentinel) 462c473 < cache[key] = result --- > cache[cache_key] = result 471c482 < key = make_key(args, kwds, typed) --- > cache_key = make_key(args, kwds, typed, key) 473c484 < link = cache_get(key) --- > link = cache_get(cache_key) 487c498 < if key in cache: --- > if cache_key in cache: 496c507 < oldroot[KEY] = key --- > oldroot[KEY] = cache_key 513c524 < cache[key] = oldroot --- > cache[cache_key] = oldroot 517,518c528,529 < link = [last, root, key, result] < last[NEXT] = root[PREV] = cache[key] = link --- > link = [last, root, cache_key, result] > last[NEXT] = root[PREV] = cache[cache_key] = link On Wed, Dec 30, 2015 at 11:10 PM, Michael Selik <mike(a)selik.org> wrote: > On Tue, Dec 29, 2015 at 2:14 AM Franklin? Lee < > leewangzhong+python(a)gmail.com> wrote: > >> On Sat, Dec 12, 2015 at 1:34 PM, Michael Selik <mike(a)selik.org> wrote: >> > On Fri, Dec 11, 2015, 8:20 PM Franklin? Lee < >> leewangzhong+python(a)gmail.com> >> > wrote: >> > > This whole thing is probably best implemented as two separate functions >> > rather than using a closure, depending on how intertwined the code >> paths are >> > for the shortcut/non-shortcut versions. >> >> I like the closure because it has semantic ownership: the inner >> function is a worker for the outer function. >> > > True, a closure has better encapsulation, making it less likely someone > will misuse the helper function. On the other hand, that means there's less > modularity and it would be difficult for someone to use the inner function. > It's hard to know the right choice without seeing the exact problem the > original author was working on. > > >> >> On Fri, Dec 11, 2015 at 8:01 PM, Franklin? Lee >> >> <leewangzhong+python(a)gmail.com> wrote: >> >> > 1. Rewrite your recursive function so that the partial state is a >> >> > nonlocal variable (in the closure), and memoize the recursive part. >> > >> > I'd flip the rare-case to the except block and put the normal-case in >> the >> > try block. I believe this will be more compute-efficient and more >> readable. >> >> The rare case is in the except block, though. >> > > You're correct. Sorry, I somehow misinterpreted the comment, "# To trigger > the exception the first time" as indicating that code path would run only > once. > [View Less]

3 4

Proposal to extend PEP 484 (gradual typing) to support Python 2.7
by Guido van Rossum Jan. 13, 2016

Jan. 13, 2016

At Dropbox we're trying to be good citizens and we're working towards introducing gradual typing (PEP 484) into our Python code bases (several million lines of code). However, that code base is mostly still Python 2.7 and we believe that we should introduce gradual typing first and start working on conversion to Python 3 second (since having static types in the code can help a big refactoring like that). Since Python 2 doesn't support function annotations we've had to look for alternatives. We … [View More]considered stub files, a magic codec, docstrings, and additional `# type:` comments. In the end we decided that `# type:` comments are the most robust approach. We've experimented a fair amount with this and we have a proposal for a standard. The proposal is very simple. Consider the following function with Python 3 annotations: def embezzle(self, account: str, funds: int = 1000000, *fake_receipts: str) -> None: """Embezzle funds from account using fake receipts.""" <code goes here> An equivalent way to write this in Python 2 is the following: def embezzle(self, account, funds=1000000, *fake_receipts): # type: (str, int, *str) -> None """Embezzle funds from account using fake receipts.""" <code goes here> There are a few details to discuss: - Every argument must be accounted for, except 'self' (for instance methods) or 'cls' (for class methods). Also the return type is mandatory. If in Python 3 you would omit some argument or the return type, the Python 2 notation should use 'Any'. - If you're using names defined in the typing module, you must still import them! (There's a backport on PyPI.) - For `*args` and `**kwds`, put 1 or 2 starts in front of the corresponding type annotation. As with Python 3 annotations, the annotation here denotes the type of the individual argument values, not of the tuple/dict that you receive as the special argument value 'args' or 'kwds'. - The entire annotation must be one line. (However, see https://github.com/JukkaL/mypy/issues/1102.) We would like to propose this as a standard (either to be added to PEP 484 or as a new PEP) rather than making it a "proprietary" extension to mypy only, so that others in a similar situation can also benefit. A brief discussion of the considered alternatives: - Stub files: this would complicate the analysis in mypy quite a bit, because it would have to parse both the .py file and the .pyi file and somehow combine the information gathered from both, and for each function it would have to use the types from the stub file to type-check the body of the function in the .py file. This would require a lot of additional plumbing. And if we were using Python 3 we would want to use in-line annotations anyway. - A magic codec was implemented over a year ago ( https://github.com/JukkaL/mypy/tree/master/mypy/codec) but after using it for a bit we didn't like it much. It slows down imports, it requires a `# coding: mypy` declaration, it would conflict with pyxl ( https://github.com/dropbox/pyxl), things go horribly wrong when the codec isn't installed and registered, other tools would be confused by the Python 3 syntax in Python 2 source code, and because of the way the codec was implemented the Python interpreter would occasionally spit out confusing error messages showing the codec's output (which is pretty bare-bones). - While there are existing conventions for specifying types in docstrings, we haven't been using any of these conventions (at least not consistently, nor at an appreciable scale), and they are much more verbose if all you want is adding argument annotations. We're working on a tool that automatically adds type annotations[1], and such a tool would be complicated by the need to integrate the generated annotations into existing docstrings (which, in our code base, unfortunately are wildly incongruous in their conventions). - Finally, the proposed comment syntax is easy to mechanically translate into standard Python 3 function annotations once we're ready to let go of Python 2.7. __________ [1] I have a prototype of such a tool, i mplemented as a 2to3 fixer. It's a bit over 200 lines. It's not very interesting yet, since it sets the types of nearly all arguments to 'Any'. We're considering building a much more advanced version that tries to guess much better argument types using some form of whole-program analysis. I've heard that Facebook's Hack project got a lot of mileage out of such a tool. I don't yet know how to write it yet -- possibly we could use a variant of mypy's type inference engine, or alternatively we might be able to use something like Jedi ( https://github.com/davidhalter/jedi). -- --Guido van Rossum (python.org/~guido) [View Less]

14 29