Python-ideas July 2016

python-ideas@python.org

75 participants
31 discussions

by Antony Lee

Hi all, I have opened http://bugs.python.org/issue27175 a month and a half ago but didn't get any feedback there, so I'll copy-paste it here for comments: Currently, pickling Path objects lead to issues when working across platforms: Paths (and, thus, objects that contain Paths) created on a POSIX platform (PosixPaths) cannot be unpickled on Windows and vice versa. There are a few possibilities around this issue. - Don't do anything about it, you should use PurePaths if you care about cross-platform compatibility: this would be pretty awkward, as any call to the Path API would require converting back the PurePath to a Path first. - Silently convert Paths to PurePaths during pickling (a solution that seems to have been adopted by http://docs.menpo.org/en/stable/api/menpo/io/export_pickle.html for example): it would be better if Paths at least roundtripped correctly within a single platform. - Convert Paths to PurePaths at unpickling time, only if the platform is different (and possibly with a warning): this is the least bad solution I came up with so far. Note that calls to the Path API on a "converted" PurePath object would be invalid anyways as the PurePath (being of a different platform) cannot be representing a valid path. Thoughts? Antony

3 years, 10 months

Use unbound bytes methods with objects supporting the buffer protocol

by Serhiy Storchaka

Unbound methods can be used as functions in python. bytes.lower(b) is the same as b.lower() if b is an instance of bytes. Many functions and methods that work with bytes accept not just bytes, but arbitrary objects that support the buffer protocol. Including bytes methods: >>> b'a:b'.split(memoryview(b':')) [b'a', b'b'] But the first argument of unbound bytes method can be only a bytes instance. >>> bytes.split(memoryview(b'a:b'), b':') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: descriptor 'split' requires a 'bytes' object but received a 'memoryview' I think it would be helpful to allow using unbound bytes methods with arbitrary objects that support the buffer protocol as the first argument. This would allow to avoid unneeded copying (the primary purpose of the buffer protocol). >>> bytes.split(memoryview(b'a:b'), b':') [b'a', b'b']

3 years, 10 months

Re: [Python-ideas] allow `lambda' to be spelled ?

by אלעזר

`def` is a much more readable alternative, which will be more familiar to newcomers. The expression/block ambiguity can be resolved using different token instead of colon (perhaps := or an arrow). I can't understand the reaching for the Greek letter - it is a computer-science specific term with no intuitive meaning. On Thu, Jul 14, 2016 at 4:49 PM <python-ideas-request(a)python.org> wrote: > Send Python-ideas mailing list submissions to > python-ideas(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request(a)python.org > > You can reach the person managing the list at > python-ideas-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Re: allow `lambda' to be spelled ? (David Mertz) > 2. Re: Use unbound bytes methods with objects supporting the > buffer protocol (Nick Coghlan) > 3. Re: allow `lambda' to be spelled ? (Random832) > 4. Re: allow `lambda' to be spelled ? (SW) > 5. Re: allow `lambda' to be spelled ? (Chris Angelico) > 6. Re: allow `lambda' to be spelled ? (SW) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 13 Jul 2016 23:39:26 -0700 > From: David Mertz <mertz(a)gnosis.cx> > To: "Steven D'Aprano" <steve(a)pearwood.info> > Cc: python-ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] allow `lambda' to be spelled ? > Message-ID: > < > CAEbHw4aVKCXLu2DC7K6ArxVHAjOkc15yk+irtPKfBa5xEy0G2A(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Wed, Jul 13, 2016 at 7:44 PM, Steven D'Aprano <steve(a)pearwood.info> > wrote: > > > - suppose that reading ? has a *tiny* benefit of 1% over "lambda" > > (for those who have learned what it means); > > - but typing it is (lets say) 50 times harder than typing "lambda"; > > - but we read code 50 times as often as we type it; > > - so the total benefit (50*1.01 - 50) is positive. > > > I actually *do* think ? is a little bit more readable. And I have no idea > how to type it directly on my El Capitan system with the ABC Extended > keyboard. But I still get 100% of the benefit in readability simply by > using vim's conceal feature. If I used a different editor I'd have to hope > for a similar feature (or program it myself), but this is purely a display > question. Similarly, I think syntax highlighting makes my code much more > readable, but I don't want colors for keywords built into the language. > That is, and should remain, a matter of tooling not core language (I don't > want https://en.wikipedia.org/wiki/ColorForth for Python). > > FWIW, my conceal configuration is at link I give in a moment. I've > customized a bunch of special stuff besides lambda, take it or leave it: > > http://gnosis.cx/bin/.vim/after/syntax/python.vim > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. >

3 years, 10 months

Proposal to add const-ness type hints to PEP-484

by Siva Chandra

Hello, I have submitted a proposal to add const-ness type hint constructs here: https://github.com/python/typing/issues/242 Looking for feedback on the proposal. Thanks, Siva Chandra

3 years, 10 months

Smoothing transition to Python 3

by Neil Schemenauer

I'm nearly finished porting a decently size application from Python 2 to Python 3. It has been a lot of work. I am dubious as to whether the port was really the best use of time. I can imagine there is still millions, possibly billions of lines of Python 2 code that has not yet been converted. Further, my guess is the lines of Python 2 code are still growing faster than the lines of Python 3 compatible code. IMHO, we need to do better in making it easier for people to port code. Here is a thought that occured to me. Create a patched version of Python 3.x, making a stepping stone version for people porting from Python 2. I know this would have been useful for me. Specific changes that would be helpful, all generating warnings so code can be eventually fixed: - comparision with None, smaller than all other types - comparision of distinct types: use Python 2 behavior (i.e. order by type name) - mixing of unicode strings with byte strings: decode/encode using latin-1 - dict objects: make keys() items() values() return special sequence that warns if iterated over multiple times or indexed as sequence Changes that are not necessary as porting code is easy: - print function syntax - try/except syntax - standard module renaming - __next__/next() - long type, literal syntax - true division as default

3 years, 10 months

Add an option that allow check Type Hints in runtime

by Ma Lin

Hi, all, Is it possible to add an option that allow check Type Hints in runtime? So that I can enable dynamic type checking when debugging. And disable it in final deploy, no performance lose. regards.

3 years, 11 months

Re: [Python-ideas] Python-ideas Digest, Vol 116, Issue 12

by אלעזר

Function definition can also be changed inside the interpreter, effectively doing the same work as tge interpreter. The parser can add implicit decorator too. This raises a different approach: adding a module decorator, allowing such implicit decorations accross module functions. Has it been discussed before? בתאריך יום ו׳, 8 ביול' 2016, 12:18, מאת ‏<python-ideas-request(a)python.org>: > Send Python-ideas mailing list submissions to > python-ideas(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request(a)python.org > > You can reach the person managing the list at > python-ideas-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Re: Add an option that allow check Type Hints in runtime > (Chris Angelico) > 2. Re: Add an option that allow check Type Hints in runtime > (Paul Moore) > 3. Re: Add an option that allow check Type Hints in runtime > (Chris Angelico) > 4. Re: Add an option that allow check Type Hints in runtime > (Chris Angelico) > 5. Re: Add an option that allow check Type Hints in runtime (Ma Lin) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 8 Jul 2016 18:30:52 +1000 > From: Chris Angelico <rosuav(a)gmail.com> > Cc: python-ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Add an option that allow check Type Hints > in runtime > Message-ID: > <CAPTjJmpgFaxqd7gU407u5c+tURayv3OkvdTJ8RH7N= > KbkXXi3Q(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Fri, Jul 8, 2016 at 6:19 PM, Ma Lin <animalize81(a)hotmail.com> wrote: > > I mean real check by the interpreter, not an '__annotations__' attribute. > > And I just had a look, mypy is a static checker, without actually running > > the code. > > > > My thought is modifying CALL_FUNCTION and RETURN_VALUE of the > interpreter, > > so that allow them to check in each function calling when option enalbed. > > > > Cite a case: > > > > def fun(a: str, b: int) -> int: > > return len(a) + b > > fun('arg1', 'arg2') > > > > Whit the option and run this code, the interpreter will give an error > > prompt: > > Type of argument b is wrong. (or something like this) > > > > This is very useful for development and debugging. > > You should still be able to do it without changing the core interpreter. > > def enforce_types(func): > @functools.wraps(func) > def wrapped(*args, **kw): > if not args_valid(args, kw, func.__annotations__): > raise TypeError > ret = func(*args, **kw) > if not type_valid(ret, func.__annotations__["return"]): > raise TypeError > return wrapped > > @enforce_types > def fun(a: str, b: int) -> int: > return len(a) + b > > fun('arg1', 'arg2') > > > Now all you have to do is write the type_valid function (takes an > object and an annotation, returns True if it matches) and args_valid > (takes the positional and keyword args, pairs them up against the > function's params, and returns True if all(type_valid(...)) for the > args). The only benefit of hacking the interpreter core for this is > that you'd save the hassle of pairing up positional/keyword args > against the parameters. > > ChrisA > > > ------------------------------ > > Message: 2 > Date: Fri, 8 Jul 2016 09:50:53 +0100 > From: Paul Moore <p.f.moore(a)gmail.com> > To: Chris Angelico <rosuav(a)gmail.com> > Cc: python-ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Add an option that allow check Type Hints > in runtime > Message-ID: > < > CACac1F-+Si-uXwce2CKPiyM576MX7G+Cp4u8n-TXBO_89UTbdQ(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On 8 July 2016 at 09:30, Chris Angelico <rosuav(a)gmail.com> wrote: > > On Fri, Jul 8, 2016 at 6:19 PM, Ma Lin <animalize81(a)hotmail.com> wrote: > >> I mean real check by the interpreter, not an '__annotations__' > attribute. > >> And I just had a look, mypy is a static checker, without actually > running > >> the code. > >> > >> My thought is modifying CALL_FUNCTION and RETURN_VALUE of the > interpreter, > >> so that allow them to check in each function calling when option > enalbed. > [...] > > > > Now all you have to do is write the type_valid function (takes an > > object and an annotation, returns True if it matches) and args_valid > > (takes the positional and keyword args, pairs them up against the > > function's params, and returns True if all(type_valid(...)) for the > > args). The only benefit of hacking the interpreter core for this is > > that you'd save the hassle of pairing up positional/keyword args > > against the parameters. > > Just to clarify for the OP - the reason you're getting pushback on > making this a built in interpreter function is twofold - first, an > external facility could be used *now*, and would not need to be > limited to Python 3.6+, and second, adding checks to the function call > protocol is not free, and would therefore affect not just the minority > of people who want to add checks for debugging, but also all other > users (which include people for whom Python's performance is a key > issue). The Python interpreter's function calling process is known to > be slower than (some) people would like, and making it slower - even > marginally - isn't something we would want to do lightly. > > With the ability to use external static analyzers like mypy, as well > as the possibility of writing custom solutions like Chris described > above, people wanting such checks have a number of options already, > and so the cost of adding the feature to the interpreter (and possibly > the language - do you see this as a CPython implementation detail, or > would you expect it to be a language-level facility, so other Python > implementations such as PyPy and Jython would be expected to have it > as well?) needs to be justified. > > Personally, I'd view this feature as an interesting addition, but not > something I'd use myself, and so I'm inclined towards the "why should > I pay for something I won't use?" position. > > Paul > > > ------------------------------ > > Message: 3 > Date: Fri, 8 Jul 2016 19:02:21 +1000 > From: Chris Angelico <rosuav(a)gmail.com> > Cc: python-ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Add an option that allow check Type Hints > in runtime > Message-ID: > <CAPTjJmp_=6PFfnKEeboGuXm= > ffM+NB_ysUPNQxvXJUm4UtbFZg(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Fri, Jul 8, 2016 at 6:50 PM, Paul Moore <p.f.moore(a)gmail.com> wrote: > > Just to clarify for the OP - the reason you're getting pushback on > > making this a built in interpreter function is twofold - first, an > > external facility could be used *now*, and would not need to be > > limited to Python 3.6+, and second, adding checks to the function call > > protocol is not free, and would therefore affect not just the minority > > of people who want to add checks for debugging, but also all other > > users (which include people for whom Python's performance is a key > > issue). The Python interpreter's function calling process is known to > > be slower than (some) people would like, and making it slower - even > > marginally - isn't something we would want to do lightly. > > Exactly. The main reason IMO is the first one - you can add this > checker to any Python 3 program, rather than wait for it to get coded, > debugged, and shipped. > > Oh, and you can easily have a flag that disables checking for performance: > > if no_type_checks: > def enforce_types(func): > return func > > Now your decorator does nothing, and there is literally ZERO run-time > cost (the only cost is a single cheap check when the function is > defined, which usually means on module import), closures aside. That > wouldn't be possible with a core interpreter change, unless it's made > a compile-time option *for the interpreter* - something like > "./configure --with-annotations-enforced" - and then you run python3.6 > for the fast one, or python3.6tc for the type-checking one. That would > be a pain. > > ChrisA > > > ------------------------------ > > Message: 4 > Date: Fri, 8 Jul 2016 19:05:44 +1000 > From: Chris Angelico <rosuav(a)gmail.com> > Cc: python-ideas <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Add an option that allow check Type Hints > in runtime > Message-ID: > < > CAPTjJmrJstnRzEK48Kj7gOZDi5kkfBPrUCNYQqGds3qje-ZFbA(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Fri, Jul 8, 2016 at 7:02 PM, Chris Angelico <rosuav(a)gmail.com> wrote: > > Now your decorator does nothing, and there is literally ZERO run-time > > cost (the only cost is a single cheap check when the function is > > defined, which usually means on module import), closures aside. That > > wouldn't be possible with a core interpreter change... > > I really shouldn't say impossible. Literally the moment I clicked > Send, I thought of a way it could be done with minimal run-time cost: > just create a completely separate opcode for "type-checked call". I > don't think that would materially slow down the regular call > operation. As a bonus, any functions that don't have annotations could > use the normal call opcode, so they wouldn't be slowed down. > > But the main point is, this would all entail additional complexity > inside CPython, where the decorator method keeps the complexity in > your application - and doesn't need nearly as much of it. The one > advantage that CPython has is, as mentioned, the matching of arguments > to parameters; maybe that could be provided in functools somewhere? > > ChrisA > > > ------------------------------ > > Message: 5 > Date: Fri, 8 Jul 2016 09:17:56 +0000 > From: Ma Lin <animalize81(a)hotmail.com> > To: "python-ideas(a)python.org" <python-ideas(a)python.org> > Subject: Re: [Python-ideas] Add an option that allow check Type Hints > in runtime > Message-ID: > < > PS1PR06MB1099F602EB64A07AAC2063F0B63C0(a)PS1PR06MB1099.apcprd06.prod.outlook.com > > > > Content-Type: text/plain; charset="Windows-1252" > > I see, Chris' decorator is amazing, I didn't think that way. > > I mean add an runtime option, not a build option, it can be only enabled > in development/debugging: > python -tc a.py > Without the option, the interpreter just ignores Type Hints: > python a.py > > Anyway, I just throw an idea to here. Greet to all of you for bring > Python to me. > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > > > ------------------------------ > > End of Python-ideas Digest, Vol 116, Issue 12 > ********************************************* >

3 years, 11 months

should `dict` supply a default `__missing__` implementation?

by Matt Gilson

The following code raises an `AttributeError`: >>> class D(dict): ... def __missing__(self, k): ... super(D, self).__missing__(k) ... >>> d = D() >>> d['key'] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in __missing__ AttributeError: 'super' object has no attribute '__missing__' I find this behavior to be a little bit odd as I would have expected the default implementation to have a `__missing__` method and I would expect the `__missing__` method to raise an appropriate `KeyError`. I think that this would facilitate a class hierarchy where each class can decide "I don't know hot to handle this missing key, maybe something further up the MRO does". Currently, to do that (and adhere to the ["Super considered Super"](https://rhettinger.wordpress.com/2011/05/26/super-considered-super/) methodology) you'd have to add an extra base-class in the hierarchy which seems unnecessary. Obviously this is probably a pretty niche case and may not be worth considering, but I thought I'd throw it out there just in case. Also, FWIW, I believe that the current behavior is correct based on [the documentation](https://docs.python.org/2/library/stdtypes.html#dict) (under `d[key]`).

3 years, 11 months

__enter__ and __exit__ on generators

by Nathaniel Smith

Hi all, This might be too trivial for python-ideas even, but... I recently stumbled slightly over the way that generators define a close method (the one that injects a GeneratorExit), but don't directly implement the context manager protocol. So if you want a generator's internal resources to be cleaned up promptly, you can't write with genfunc() as it: for obj in it: ... but instead have to write with contextlib.closing(genfunc()) as it: for obj in it: ... Is there any particular reason for this, or is it just an oversight? (Why is it important to close generators? Consider a generator that holds an open file descriptor: def genfunc(): with open(...) as f: for line in f: yield ... Python these days very much encourages you to use a "with" statement for the file open, with ResourceWarnings and all that. But in this case, the file won't be closed until someone calls generator.close(), so the with statement above is only really effective if genfunc's caller also uses a "with" statement. Unfortunately, having the "with" statement inside genfunc *is* enough to unconditionally hide the ResourceWarning, even if it's otherwise ineffective, because the warning is triggered when file.__del__ calls file.__close__, and here we have generator.__del__ calling file.__close__.) -n

3 years, 11 months

Unpacking a dict

by Michael Selik

Python's iterable unpacking is what Lispers might call a destructuring bind. py> iterable = 1, 2, 3, 4, 5 py> a, b, *rest = iterable py> a, b, rest (1, 2, (3, 4, 5)) Clojure also supports mapping destructuring. Let's add that to Python! py> mapping = {"a": 1, "b": 2, "c": 3} py> {"a": x, "b": y, "c": z} = mapping py> x, y, z (1, 2, 3) py> {"a": x, "b": y} = mapping Traceback: ValueError: too many keys to unpack This will be approximately as helpful as iterable unpacking was before PEP 3132 (https://www.python.org/dev/peps/pep-3132/). I hope to keep discussion in this thread focused on the most basic form of dict unpacking, but we could extended mapping unpacking similarly to how PEP 3132 extended iterable unpacking. Just brainstorming... py> mapping = {"a": 1, "b": 2, "c": 3} py> {"a": x, **rest} = mapping py> x, rest (1, {"b": 2, "c": 3})

3 years, 11 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas July 2016