Python-ideas September 2015

python-ideas@python.org

107 participants
55 discussions

Re: [Python-ideas] Structural type checking for PEP 484

by Jukka Lehtosalo

On Wed, Sep 9, 2015 at 10:48 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: > On Sep 9, 2015, at 21:34, Jukka Lehtosalo <jlehtosalo(a)gmail.com> wrote: > > I'm not sure if I fully understand what you mean by implicit vs. explicit > ABCs (and the static/runtime distinction). Could you define these terms and > maybe give some examples of each? > > > I just gave examples just one paragraph above. > > A (runtime) implicit ABC is something that uses a __subclasshook__ > (usually implementing a structural check). So, for instance, any type that > implements __iter__ is-a Iterable, e.g., according to isinstance or > issubclass or @singledispatch, because that's what > Iterable.__subclasshook__ checks for. > > A (runtime) explicit ABC is something that isn't implicit, like Sequence: > no hook, so nothing is-a Sequence unless it either inherits the ABC or > registers with it. > > You're proposing a parallel but separate distinction at static typing > time. Any ABC that's a Protocol is checked based on a structural check; > otherwise, it's checked based on inheritance. > In my proposal I actually suggest that protocols shouldn't support isinstance or issubclass (these operations should raise an exception) by default. A protocol is free to override the default exception-raising __subclasshook__ to implement a structural check, and a static type checker would allow isinstance and issubclass for protocols that do this. I'll need to explain this idea in more detail, as clearly the current explanation is too easy to misundertand. Here's a concrete example: class X(Protocol): def f(self): ... class A: def f(self): print('f') if isinstance(A(), X): ... # Raise an exception, because no __subclasshook__ override in X Previously I toyed with the idea of having a default implementation of __subclasshook__ that actually does a structural check, but I'm no longer sure if that would be desirable, as it's difficult to come up with an implementation that does the right thing in all reasonable cases. For example, consider a structural type like this that people might want to use to work around the current limitations of Callable (it doesn't support keyword arguments, for example): class MyCallable(Protocol): def __call__(self, x, y): ... (This example has some other potential issues that I'm hand-waving away for now.) Now how would the default isinstance work? Preferably it should only accept callables that are compatible with the signature, but doing that check is pretty difficult for arbitrary functions and should probably be out of scope for the typing module. Just checking whether __call__ exists would be too general, as the programmer probably expects that he's able to call the method with the specific arguments the type suggests. Also, sometimes checking the argument names would be a good thing to do, but sometimes any names (as long the the number of arguments is compatible) would be fine. > This means it's now possible to create supertypes that are implicit at > runtime but explicit at static typing time (which might occasionally be > useful), or vice-versa (which I can't imagine why you'd ever want). > As I showed above, you wouldn't get the latter unless you really try very hard (consenting adults and all). > > Besides the obvious negatives in having two not-quite-compatible and > very-different-looking ways of expressing the same concept, this is going > to lead to people wanting to know why their type checker is complaining > about perfectly good code ("I tested that constant with isinstance, and it > really is-a Spammable, and the type checker is inferring its type properly, > and yet I get an error passing it to a function that wants a Spammable") or > allowing blatantly invalid code ("I annotated my function to only take > Spammable arguments, but someone is passing something that calls the > fallback implementation of my singledispatch function instead of the > Spammable overload"). > I agree that having the default nominal/explicit isinstance semantics for a protocol type would be a very bad idea. > > Maybe the solution is to expand your proposal a little: make Protocol > automatically create a __subclasshook__ (which you listed as an optional > idea in the proposal), and also change all of the existing stdlib implicit > ABCs to Protocols and scrap their manual hooks, and also update the > relevant documentation (e.g., the abc module and the data model section on > __subclasshook__) to recommend using Protocol instead of implementing a > manual hook if the only thing you want is structural subtyping. Of course > the backward compatibility isn't perfect (unless you want to manually munge > up collections.abc when typing is imported), and people using legacy > third-party code might need to add stubs (although that seems necessary > anyway). But for most people, everything should just work as people expect. > A type is either structurally typed or explicitly (via inheritance or > registration) types, both at static typing time and a runtime, and that's > always expressed by the name Protocol. (But for the rare cases when you > really need a type check that's looser at runtime, you can still write a > manual hook to handle that.) > > Yeah, this would be nice, but as I argued above, implementing a generic __subclasshook__ is actually quite tricky. Jukka

6 years, 1 month

BUG in standard while statement

by Stephan Sahm

Dear all I found a BUG in the standard while statement, which appears both in python 2.7 and python 3.4 on my system. It usually won't appear because I only stumbled upon it after trying to implement a nice repeat structure. Look: ``` class repeat(object): def __init__(self, n): self.n = n def __bool__(self): self.n -= 1 return self.n >= 0 __nonzero__=__bool__ a = repeat(2) ``` the meaning of the above is that bool(a) returns True 2-times, and after that always False. Now executing ``` while a: print('foo') ``` will in fact print 'foo' two times. HOWEVER ;-) .... ``` while repeat(2): print('foo') ``` will go on and go on, printing 'foo' until I kill it. Please comment, explain or recommend this further if you also think that both while statements should behave identically. hoping for responses, best, Stephan

6 years, 1 month

Desperate need for enhanced print function

by Anand Krishnakumar

Hi! This is my first time I'm sending an email to the python-ideas mailing list. I've got an enhancement idea for the built-in print function and I hope it is as good as I think it is. Imagine you have a trial.py file like this: a = 4 b = "Anand" print("Hello, I am " + b + ". My favorite number is " + str(a) + ".") OR print("Hello, I am ", b, ". My favorite number is ", a, ".") Well I've got an idea for a function named "print_easy" (The only valid name I could come up with right now). So print_easy will be a function which will be used like this (instead of the current print statement in trial.py) : print_easy("Hello, I am", b, ". My favorite number is", a ".") Which gives out: Hello, I am Anand. My favorite number is 4 The work it does is that it casts the variables and it also formats the sentences it is provided with. It is exclusively for beginners. I'm 14 and I came up with this idea after seeing my fellow classmates at school struggling to do something like this with the standard print statement. Sure, you can use the format method but won't that be a bit too much for beginners? (Also, casting is inevitable in every programmer's career) Please let me know how this sounds. If it gains some traction, I'll work on it a bit more and clearly list out the features. Thanks, Anand. -- Anand.

6 years, 1 month

Wheels For ...

by Sven R. Kunze

Hi folks, currently, I came across http://pythonwheels.com/ during researching how to make a proper Python distribution for PyPI. I thought it would be great idea to tell other maintainers to upload their content as wheels so I approached a couple of them. Some of them already provided wheels. Happy being able to have built my own distribution, I discussed the issue at hand with some people and I would like to share my findings and propose some ideas: 1) documentation is weirdly split up/distributed and references old material 2) once up and running (setup.cfg, setup.py etc. etc.) it works but everybody needs to do it on their own 3) more than one way to do (upload, wheel, source/binary etc.) it (sigh) 4) making contact to propose wheels on github or per email is easy otherwise almost impossible or very tedious 5) reactions went evenly split from "none", "yes", "when ready" to "nope" None: well, okay yes: that's good when ready: well, okay nope: what a pity for wheels; example: https://github.com/simplejson/simplejson/issues/122 I personally find the situation not satisfying. Someone proposes the following solution in form of a question: Why do developers need to build their distribution themselves? I had not real answer to him, but pondering a while over it, I found it really insightful. Viewing this from a different angle, packaging your own distribution is actually a waste of time. It is a tedious, error-prone task involving no creativity whatsoever. Developers on the other hand are actually people with very little time and a lot of creativity at hand which should spend better. The logical conclusion would be that PyPI should build wheels for the developers for every python/platform combination necessary. With this post, I would like raise awareness of the people in charge of the Python infrastructure. Best, Sven

6 years, 1 month

Re: [Python-ideas] Should our default random number generator be secure?

by Tim Peters

[Tim, on parallel PRNGs] >> There are some clean and easy approaches to this based on >> crypto-inspired schemes, but giving up crypto strength for speed. If >> you haven't read it, this paper is delightful: >> >> http://www.thesalmons.org/john/random123/papers/random123sc11.pdf [Nathaniel Smith] > It really is! As AES acceleration instructions become more common > (they're now standard IIUC on x86, x86-64, and even recent ARM?), even > just using AES in CTR mode becomes pretty compelling -- it's fast, > deterministic, provably equidistributed, *and* cryptographically > secure enough for many purposes. Excellent - we're going to have a hard time finding something real to disagree about :-) > (Compared to a true state-of-the-art CPRNG the naive version fails due > to lack of incremental mixing, and the use of a reversible transition > function. But even these are mostly only important to protect against > attackers who have access to your memory -- which is not trivial as > heartbleed shows, but still, it's *waaay* ahead of something like MT > on basically every important axis.) Except for wide adoption. Most people I bump into never even heard of this kind of approach. Nobody ever got fired for buying IBM, and nobody ever got fired for recommending MT - it's darned near a checklist item when shopping for a language. I may have to sneak the code in while you distract Guido with a provocative rant about the inherent perfidy of the Dutch character ;-)

6 years, 1 month

Re: [Python-ideas] Should our default random number generator be secure?

by Guido van Rossum

---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 10:42 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org been speaking to a significant go person. confirmed. it takes data out of that buffer, and does not zero it behind itself. obviously for performance reasons. same type of thing happens with MT-style engines. in practice, they can be would backwards. a proper stream cipher cannot be turned backwards. however, that's just an academic observation. or maybe it indicates that well-financed groups can get it wrong too. by the way, chacha arc4random can create random values faster than a memcpy -- the computation of fresh output is faster than doing gross-cost of "read" from memory (when cache dirtying is accounted for). -- --Guido van Rossum (python.org/~guido)

6 years, 1 month

Re: [Python-ideas] Should our default random number generator be secure?

by Guido van Rossum

---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 10:36 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org > Yet another thing. Where do you see that Go and Swift have secure random as > a keyword? Searching for "golang random" gives the math/rand package as the > first hit, which has a note reminding the reader to use crypto/rand for > security work. yes, well, look at the other phrase it uses... that produces a deterministic sequence of values each time a program is run it documents itself as being decidely non-random. that documentation change happened soon after this event: https://lwn.net/Articles/625506/ these days, the one people are using is found using "go secure random" https://golang.org/pkg/crypto/rand/ that opens /dev/urandom or uses the getrandom system call depending on system. it also has support for the windows entropy API. it pulls data into a large buffer, a cache. then each subsequent call, it consumes some, until it rus out, and has to do a fresh read. it appears to not clean the buffer behind itself, probably for performance reasons, so the memory is left active. (forward secrecy violated) i don't think they are doing the best they can... i think they should get forward secrecy and higher performance by having an in-process chacha. but you can sense the trend. here's an example of the fallout.. https://github.com/golang/go/issues/9205 > For Swift it's much the same -- there's an arc4random() in > the Darwin package but nothing in the core language. that is what people are led to use. -- --Guido van Rossum (python.org/~guido)

6 years, 1 month

Re: [Python-ideas] Should our default random number generator be secure?

by Guido van Rossum

I'm just going to forward further missives by Theo. ---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 9:59 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org > Thanks. And one last thing: unless Go and Swift, Python has no significant > corporate resources behind it -- it's mostly volunteers begging their > employers to let them work on Python for a few hours per week. i understand because I find myself in the same situation. however i think you overstate the difficulty involved. high-availibility random is kind of a new issue. so final advice from me; feel free to forward as you like. i think arc4random would be a better API to call on the back side than getentropy/getrandom. arc4random can seed initialize with a single getentropy/getrandom call at startup. that is done automatically. you can then use arc4random's results to initialize the MT. in a system call trace, this will show up as one getentropy/getrandom at process startup, which gets both subsystems going. really cheap. in the case of longer run times, the userland arc4random PRNG folding reduces the system calls required. this helps older kernels with slower entropy creation, taking pressure off their subsystem. driving everyone towards this one API which is so high performance is the right goal. chacha arc4random is really fast. if you were to create such an API in python, maybe this is how it will go: say it becomes arc4random in the back end. i am unsure what advice to give you regarding a python API name. in swift, they chose to use the same prefix "arc4random" (id = arc4random(), id = arc4random_uniform(1..n)"; it is a little bit different than the C API. google has tended to choose other prefixes. we admit the name is a bit strange, but we can't touch the previous attempts like drand48.... I do suggest you have the _uniform and _buf versions. Maybe apple chose to stick to arc4random as a name simply because search engines tend to give above average advice for this search string? so arc4random is natively available in freebsd, macos, solaris, and other systems like andriod libc (bionic). some systems lack it: win32, glibc, hpux, aix, so we wrote replacements for libressl: https://github.com/libressl-portable/openbsd/tree/master/src/lib/libcrypto/… https://github.com/libressl-portable/portable/tree/master/crypto/compat the first is the base openbsd tree where we maintain/develop this code for other systems, the 2nd part is scaffold in libressl that makes this available to others. it contains arc4random for those systems, and supplies getentropy() stubs for challenged systems. we'll admit we haven't got solutions for every system known to man. we are trying to handle fork issues, and systems with very bad entropy feeding. that's free code. the heavy lifting is done, and we'll keep maintaining that until the end of days. i hope it helps. -- --Guido van Rossum (python.org/~guido)

6 years, 1 month

NuGet/Chocolatey feed for releases

by Alexander Walters

It would be incredibly convenient, especially for users of AppVayor's continuous integration service, if there were a(n official) repository for chocolatey containing recent releases of python. The official Chocolatey gallery contains installers for the latest 2.7 and 3.4 (as of this post). What I am proposing would contain the most commonly used pythons in testing (2.6 2.7 3.3 3.4 and future releases). I am perfectly willing to set up a repo for my own use, but am posting this to see if there is community support...or psf support... for setting up an official repo.

6 years, 1 month

Re: [Python-ideas] Wheels For ...

by Laura Creighton

In a message of Sun, 06 Sep 2015 15:31:16 -0400, Terry Reedy writes: >On 9/6/2015 1:33 PM, Sven R. Kunze wrote: >> >> With this post, I would like raise awareness of the people in charge of >> the Python infrastructure. > >pypa is in charge of packaging. https://github.com/pypa >I believe the google groups link is their discussion forum. They have one -- https://groups.google.com/forum/#!forum/pypa-dev but you can also get them at the disutils mailing list. https://mail.python.org/mailman/listinfo/distutils-sig I think, rather than discussion, it is 'people willing to write code' that they are short of ... Laura

6 years, 1 month

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas September 2015