Python-ideas September 2015

python-ideas@python.org

107 participants
54 discussions

by Stephan Sahm

Dear all I found a BUG in the standard while statement, which appears both in python 2.7 and python 3.4 on my system. It usually won't appear because I only stumbled upon it after trying to implement a nice repeat structure. Look: ``` class repeat(object): def __init__(self, n): self.n = n def __bool__(self): self.n -= 1 return self.n >= 0 __nonzero__=__bool__ a = repeat(2) ``` the meaning of the above is that bool(a) returns True 2-times, and after that always False. Now executing ``` while a: print('foo') ``` will in fact print 'foo' two times. HOWEVER ;-) .... ``` while repeat(2): print('foo') ``` will go on and go on, printing 'foo' until I kill it. Please comment, explain or recommend this further if you also think that both while statements should behave identically. hoping for responses, best, Stephan

4 years, 10 months

Desperate need for enhanced print function

by Anand Krishnakumar

Hi! This is my first time I'm sending an email to the python-ideas mailing list. I've got an enhancement idea for the built-in print function and I hope it is as good as I think it is. Imagine you have a trial.py file like this: a = 4 b = "Anand" print("Hello, I am " + b + ". My favorite number is " + str(a) + ".") OR print("Hello, I am ", b, ". My favorite number is ", a, ".") Well I've got an idea for a function named "print_easy" (The only valid name I could come up with right now). So print_easy will be a function which will be used like this (instead of the current print statement in trial.py) : print_easy("Hello, I am", b, ". My favorite number is", a ".") Which gives out: Hello, I am Anand. My favorite number is 4 The work it does is that it casts the variables and it also formats the sentences it is provided with. It is exclusively for beginners. I'm 14 and I came up with this idea after seeing my fellow classmates at school struggling to do something like this with the standard print statement. Sure, you can use the format method but won't that be a bit too much for beginners? (Also, casting is inevitable in every programmer's career) Please let me know how this sounds. If it gains some traction, I'll work on it a bit more and clearly list out the features. Thanks, Anand. -- Anand.

4 years, 10 months

Wheels For ...

by Sven R. Kunze

Hi folks, currently, I came across http://pythonwheels.com/ during researching how to make a proper Python distribution for PyPI. I thought it would be great idea to tell other maintainers to upload their content as wheels so I approached a couple of them. Some of them already provided wheels. Happy being able to have built my own distribution, I discussed the issue at hand with some people and I would like to share my findings and propose some ideas: 1) documentation is weirdly split up/distributed and references old material 2) once up and running (setup.cfg, setup.py etc. etc.) it works but everybody needs to do it on their own 3) more than one way to do (upload, wheel, source/binary etc.) it (sigh) 4) making contact to propose wheels on github or per email is easy otherwise almost impossible or very tedious 5) reactions went evenly split from "none", "yes", "when ready" to "nope" None: well, okay yes: that's good when ready: well, okay nope: what a pity for wheels; example: https://github.com/simplejson/simplejson/issues/122 I personally find the situation not satisfying. Someone proposes the following solution in form of a question: Why do developers need to build their distribution themselves? I had not real answer to him, but pondering a while over it, I found it really insightful. Viewing this from a different angle, packaging your own distribution is actually a waste of time. It is a tedious, error-prone task involving no creativity whatsoever. Developers on the other hand are actually people with very little time and a lot of creativity at hand which should spend better. The logical conclusion would be that PyPI should build wheels for the developers for every python/platform combination necessary. With this post, I would like raise awareness of the people in charge of the Python infrastructure. Best, Sven

4 years, 10 months

Re: [Python-ideas] Should our default random number generator be secure?

by Tim Peters

[Tim, on parallel PRNGs] >> There are some clean and easy approaches to this based on >> crypto-inspired schemes, but giving up crypto strength for speed. If >> you haven't read it, this paper is delightful: >> >> http://www.thesalmons.org/john/random123/papers/random123sc11.pdf [Nathaniel Smith] > It really is! As AES acceleration instructions become more common > (they're now standard IIUC on x86, x86-64, and even recent ARM?), even > just using AES in CTR mode becomes pretty compelling -- it's fast, > deterministic, provably equidistributed, *and* cryptographically > secure enough for many purposes. Excellent - we're going to have a hard time finding something real to disagree about :-) > (Compared to a true state-of-the-art CPRNG the naive version fails due > to lack of incremental mixing, and the use of a reversible transition > function. But even these are mostly only important to protect against > attackers who have access to your memory -- which is not trivial as > heartbleed shows, but still, it's *waaay* ahead of something like MT > on basically every important axis.) Except for wide adoption. Most people I bump into never even heard of this kind of approach. Nobody ever got fired for buying IBM, and nobody ever got fired for recommending MT - it's darned near a checklist item when shopping for a language. I may have to sneak the code in while you distract Guido with a provocative rant about the inherent perfidy of the Dutch character ;-)

4 years, 10 months

Re: [Python-ideas] Should our default random number generator be secure?

by Guido van Rossum

---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 10:42 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org been speaking to a significant go person. confirmed. it takes data out of that buffer, and does not zero it behind itself. obviously for performance reasons. same type of thing happens with MT-style engines. in practice, they can be would backwards. a proper stream cipher cannot be turned backwards. however, that's just an academic observation. or maybe it indicates that well-financed groups can get it wrong too. by the way, chacha arc4random can create random values faster than a memcpy -- the computation of fresh output is faster than doing gross-cost of "read" from memory (when cache dirtying is accounted for). -- --Guido van Rossum (python.org/~guido)

4 years, 10 months

Re: [Python-ideas] Should our default random number generator be secure?

by Guido van Rossum

---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 10:36 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org > Yet another thing. Where do you see that Go and Swift have secure random as > a keyword? Searching for "golang random" gives the math/rand package as the > first hit, which has a note reminding the reader to use crypto/rand for > security work. yes, well, look at the other phrase it uses... that produces a deterministic sequence of values each time a program is run it documents itself as being decidely non-random. that documentation change happened soon after this event: https://lwn.net/Articles/625506/ these days, the one people are using is found using "go secure random" https://golang.org/pkg/crypto/rand/ that opens /dev/urandom or uses the getrandom system call depending on system. it also has support for the windows entropy API. it pulls data into a large buffer, a cache. then each subsequent call, it consumes some, until it rus out, and has to do a fresh read. it appears to not clean the buffer behind itself, probably for performance reasons, so the memory is left active. (forward secrecy violated) i don't think they are doing the best they can... i think they should get forward secrecy and higher performance by having an in-process chacha. but you can sense the trend. here's an example of the fallout.. https://github.com/golang/go/issues/9205 > For Swift it's much the same -- there's an arc4random() in > the Darwin package but nothing in the core language. that is what people are led to use. -- --Guido van Rossum (python.org/~guido)

4 years, 10 months

Re: [Python-ideas] Should our default random number generator be secure?

by Guido van Rossum

I'm just going to forward further missives by Theo. ---------- Forwarded message ---------- From: Theo de Raadt Date: Wed, Sep 9, 2015 at 9:59 AM Subject: Re: getentropy, getrandom, arc4random() To: guido(a)python.org > Thanks. And one last thing: unless Go and Swift, Python has no significant > corporate resources behind it -- it's mostly volunteers begging their > employers to let them work on Python for a few hours per week. i understand because I find myself in the same situation. however i think you overstate the difficulty involved. high-availibility random is kind of a new issue. so final advice from me; feel free to forward as you like. i think arc4random would be a better API to call on the back side than getentropy/getrandom. arc4random can seed initialize with a single getentropy/getrandom call at startup. that is done automatically. you can then use arc4random's results to initialize the MT. in a system call trace, this will show up as one getentropy/getrandom at process startup, which gets both subsystems going. really cheap. in the case of longer run times, the userland arc4random PRNG folding reduces the system calls required. this helps older kernels with slower entropy creation, taking pressure off their subsystem. driving everyone towards this one API which is so high performance is the right goal. chacha arc4random is really fast. if you were to create such an API in python, maybe this is how it will go: say it becomes arc4random in the back end. i am unsure what advice to give you regarding a python API name. in swift, they chose to use the same prefix "arc4random" (id = arc4random(), id = arc4random_uniform(1..n)"; it is a little bit different than the C API. google has tended to choose other prefixes. we admit the name is a bit strange, but we can't touch the previous attempts like drand48.... I do suggest you have the _uniform and _buf versions. Maybe apple chose to stick to arc4random as a name simply because search engines tend to give above average advice for this search string? so arc4random is natively available in freebsd, macos, solaris, and other systems like andriod libc (bionic). some systems lack it: win32, glibc, hpux, aix, so we wrote replacements for libressl: https://github.com/libressl-portable/openbsd/tree/master/src/lib/libcrypto/… https://github.com/libressl-portable/portable/tree/master/crypto/compat the first is the base openbsd tree where we maintain/develop this code for other systems, the 2nd part is scaffold in libressl that makes this available to others. it contains arc4random for those systems, and supplies getentropy() stubs for challenged systems. we'll admit we haven't got solutions for every system known to man. we are trying to handle fork issues, and systems with very bad entropy feeding. that's free code. the heavy lifting is done, and we'll keep maintaining that until the end of days. i hope it helps. -- --Guido van Rossum (python.org/~guido)

4 years, 10 months

NuGet/Chocolatey feed for releases

by Alexander Walters

It would be incredibly convenient, especially for users of AppVayor's continuous integration service, if there were a(n official) repository for chocolatey containing recent releases of python. The official Chocolatey gallery contains installers for the latest 2.7 and 3.4 (as of this post). What I am proposing would contain the most commonly used pythons in testing (2.6 2.7 3.3 3.4 and future releases). I am perfectly willing to set up a repo for my own use, but am posting this to see if there is community support...or psf support... for setting up an official repo.

4 years, 10 months

Re: [Python-ideas] Wheels For ...

by Laura Creighton

In a message of Sun, 06 Sep 2015 15:31:16 -0400, Terry Reedy writes: >On 9/6/2015 1:33 PM, Sven R. Kunze wrote: >> >> With this post, I would like raise awareness of the people in charge of >> the Python infrastructure. > >pypa is in charge of packaging. https://github.com/pypa >I believe the google groups link is their discussion forum. They have one -- https://groups.google.com/forum/#!forum/pypa-dev but you can also get them at the disutils mailing list. https://mail.python.org/mailman/listinfo/distutils-sig I think, rather than discussion, it is 'people willing to write code' that they are short of ... Laura

4 years, 10 months

PyParallel update

by Trent Nelson

[CC'ing python-dev@ for those that are curious; please drop and keep follow-up discussion to python-ideas@] Hi folks, I've made a lot of progress on PyParallel since the PyCon dev summit (https://speakerdeck.com/trent/pyparallel-pycon-2015-language-summit); I fixed the outstanding breakage with generators, exceptions and whatnot. I got the "instantaneous Wiki search server" working[1] and implemented the entire TechEmpower Frameworks Benchmark Suite[2], including a PyParallel-friendly pyodbc module, allowing database connections and querying in parallel. [1]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki… [2]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/tefb… I set up a landing page for the project: http://pyparallel.org And there was some good discussion on reddit earlier this week: https://www.reddit.com/r/programming/comments/3jhv80/pyparallel_an_experime… I've put together some documentation on the project, its aims, and the key parts of the solution regarding the parallelism through simple client/server paradigms. This documentation is available directly on the github landing page for the project: https://github.com/pyparallel/pyparallel Writing that documentation forced me to formalize (or at least commit) to the restrictions/trade-offs that PyParallel would introduce, and I'm pretty happy I was basically able to boil it down into a single rule: Don't persist parallel objects. That keeps the mental model very simple. You don't need to worry about locking or ownership or races or anything like that. Just don't persist parallel objects, that's the only thing you have to remember. It's actually really easy to convert existing C code or Python code into something that is suitable for calling from within a parallel callback by just ensuring that rule isn't violated. It took about four hours to figure out how NumPy allocated stuff and add in the necessary PyParallel-aware tweaks, and not that much longer for pyodbc. (Most stuff "just works", though.) (The ABI changes would mean this is a Python 4.x type of thing; there are fancy ways we could avoid ABI changes and get this working on Python 3.x, but, eh, I like the 4.x target. It's realistic.) The other thing that clicked is that asyncio and PyParallel would actually work really well together for exploiting client-driven parallelism (PyParallel really is only suited to server-oriented parallelism at the moment, i.e. serving HTTP requests in parallel). With asyncio, though, you could keep the main-thread/single-thread client-drives-computation paradigm, but have it actually dispatch work to parallel.server() objects behind the scenes. For example, in order to process all files in a directory in parallel, asyncio would request a directory listing (i.e. issue a GET /) which the PyParallel HTTP server would return, it would then create non-blocking client connections to the same server and invoke whatever HTTP method is desired to do the file processing. You can either choose to write the new results from within the parallel context (which could then be accessed as normal files via HTTP), or you could have PyParallel return json/bytes, which could then be aggregated by asyncio. Everything is within the same process, so you get all the benefits that provides (free access to anything within scope, like large data structures, from within parallel contexts). You can synchronously call back into the main thread from a parallel thread, too, if you wanted to update a complex data structure directly. The other interesting thing that documentation highlights is the advantage of the split brain "main thread vs parallel thread" GC and non-GC allocators. I'm not sure if I've ever extolled the virtue of such an approach on paper or in e-mail. It's pretty neat though and allows us to avoid a whole raft of problems that need to be solved when you have a single GC/memory model. Next steps: once 3.5 is tagged, I'm going to bite the bullet and rebase. That'll require a bit of churn, so if there's enough interest from others, I figured we'd use the opportunity to at least get it building again on POSIX (Linux/OSX/FreeBSD). From there people can start implementing the missing bits for implementing the parallel machinery behind the scenes. The parallel interpreter thread changes I made are platform agnostic, the implementation just happens to be on Windows at the moment; don't let the Windows-only thing detract from what's actually being pitched: a (working, demonstrably-performant) solution to "Python's GIL problem". Regards, Trent.

4 years, 10 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas September 2015