Hi!
This is my first time I'm sending an email to the python-ideas mailing
list. I've got an enhancement idea for the built-in print function and I
hope it is as good as I think it is.
Imagine you have a trial.py file like this:
a = 4
b = "Anand"
print("Hello, I am " + b + ". My favorite number is " + str(a) + ".")
OR
print("Hello, I am ", b, ". My favorite number is ", a, ".")
Well I've got an idea for a function named "print_easy" (The only valid
name I could come up with right now).
So print_easy will be a function which will be used like this (instead of
the current print statement in trial.py) :
print_easy("Hello, I am", b, ". My favorite number is", a ".")
Which gives out:
Hello, I am Anand. My favorite number is 4
The work it does is that it casts the variables and it also formats the
sentences it is provided with. It is exclusively for beginners.
I'm 14 and I came up with this idea after seeing my fellow classmates at
school struggling to do something like this with the standard print
statement.
Sure, you can use the format method but won't that be a bit too much for
beginners? (Also, casting is inevitable in every programmer's career)
Please let me know how this sounds. If it gains some traction, I'll work on
it a bit more and clearly list out the features.
Thanks,
Anand.
--
Anand.
Hi folks,
currently, I came across http://pythonwheels.com/ during researching how
to make a proper Python distribution for PyPI. I thought it would be
great idea to tell other maintainers to upload their content as wheels
so I approached a couple of them. Some of them already provided wheels.
Happy being able to have built my own distribution, I discussed the
issue at hand with some people and I would like to share my findings and
propose some ideas:
1) documentation is weirdly split up/distributed and references old material
2) once up and running (setup.cfg, setup.py etc. etc.) it works but
everybody needs to do it on their own
3) more than one way to do (upload, wheel, source/binary etc.) it (sigh)
4) making contact to propose wheels on github or per email is easy
otherwise almost impossible or very tedious
5) reactions went evenly split from "none", "yes", "when ready" to "nope"
None: well, okay
yes: that's good
when ready: well, okay
nope: what a pity for wheels; example:
https://github.com/simplejson/simplejson/issues/122
I personally find the situation not satisfying. Someone proposes the
following solution in form of a question:
Why do developers need to build their distribution themselves?
I had not real answer to him, but pondering a while over it, I found it
really insightful. Viewing this from a different angle, packaging your
own distribution is actually a waste of time. It is a tedious,
error-prone task involving no creativity whatsoever. Developers on the
other hand are actually people with very little time and a lot of
creativity at hand which should spend better. The logical conclusion
would be that PyPI should build wheels for the developers for every
python/platform combination necessary.
With this post, I would like raise awareness of the people in charge of
the Python infrastructure.
Best,
Sven
[Tim, on parallel PRNGs]
>> There are some clean and easy approaches to this based on
>> crypto-inspired schemes, but giving up crypto strength for speed. If
>> you haven't read it, this paper is delightful:
>>
>> http://www.thesalmons.org/john/random123/papers/random123sc11.pdf
[Nathaniel Smith]
> It really is! As AES acceleration instructions become more common
> (they're now standard IIUC on x86, x86-64, and even recent ARM?), even
> just using AES in CTR mode becomes pretty compelling -- it's fast,
> deterministic, provably equidistributed, *and* cryptographically
> secure enough for many purposes.
Excellent - we're going to have a hard time finding something real to
disagree about :-)
> (Compared to a true state-of-the-art CPRNG the naive version fails due
> to lack of incremental mixing, and the use of a reversible transition
> function. But even these are mostly only important to protect against
> attackers who have access to your memory -- which is not trivial as
> heartbleed shows, but still, it's *waaay* ahead of something like MT
> on basically every important axis.)
Except for wide adoption. Most people I bump into never even heard of
this kind of approach. Nobody ever got fired for buying IBM, and
nobody ever got fired for recommending MT - it's darned near a
checklist item when shopping for a language. I may have to sneak the
code in while you distract Guido with a provocative rant about the
inherent perfidy of the Dutch character ;-)
---------- Forwarded message ----------
From: Theo de Raadt
Date: Wed, Sep 9, 2015 at 10:42 AM
Subject: Re: getentropy, getrandom, arc4random()
To: guido(a)python.org
been speaking to a significant go person.
confirmed.
it takes data out of that buffer, and does not zero it behind itself.
obviously for performance reasons.
same type of thing happens with MT-style engines. in practice, they
can be would backwards. a proper stream cipher cannot be turned
backwards.
however, that's just an academic observation. or maybe it indicates
that well-financed groups can get it wrong too.
by the way, chacha arc4random can create random values faster than a
memcpy -- the computation of fresh output is faster than doing
gross-cost of "read" from memory (when cache dirtying is accounted for).
--
--Guido van Rossum (python.org/~guido)
---------- Forwarded message ----------
From: Theo de Raadt
Date: Wed, Sep 9, 2015 at 10:36 AM
Subject: Re: getentropy, getrandom, arc4random()
To: guido(a)python.org
> Yet another thing. Where do you see that Go and Swift have secure random
as
> a keyword? Searching for "golang random" gives the math/rand package as
the
> first hit, which has a note reminding the reader to use crypto/rand for
> security work.
yes, well, look at the other phrase it uses...
that produces a deterministic sequence of values each time a program is
run
it documents itself as being decidely non-random. that documentation
change happened soon after this event:
https://lwn.net/Articles/625506/
these days, the one people are using is found using "go secure random"
https://golang.org/pkg/crypto/rand/
that opens /dev/urandom or uses the getrandom system call depending on
system. it also has support for the windows entropy API. it pulls
data into a large buffer, a cache. then each subsequent call, it
consumes some, until it rus out, and has to do a fresh read. it
appears to not clean the buffer behind itself, probably for
performance reasons, so the memory is left active. (forward secrecy
violated)
i don't think they are doing the best they can... i think they should
get forward secrecy and higher performance by having an in-process
chacha. but you can sense the trend.
here's an example of the fallout..
https://github.com/golang/go/issues/9205
> For Swift it's much the same -- there's an arc4random() in
> the Darwin package but nothing in the core language.
that is what people are led to use.
--
--Guido van Rossum (python.org/~guido)
I'm just going to forward further missives by Theo.
---------- Forwarded message ----------
From: Theo de Raadt
Date: Wed, Sep 9, 2015 at 9:59 AM
Subject: Re: getentropy, getrandom, arc4random()
To: guido(a)python.org
> Thanks. And one last thing: unless Go and Swift, Python has no significant
> corporate resources behind it -- it's mostly volunteers begging their
> employers to let them work on Python for a few hours per week.
i understand because I find myself in the same situation.
however i think you overstate the difficulty involved.
high-availibility random is kind of a new issue.
so final advice from me; feel free to forward as you like.
i think arc4random would be a better API to call on the back side than
getentropy/getrandom.
arc4random can seed initialize with a single getentropy/getrandom call
at startup. that is done automatically. you can then use
arc4random's results to initialize the MT. in a system call trace,
this will show up as one getentropy/getrandom at process startup,
which gets both subsystems going. really cheap.
in the case of longer run times, the userland arc4random PRNG folding
reduces the system calls required. this helps older kernels with
slower entropy creation, taking pressure off their subsystem. driving
everyone towards this one API which is so high performance is the
right goal.
chacha arc4random is really fast.
if you were to create such an API in python, maybe this is how it will
go:
say it becomes arc4random in the back end. i am unsure what advice to
give you regarding a python API name. in swift, they chose to use the
same prefix "arc4random" (id = arc4random(), id = arc4random_uniform(1..n)";
it is a little bit different than the C API. google has tended to choose
other prefixes. we admit the name is a bit strange, but we can't touch
the previous attempts like drand48....
I do suggest you have the _uniform and _buf versions. Maybe apple
chose to stick to arc4random as a name simply because search engines
tend to give above average advice for this search string?
so arc4random is natively available in freebsd, macos, solaris, and
other systems like andriod libc (bionic). some systems lack it:
win32, glibc, hpux, aix, so we wrote replacements for libressl:
https://github.com/libressl-portable/openbsd/tree/master/src/lib/libcrypt...https://github.com/libressl-portable/portable/tree/master/crypto/compat
the first is the base openbsd tree where we maintain/develop this code
for other systems, the 2nd part is scaffold in libressl that makes this
available to others.
it contains arc4random for those systems, and supplies getentropy()
stubs for challenged systems.
we'll admit we haven't got solutions for every system known to man.
we are trying to handle fork issues, and systems with very bad
entropy feeding.
that's free code. the heavy lifting is done, and we'll keep maintaining
that until the end of days. i hope it helps.
--
--Guido van Rossum (python.org/~guido)
It would be incredibly convenient, especially for users of AppVayor's
continuous integration service, if there were a(n official) repository
for chocolatey containing recent releases of python. The official
Chocolatey gallery contains installers for the latest 2.7 and 3.4 (as of
this post). What I am proposing would contain the most commonly used
pythons in testing (2.6 2.7 3.3 3.4 and future releases).
I am perfectly willing to set up a repo for my own use, but am posting
this to see if there is community support...or psf support... for
setting up an official repo.
In a message of Sun, 06 Sep 2015 15:31:16 -0400, Terry Reedy writes:
>On 9/6/2015 1:33 PM, Sven R. Kunze wrote:
>>
>> With this post, I would like raise awareness of the people in charge of
>> the Python infrastructure.
>
>pypa is in charge of packaging. https://github.com/pypa
>I believe the google groups link is their discussion forum.
They have one -- https://groups.google.com/forum/#!forum/pypa-dev
but you can also get them at the disutils mailing list.
https://mail.python.org/mailman/listinfo/distutils-sig
I think, rather than discussion, it is 'people willing to write code'
that they are short of ...
Laura
[CC'ing python-dev@ for those that are curious; please drop and keep
follow-up discussion to python-ideas@]
Hi folks,
I've made a lot of progress on PyParallel since the PyCon dev summit
(https://speakerdeck.com/trent/pyparallel-pycon-2015-language-summit); I
fixed the outstanding breakage with generators, exceptions and whatnot.
I got the "instantaneous Wiki search server" working[1] and implemented
the entire TechEmpower Frameworks Benchmark Suite[2], including a
PyParallel-friendly pyodbc module, allowing database connections and
querying in parallel.
[1]:
https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wi...
[2]:
https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/te...
I set up a landing page for the project:
http://pyparallel.org
And there was some good discussion on reddit earlier this week:
https://www.reddit.com/r/programming/comments/3jhv80/pyparallel_an_experi...
I've put together some documentation on the project, its aims, and
the key parts of the solution regarding the parallelism through
simple client/server paradigms. This documentation is available
directly on the github landing page for the project:
https://github.com/pyparallel/pyparallel
Writing that documentation forced me to formalize (or at least commit)
to the restrictions/trade-offs that PyParallel would introduce, and I'm
pretty happy I was basically able to boil it down into a single rule:
Don't persist parallel objects.
That keeps the mental model very simple. You don't need to worry about
locking or ownership or races or anything like that. Just don't persist
parallel objects, that's the only thing you have to remember.
It's actually really easy to convert existing C code or Python code into
something that is suitable for calling from within a parallel callback by
just ensuring that rule isn't violated. It took about four hours to figure
out how NumPy allocated stuff and add in the necessary PyParallel-aware
tweaks, and not that much longer for pyodbc. (Most stuff "just works",
though.) (The ABI changes would mean this is a Python 4.x type of
thing; there are fancy ways we could avoid ABI changes and get this
working on Python 3.x, but, eh, I like the 4.x target. It's realistic.)
The other thing that clicked is that asyncio and PyParallel would
actually work really well together for exploiting client-driven
parallelism (PyParallel really is only suited to server-oriented
parallelism at the moment, i.e. serving HTTP requests in parallel).
With asyncio, though, you could keep the main-thread/single-thread
client-drives-computation paradigm, but have it actually dispatch work
to parallel.server() objects behind the scenes. For example, in order
to process all files in a directory in parallel, asyncio would request a
directory listing (i.e. issue a GET /) which the PyParallel HTTP server
would return, it would then create non-blocking client connections to
the same server and invoke whatever HTTP method is desired to do the
file processing. You can either choose to write the new results from
within the parallel context (which could then be accessed as normal
files via HTTP), or you could have PyParallel return json/bytes, which
could then be aggregated by asyncio.
Everything is within the same process, so you get all the benefits that
provides (free access to anything within scope, like large data
structures, from within parallel contexts). You can synchronously call
back into the main thread from a parallel thread, too, if you wanted to
update a complex data structure directly.
The other interesting thing that documentation highlights is the
advantage of the split brain "main thread vs parallel thread" GC and
non-GC allocators. I'm not sure if I've ever extolled the virtue of
such an approach on paper or in e-mail. It's pretty neat though and
allows us to avoid a whole raft of problems that need to be solved when
you have a single GC/memory model.
Next steps: once 3.5 is tagged, I'm going to bite the bullet and rebase.
That'll require a bit of churn, so if there's enough interest from
others, I figured we'd use the opportunity to at least get it building
again on POSIX (Linux/OSX/FreeBSD). From there people can start
implementing the missing bits for implementing the parallel machinery
behind the scenes.
The parallel interpreter thread changes I made are platform agnostic,
the implementation just happens to be on Windows at the moment; don't
let the Windows-only thing detract from what's actually being pitched: a
(working, demonstrably-performant) solution to "Python's GIL problem".
Regards,
Trent.
Responses below, but first, another issue:
Things like app data and prefs aren't a single directory. XDG has a notion of a search path rather than a single directory; Windows has a notion of separate search domains; OS X makes things as fun as possible by having both. For writing a new file, you're usually fine just writing to the first path in the default domain (as long as it exists or you can create it), but for reading files you're supposed to look in /etc or All Users or whatever if it's not found there. Most cross-platform wrappers I've used in the past didn't deal with this automatically, and a lot of them didn't even make it easy to do manually.
On Sep 1, 2015, at 14:19, Nathaniel Smith <njs(a)vorpus.org> wrote:
>
> On Sep 1, 2015 02:45, "Andrew Barnert via Python-ideas"
> <python-ideas(a)python.org> wrote:
>>
>>> On Sep 1, 2015, at 01:00, Philipp A. <flying-sheep(a)web.de> wrote:
>>>
>>> When defining a place for config files, cache files, and so on, people usually hack around in a OS-dependent, misinformed, and therefore wrong way.
>>>
>>> Thanks to the tempfile API we at least don’t see people hardcoding /tmp/ too much.
>>>
>>> There is a beautiful little module that does things right and is easy to use: appdirs
>>
>>
>> Is appdirs compatible with the OS X recommendations (as required by the App Store). Apple only gives you cache and app data directories; prefs are supposed to use NSDefaults API or emulate the file names and formats properly, and you have to be sensitive to the sandbox.)
>
> No, AFAICT it doesn't get this right -- it just hard-codes the OS X
> directories.
The biggest problem with most of the cross-platform libraries I've seen is that they assume there is a prefs directory, and on OS X, that really isn't true. If your app explicitly opens the exact same file that NSDefaults would have opened, you're breaking the rules. (Since the mandatory sandbox went into effect, this usually doesn't get you rejected from the App Store anymore, but before that it did.) And taking a quick look at appdirs, it has a user_config_dir that seems to mean exactly that. So, how can a stdlib library handle that?
Meanwhile, it looks like appdirs expects you to give it an app name and company name to construct the paths. What happens if you give it names that don't match the ones in your bundle? You're opening files that belong to another app. Which is again violating Apple's rules.
One more thing: I don't know if it's guaranteed that the right way of doing things on OS X (whether via Cocoa or CoreFoundation) won't spawn a background thread for you. After all, the APIs can talk to the sandbox service and sometimes even the iCloud service. Is that a problem for something in the stdlib?
> It also didn't quite implement the XDG spec correctly
> (there's some fallback behavior you're supposed to do if the magic
> envvars don't make sense that it skips -- very unusual that this will
> matter). And windows I'm not sure about -- the logic in appdirs looked
> reasonable to me when I was reviewing this a few months ago, but there
> seem to be a bunch of semi-contradictory standards and so it's hard to
> know what's even "correct" in the tricky cases.
>
> All of this is probably as much an argument *for* providing the
> functionality as a standard thing as it is against,
> but any PEP here
> probably needs to be thorough about citing the research to show that
> it's actually getting the various platform standards correct.
>
> What makes it particularly difficult is that if you "fix a bug" in a
> library like appdirs, so that it starts suddenly returning different
> results on some computer somewhere, then what it looks like to the end
> user is that their data/settings/whatever have suddenly evaporated and
> whatever disk space was being used for caches never gets cleaned up
> and so forth. Generally when applications change how they compute
> these directories, they also include tricky migration logic to check
> both the old and new names, move stuff over if needed, but I'm not
> sure how a low-level library like this can support that usefully...
And that's especially true in the case of Apple's standards, which also include specific rules about how you're supposed to do such a migration, and doing so requires stuff that can't be done from entirely inside the code.
