[Security-sig] Take a decision for os.urandom() in Python 3.6

Fri Aug 5 22:39:35 EDT 2016

Thanks, Victor!

I've (mostly) read both PEPs and your summary.

It seems there are several risks that need to be weighed.

1. An important secret is generated using insufficient entropy.
2. An app blocks unnecessarily.
3. An app crashes unexpectedly.
4. Bad code gets cargo-culted (e.g. through StackOverflow).

Both PEPs prevent bad secrets (1), which the status quo (in theory)
could lead to this problem.

PEP 524 (make os.urandom() block) can cause blocking (2), but prevents
crashes (3).

PEP 522 (make it raise) can cause  crashes (3) but prevents blocking (2).

Re (4): With PEP 524, people worried about blocking may be driven to
unnecessarily write more complicated code using os.getrandom(). With
PEP 522, people worried about crashes may be driven to unnecessarily
call secrets.wait_for_system_rng() or put try/except blocks catching
raise BlockingIOError around all their os.urandom()-based calls.

There's some doubt that (2) or (3) will actually ever happen, because
you'd have to be really early in the startup process. On several
platforms it is known to be impossible (e.g. on Mac you run Python
before the kernel has enough entropy), on other platforms there is no
way to avoid the blocking (e.g. Windows), and the proposals really
only differ on Linux (and a few other systems, like Solaris, that have
getrandom()).

My own biggest worry is about (4), cargo-culting -- undoubtedly people
will worry about (2) or (3) and they will pass on "robust" code that
is unnecessarily complicated and risks being wrong (since the failure
mode is *very* hard to reproduce for testing).

The cargo-cult code shown in PEP 524
(https://www.python.org/dev/peps/pep-0524/#best-effort-rng) is much
worse than that recommended by PEP 522 (secrets.wait_for_system_rng()
-- though I could also imagine people wrapping os.urandom() in
try/except).

But for me personally, it is much easier to stop worrying about a tiny
chance of blocking (2) than it would be to believe that the chance of
crashes (3) is truly so small that I don't have to do anything about
it.

Especially since on non-Linux platforms there is a chance of blocking
anyway, with no way to prevent it. It's easy to stop worrying about
something you can't control. (I don't really worry about that big
asteroid that's going to hit the earth in the next millennium either.
:-)

So I'm in favor of PEP 524. I think I would be even more in favor of
it if it didn't add os.getrandom(), since then the whole possibility
of cargo-culting unnecessary countermeasures would be pretty much gone
(though for the die-hards there's always ctypes...). But I also agree
with the idea of exposing the platform's primitive operations when
they exist.

--Guido

On Fri, Aug 5, 2016 at 4:13 PM, Victor Stinner <victor.stinner at gmail.com> wrote:
> Le 5 août 2016 9:42 PM, "Ethan Furman" <ethan at stoneleaf.us> a écrit :
>> Can someone write a brief summary of the differences between the two?
>
> Oh, it's hard to summarize. Let me try. As you may expect, my summary is far
> from being fair :-D
>
> --
>
> The two PEPs propose a very different behaviour when os.urandom() would
> block: raise an exception (522) or wait (524).
>
> The PEP 522 forces developers to explicitly handle a rare case (when urandom
> blocks).
>
> The PEP 524 proposes to be optimistic and hope that if urandom hangs, it
> doesn't hang too long.
>
> The corner case of the corner case is when urandom blocks really too long
> (longer than 60 seconds, or simply forever).
>
> The PEP 524 doesn't handle it (block). The PEP 522 makes the exceptional
> corner case as important as the common case (urandom just blocks a few
> seconds, or don't block at all).
>
> --
>
> Both PEPs want to make Python more secure: don't fall back on the "weak"
> /dev/urandom (define weak: XXX) in os.urandom() before system urandom is
> initialized.
>
> Most differences between the two PEPs only impact applications calling
> os.urandom() very early during system initialization (before system urandom
> initialization) on a system with very slow entropy source or just no entropy
> (VM, embeded device, ...).
>
> The PEP 522 proposes to raise an exception on such case. It forces
> developers to modify their code to decide how to handle such corner case:
> wait a few seconds, switch to a weaker entropy source (and maybe log a
> waning/error), etc.
>
> The PEP 524 (mine) proposes to block. Applications don't need to be
> modified. The expectation is that the kernel will be able to get enough
> entropy fast enough. By the way, blocking on system urandom is not something
> new, SSH has the same behaviour for example (try SSH on such VM with no
> entropy...).
>
> --
>
> The PEP 524 proposes also to add a new function os.getrandom() for people
> who understand low level stuff and security and want to enhance their
> application on the low entropy case. It allows to reimplement the PEP 522 on
> Linux in a few lines of pure Python, so give control when urandom would
> block (no black magic, just call os.getrandom(os.GRND_NOBLOCK) which raises
> BlockingIOError).
>
> The PEP 522 proposes a new function to wait for system urandom
> initialization. Something similar to the PEP 524 but it requires to modify
> all applications to use it (to get PEP 524 behaviour).
>
> --
>
> IMO PEP 524 has a lower impact on backward compatibility and is easier to
> implement.
>
> The risk of the PEP 524 is that developers start to expect that os.urandom()
> will *never* block which simply cannot be implemented on all platforms. Both
> PEP are specific to Linux (even if Solaris will benefit of the same
> enhancement), but even just on Linux os.urandom() can still block (don't
> raise the expected BlockingIOError) on Linux older than 3.17.
>
> Victor
>
>
> _______________________________________________
> Security-SIG mailing list
> Security-SIG at python.org
> https://mail.python.org/mailman/listinfo/security-sig
>

-- 
--Guido van Rossum (python.org/~guido)