<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
A problem has surfaced just this week in 3.5.1. Obviously this is a
good time to fix it for 3.5.2. But there's a big argument over what
is "broken" and what is an appropriate "fix".<br>
<br>
As 3.5 Release Manager, I can put my foot down and make rulings, and
AFAIK the only way to overrule me is with the BDFL. In two of three
cases I've put my foot down. In the third I'm pretty sure I'm
right, but IIUC literally everyone with a stated opinion else
disagrees with me. So I thought it best I escalate it. Note that
3.5.2 is going to wait until the issue is settled and any changes to
behavior are written and checked in.<br>
<br>
(Blanket disclaimer for the below: in some places I'm trying to
communicate other's people positions. I apologize if I
misrepresented yours; please reply and correct my mistake. Also,
sorry for the length of this email. But feel even sorrier for me:
this debate has already eaten two days this week.)<br>
<br>
<br>
BACKGROUND<br>
<br>
For 3.5 os.urandom() was changed: instead of reading from
/dev/urandom, it uses the new system call getrandom() where
available. This is a new system call on Linux (which has already
been cloned by Solaris). getrandom(), as CPython uses it, reads
from the same PRNG that /dev/urandom gets its bits from. But
because it's a system call you don't have to mess around with file
handles. Also it always works in chrooted environments. Sounds
like a fine idea.<br>
<br>
Also for 3.5, several other places where CPython internally needs
random bits were switched from reading from /dev/urandom to calling
getrandom(). The two that I know of: choosing the seed for hash
randomization, and initializing the default Mersenne Twister for the
random module.<br>
<br>
There's one subtle but important difference between /dev/urandom and
getrandom(). At startup, Linux seeds the urandom PRNG from the
entropy pool. If the entropy pool is uninitialized, what happens?
CPython's calls to getrandom() will block until the entropy pool is
initialized, which is usually just a few seconds (or less) after
startup. But /dev/urandom *guarantees* that reads will *always*
work. If the entropy pool hasn't been initialized, it pulls numbers
from the PRNG before it's been properly seeded. What this results
in depends on various aspects of the configuration (do you have ECC
RAM? how long was the machine powered down? does the system have a
correct realtime clock?). In extreme circumstances this may mean
the "random" numbers are shockingly predictable!<br>
<br>
Under normal circumstances this minor difference is irrelevant.
After all, when would the entropy pool ever be uninitialized?<br>
<br>
<br>
THE PROBLEM<br>
<br>
Issue #26839:<br>
<blockquote><a class="moz-txt-link-freetext" href="http://bugs.python.org/issue26839">http://bugs.python.org/issue26839</a><br>
</blockquote>
(warning, the issue is now astonishingly long, and exhausting to
read, and various bits of it are factually wrong)<br>
<br>
A user reports that when starting CPython soon after startup on a
fresh virtual machine, the process would hang for a long time.
Someone on the issue reported observed delays of over 90 seconds.
Later we found out: it wasn't 90 seconds before CPython became
usable, these 90 seconds delays were before systemd timed out and
simply killed the process. It's not clear what the upper bound on
the delay might be.<br>
<br>
The issue author had already identified the cause: CPython was
blocking on getrandom() in order to initialize hash randomization.
On this fresh virtual machine the entropy pool started out
uninitialized. And since the only thing running on the machine was
CPython, and since CPython was blocked on initialization, the
entropy pool was initializing very, very slowly.<br>
<br>
Other posters to the thread pointed out that the same thing would
happen in "import random", if your code could get that far. The
constructor for the Random() object would seed the Mersenne Twister,
which would call getrandom() and block.<br>
<br>
Naturally, callers to os.urandom() could also block for an unbounded
period for the same reason.<br>
<br>
<br>
MY RULINGS SO FAR<br>
<br>
1) The change in 3.5 that means "import random" may block for an
unbounded period of time on Linux due to the switch to getrandom()
must be backed out or amended so that it never blocks.<br>
<br>
I *think* everyone agrees with this. The Mersenne Twister is not a
CPRNG, so seeding it with crypto-quality bits isn't necessary. And
unbounded delays are bad.<br>
<br>
<br>
2) The change in 3.5 that means hash randomization initialization
may block for an unbounded period of time on Linux due to the switch
to getrandom() must be backed out or amended so that it never
blocks.<br>
<br>
I believe most people agree with me. The cryptography experts
disagree. IIUC both Alex Gaynor and Christian Heimes feel the
blocking is preferable to non-random hash "randomization".<br>
<br>
Yes, the bad random data means the hashing will be predictable.
Neither choice is exactly what you want. But most people feel it's
simply unreasonable that in extreme corner cases CPython can block
for an unbounded amount of time before running user code.<br>
<br>
<br>
OS.URANDOM()<br>
<br>
Here's where it gets complicated--and where everyone else thinks I'm
wrong.<br>
<br>
os.urandom() is currently the best place for a Python programmer to
get high-quality random bits. The one-line summary for os.urandom()
reads: "Return a string of n random bytes suitable for cryptographic
use."<br>
<br>
On 3.4 and before, on Linux, os.urandom() would never block, but if
the entropy pool was uninitialized it could return
very-very-poor-quality random bits. On 3.5.0 and 3.5.1, on Linux,
when using the getrandom() call, it will instead block for an
apparently unbounded period before returning high-quality random
bits. The question: is this new behavior preferable, or should we
return to the old behavior?<br>
<br>
<br>
Since I'm the one writing this email, let me make the case for my
position: I think that os.urandom() should never block on Linux.
Why?<br>
<br>
<br>
1) Functions in the os module that look like OS functions should
behave predictably like thin wrappers over those OS functions.<br>
<br>
Most of the time this is exactly what they are. In some cases
they're more sophisticated; examples include os.popen(),
os.scandir(), and the byzantine os.utime(). There are also some
functions provided by the os module that don't resemble any native
functionality, but these have unique names that don't look like
anything provided by the OS.<br>
<br>
This makes the behavior of the Python function easy to reason about:
it always behaves like your local OS function. Python provides
os.stat() and it behaves like the local stat(). So if you want to
know how any os module function behaves, just read your local man
page. Therefore, os.urandom() should behave exactly like a thin
shell around reading the local /dev/urandom.<br>
<br>
On Linux, /dev/urandom guarantees that it will never block. This
means it has undesirable behavior if read immediately after a fresh
boot. But this guarantee is so strong that Theodore Ts'o couldn't
break it to fix the undesirable behavior. Instead he added the
getrandom() system call. But he left /dev/urandom alone.
Therefore, on Linux, os.urandom() should behave the same way, and
also never block.<br>
<br>
<br>
2) It's unfair to change the semantics of a well-established
function to such a radical degree.<br>
<br>
os.urandom() has been in Python since at least 2.6--I was too lazy
to go back any further. From 2.6 to 3.4, it behaved exactly like
/dev/urandom, which meant that on Linux it would never block. As of
3.5, on Linux, it might now block for an unbounded period of time.
Any code that calls os.urandom() has had its behavior radically
changed in this extreme corner case.<br>
<br>
<br>
3) os.urandom() doesn't actually guarantee it's suitable for
cryptography.<br>
<br>
The documentation for os.urandom() has contained this sentence,
untouched, since 2.6:<br>
<blockquote>The returned data should be unpredictable enough for
cryptographic applications, though its exact quality depends on
the OS implementation. On a Unix-like system this will query
/dev/urandom, and on Windows it will use CryptGenRandom().<br>
</blockquote>
Of course, version 3.5 added this:<br>
<blockquote>On Linux 3.17 and newer, the getrandom() syscall is now
used when available. <br>
</blockquote>
But the waffling about its suitability for cryptography remains
unchanged. So, while it's undesirable that os.urandom() might
return shockingly poor quality random bits, it is *permissible*
according to the documentation.<br>
<br>
<br>
4) This really is a rare corner-case we're talking about.<br>
<br>
I just want to re-state: this case on Linux where /dev/urandom
returns totally predictable bytes, and getrandom() will block, only
happens when the entropy pool for urandom is uninitialized.
Although it has been seen in the field, it's extremely rare.
99.99999%+ of the time, reading /dev/urandom and calling getrandom()
will both return the exact same high-quality random bits without
blocking.<br>
<br>
<br>
5) This corner-case behavior is fixable externally to CPython.<br>
<br>
I don't really understand the subject, but apparently it's entirely
reasonable to expect sysadmins to directly manage the entropy pools
of virtual machines. They should be able to spin up their VMs with
a pre-filled entropy pool. So it should be possible to ensure that
os.urandom() always returns the high-quality random bits we wanted,
even on freshly-booted VMs.<br>
<br>
<br>
6) Guido and Tim Peters already decided once that os.urandom()
should behave like /dev/urandom.<br>
<br>
Issue #25003:<br>
<blockquote><a class="moz-txt-link-freetext" href="http://bugs.python.org/issue25003">http://bugs.python.org/issue25003</a><br>
</blockquote>
<br>
In 2.7.10, os.urandom() was changed to call getentropy() instead of
reading /dev/urandom when getentropy() was available. getentropy()
was "stunningly slow" on Solaris, on the order of 300x slower than
reading /dev/urandom. Guido and Tim both participated in the
discussion on the issue; Guido also apparently discussed it via
email with Theo De Raadt.<br>
<br>
While it's not quite apples-to-apples, I think this establishes some
precedent that os.urandom() should<br>
* behave like /dev/urandom, and<br>
* be fast.<br>
<br>
<br>
--<br>
<br>
On the other side is... everybody else. I've already spent an
enormous amount of time researching and writing and re-writing this
email. Rather than try (and fail) to accurately present the other
sides of this debate, I'm just going to end the email here and let
the other participants reply and voice their views.<br>
<br>
<br>
Bottom line: Guido, in this extreme corner case on Linux, should
os.urandom() return bad random data like it used to, or should it
block forever like it does in 3.5.0 and 3.5.1?<br>
<br>
<br>
<i>/arry</i><br>
</body>
</html>