Hi, I like the PEP. IMHO it's a better solution than using a CPRNG for random by default. I suggest to raise an error if token_bytes(n) if calls with n < 16 bytes (128 bits). Well, I'm not sure that 16 is the good compromise between performance and security, but we must enforce users to use a minimum number of bits of entropy. token_bytes(1) looks valid, even token_bytes(0), according to the Python code in the PEP. I don't like the idea how having two functions doing *almost* the same thing: randint() and randrange(). There is a risk that these functions will be misused. I consider that I know some stuff on PRNG but I'm still confused by randint() and randrange(). Usually, I open python and type:

...

...

x=[s.randrange(1,6) for n in range(100)] min(x), max(x) (1, 5)

Hum, ok, it's not a good dice :-) I probably wanted to use randint(). So I suggest to only add randint() to secrets. The PEP doesn't explain if secrets uses a "blocking" CPRNG (like /dev/random or getentropy() on Solaris) or a "non-blocking" CRPNG (like /dev/urandom). And it doesn't explain the rationale. Please explain, or I'm sure that the question will arise (ex: I just asked it ;-)) You may also be a little bit more explicit on the CPRNG: it *looks* like secrets will always use a CRPNG implemented in the kernel. Is it a property of the secrets module, or can it be ssl.RAND_bytes() for example? IMHO we must always use a CRPNG implemented in the kernel, there is still an issue with ssl.RAND_bytes() and fork() (two child process can produce exactly the same random numbers after a lot of fork()...). I understood that OpenSSL developers doesn't want to fix it. You may even be very explicit, list CPRNG that will be used on Python 3.6: * Linux: getrandom() syscall if available (Linux 3.17 or newer), or /dev/urandom * Solaris: getrandom() function if available (Solaris 11.3 or newer), or /dev/urandom * OpenBSD: getentropy() function (OpenBSD 5.6 or newer), or /dev/urandom * Windows: CryptAcquireContext(PROV_RSA_FULL, CRYPT_VERIFYCONTEXT) and CryptGenRandom() * Other UNIXes: /dev/urandom It's still unclear to me if getentropy() on OpenBSD can block or not if the entropy is too low :-/ Victor 2015-10-16 2:57 GMT+02:00 Steven D'Aprano <steve@pearwood.info>:

Hi,

As extensively discussed on Python-Ideas, the secrets module and PEP 506 is (I hope) ready for pronouncement.

https://www.python.org/dev/peps/pep-0506/

There is code and tests here:

https://bitbucket.org/sdaprano/secrets

or you can run

hg clone https://sdaprano@bitbucket.org/sdaprano/secrets

The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4.

-- Steve _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...

On Fri, Oct 16, 2015 at 9:04 PM, Steven D'Aprano <steve@pearwood.info> wrote:

On Fri, Oct 16, 2015 at 08:57:24AM +0200, Victor Stinner wrote:

...

Hi,

I like the PEP. IMHO it's a better solution than using a CPRNG for random by default.

I suggest to raise an error if token_bytes(n) if calls with n < 16 bytes (128 bits). Well, I'm not sure that 16 is the good compromise between performance and security, but we must enforce users to use a minimum number of bits of entropy. token_bytes(1) looks valid, even token_bytes(0), according to the Python code in the PEP.

Youtube URLs have what looks like about 8 or 9 bytes of randomness:

https://www.youtube.com/watch?v=B3KBuQHHKx0

(assuming it is base64 encoded, 11 chars is about 8 or 9 bytes).

I don't think that's randomness - it's some kind of unique ID, and I believe they lengthen with the log of the number of videos on Youtube.

Unless anyone has a good argument for why we should do this, I would be inclined to allow any value for nbytes. At most, I would be prepared to raise a warning if the nbytes was less than 16, but I don't think an error is appropriate.

Thoughts?

Not even a warning, IMO. If you ask for a certain number of bytes, you should get that many bytes. Raise ValueError on token_bytes(-1), and maybe token_bytes(0), but otherwise, do what you were told. For people who want "good enough security", token_bytes() is the correct choice; if you explicitly choose, you get what you ask for. And secrets.DEFAULT_ENTROPY is intentionally public, right? I see no __all__, and it doesn't start with an underscore. So if someone wants "twice as long as the default", s/he can spell that secrets.token_bytes(secrets.DEFAULT_ENTROPY*2). There should be no reason to hard-code a number if you don't specifically want that number.

...

You may also be a little bit more explicit on the CPRNG: it *looks* like secrets will always use a CRPNG implemented in the kernel. Is it a property of the secrets module, or can it be ssl.RAND_bytes() for example? IMHO we must always use a CRPNG implemented in the kernel, there is still an issue with ssl.RAND_bytes() and fork() (two child process can produce exactly the same random numbers after a lot of fork()...). I understood that OpenSSL developers doesn't want to fix it.

You may even be very explicit, list CPRNG that will be used on Python 3.6:

* Linux: getrandom() syscall if available (Linux 3.17 or newer), or /dev/urandom * Solaris: getrandom() function if available (Solaris 11.3 or newer), or /dev/urandom * OpenBSD: getentropy() function (OpenBSD 5.6 or newer), or /dev/urandom * Windows: CryptAcquireContext(PROV_RSA_FULL, CRYPT_VERIFYCONTEXT) and CryptGenRandom() * Other UNIXes: /dev/urandom

Does anyone else think that the secrets module should promise specific implementations?

I think it should NOT. The point of this is to provide whatever is believed to be secure, and that should never be promised. If RHEL is still shipping CPython 3.6 ten years from now, the best source of entropy might have changed, and the module should be able to be patched. It may be worth having the implementation introspectable, though - have some attribute that identifies the source and any CSPRNG algorithms used. Might not be much use in practice, given that you can just look at the source code. ChrisA

On Fri, Oct 16, 2015 at 06:35:14PM +0300, Serhiy Storchaka wrote:

I suggest to add only randrange(). randint() is historical artefact, we shouldn't repeat this mistake in new module. The secrets module is not good way to generate dice rolls. In most other cases you need to generate integers in half-open interval [0; N).

And randbelow() is absolute redundant. Random._randbelow() is implementation detail and I inclined to get rid of it (implementing randrange() in C instead).

This was discussed on Python-Ideas, and there was little consensus there either. (Looks like Tim Peters' prediction is coming true :-) Putting aside your inflammatory description of randint() as a "mistake", if you are correct that in most cases people will need to generate integers in the half-open interval [0...n) then we should keep randbelow, since that is precisely what it does. randrange([start=0,] end [, step=1]) is a complex API. It can take one, two or three arguments, like range. Are there any use-cases for providing the step argument? If not, then why offer such a complex API that will never be used? Personally, I have no sense of which of the three functions will be most useful, but if you are right about the half-open [0...n) interval, then randbelow seems to be the right API to offer. But I have seen people argue in favour of randint, and others argue in favour of randrange. Given that these are just thin wrappers or aliases to methods of random.SystemRandom, I don't think there is any harm in providing all three. I've also raised this issue on the python-list mailing list. -- Steve

Single-argument randrange(n) is the same as randbelow(n), right? I don't see any reason to have randbelow() if that's true. On Fri, Oct 16, 2015 at 11:29 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:

On 16.10.15 19:26, Steven D'Aprano wrote:

...

On Fri, Oct 16, 2015 at 06:35:14PM +0300, Serhiy Storchaka wrote:

...

I suggest to add only randrange(). randint() is historical artefact, we shouldn't repeat this mistake in new module. The secrets module is not good way to generate dice rolls. In most other cases you need to generate integers in half-open interval [0; N).

And randbelow() is absolute redundant. Random._randbelow() is implementation detail and I inclined to get rid of it (implementing randrange() in C instead).

This was discussed on Python-Ideas, and there was little consensus there either. (Looks like Tim Peters' prediction is coming true :-)

Putting aside your inflammatory description of randint() as a "mistake", if you are correct that in most cases people will need to generate integers in the half-open interval [0...n) then we should keep randbelow, since that is precisely what it does.

Andrew explained the history of the issue ( http://permalink.gmane.org/gmane.comp.python.ideas/36437). randrange was added in 61464037da53 to address a problem with unpythonic randint.

Personally, I have no sense of which of the three functions will be most

...

useful, but if you are right about the half-open [0...n) interval, then randbelow seems to be the right API to offer. But I have seen people argue in favour of randint, and others argue in favour of randrange. Given that these are just thin wrappers or aliases to methods of random.SystemRandom, I don't think there is any harm in providing all three.

Yes, randbelow provides simpler API, but randrange is more familiar for Python users due to similarity to range and because it is the public API in the random module (unlike to randbelow).

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

On Sat, Oct 17, 2015 at 03:26:46AM +1100, Steven D'Aprano wrote:

[snip]

But significanly, only *one* of the commenters has claimed to have any significant experience in crypto work, and I will quote him:

I didn't specifically claim the experience you requested in responding to your post on comp.lang.python because I thought that this was implied in making a response. In fact I have 30+ years of experience in implementing cryptographic code (much involving random numbers) so there were at least two respondents who could have made this claim. For the record, I consider it desirable in code involving security to exhibit the minimum functionality neccessary to get a job done. This is because funtionality and security very often work against each other in building secure systems. I hence support your conclusion that the module should offer randbelow alone. I would oppose offering randomrange (or offering more than one of them) since this will pretty well guarantee that, sooner or later, someone will make a mistake in using the extra functionality and possibly deploy an insecure application as a result. Brian Gladman

On Sat, Oct 17, 2015 at 2:50 AM, Steven D'Aprano <steve@pearwood.info> wrote:

[...] So you can see there is nothing even close to consensus as to which API is best, which is an argument for keeping all three functions.

No, that's not how we do it in Python. :-)

But significanly, only *one* of the commenters has claimed to have any significant experience in crypto work, and I will quote him:

Having done quite a bit of serious crypto implementation over the past 25 years, I don't recall ever wanting anything like randrange, and if I did need it, I'd probably build it inline from randbelow rather than force some hapless future code maintainer to look up the specs on randrange.

My opinion, FWIW: I like randbelow, because in modern crypto one very frequently works with integers in the range [0,M-1] for some large modulus M, and there is a constant risk of asking for something in [0,M] when one meant [0,M-1]. One can eliminate this risk, as randbelow does, by building in the -1, which normally introduces a risk of making a mistake that gives you [0,M-2], but the name "randbelow" seems like a neat fix to that problem.

-- Peter Pearson

It's not clear whether this correspondent realizes that randrange(N) is identical to randbelow(N).

This matches what Serhiy suggests: in crypto, one normally only needs to generate the half-open interval [0...n). It also matches the reason why Tim Peters added randbelow in the first place.

As the author of the PEP, I'm satisfied by this argument, and will now state that my preferred option is to drop randint and randrange, and just keep randbelow.

My second choice is to keep all three functions.

I think it is fair to say that out of the three functions, there is consensus that randbelow has the most useful functionality in a crypto context. Otherwise, people seem roughly equally split between the three functions. There doesn't seem to be any use-case for the three argument version of randrange.

I'm fine with dropping the 3rd arg. But I find the argument to introduce a new spelling for 1-arg randrange() weak. I definitely thing that randint() is an attractive nuisance so we should drop that. -- --Guido van Rossum (python.org/~guido)

[Steven D'Aprano]

...

... I think it is fair to say that out of the three functions, there is consensus that randbelow has the most useful functionality in a crypto context. Otherwise, people seem roughly equally split between the three functions. There doesn't seem to be any use-case for the three argument version of randrange.

[Guido]

I'm fine with dropping the 3rd arg. But I find the argument to introduce a new spelling for 1-arg randrange() weak.

Note the inherent absurdity here ;-) That is, we're proposing to add a module _all_ of whose functions are merely trivial (to the educated) respellings of functions that already exist elsewhere. Is there, e.g., really "a strong" argument for giving a different way to spell os.urandom()? That depends. I'm playing along with the basic assumption: that this module intends to be as idiot-proof as possible. In which case, sure, rename away. And in which case, no, randbelow is not a new spelling for 1-arg randrange. To the contrary, all the integer-like functions (including randrange, randint, and choice) in the random module are currently built on top of the private Random._randbelow() method. The latter is "the right" fundamental building block for implementing uniform choice free of statistical bias. It's also overwhelmingly most useful _on its own_ in the `secrets` context, and has the crushing (to me, in this context) advantage that its very name strongly suggests its argument is not a possible return value. Giving a strongly mnemonic name to a function with a dead simple "one required integer argument" signature is likely "as idiot-proof as possible" as it gets. BTW, it also gives a simple answer to "I'm used to using arc4random_uniform() in OpenBSD - how do I spell that in `secrets`?". Answer: `randbelow()` is identical, except in Python the argument isn't limited to uint32_t".

I definitely thing that randint() is an attractive nuisance so we should drop that.

`randrange()` isn't a nuisance at all in Python, but its signature is more convoluted than putative users of the proposed module seem to want, and long experience has shown its name is not idiot-proof. `secrets` users aren't looking to pick something uniformly from "a range" - they're looking to pick a non-negative integer less than some upper bound. Unneeded generalization beyond just that much is also an attractive nuisance, in context. "Simplest thing that can possibly suffice" is always a good way to start :-)

On 16 October 2015 at 12:04, Steven D'Aprano <steve@pearwood.info> wrote:

On Fri, Oct 16, 2015 at 08:57:24AM +0200, Victor Stinner wrote:

...

I don't like the idea how having two functions doing *almost* the same thing: randint() and randrange(). There is a risk that these functions will be misused. I consider that I know some stuff on PRNG but I'm still confused by randint() and randrange(). Usually, I open python and type:

...

...

...

x=[s.randrange(1,6) for n in range(100)] min(x), max(x) (1, 5)

Wouldn't help(randrange) be easier? :-)

Choose a random item from range(start, stop[, step]).

This fixes the problem with randint() which includes the endpoint; in Python this is usually not what you want.

I always find that comment amusing. While it is true that in slicing, half-open ranges are more useful than closed ranges, but when it comes to generating random numbers (say, simulating dice) I find randint much more useful and intuitive.

But I appreciate that some people think differently.

Folks wanting to simulate die rolls should be using the random module rather than the secrets module anyway, so the "only offer secrets.randbelow()" approach Guido suggested in his last email makes sense to me. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 20 October 2015 at 11:33, Victor Stinner <victor.stinner@gmail.com> wrote:

2015-10-20 11:11 GMT+02:00 Nick Coghlan <ncoghlan@gmail.com>:

...

Folks wanting to simulate die rolls should be using the random module rather than the secrets module anyway,

Hum, why? Dices are used in Casino where security matters because it costs money.

True, I was thinking of just-for-fun games, but in gambling games unbiased randomness can be significantly more important.

A bad API can be more likely misused and introduce security vulnerability. The C rand() API is a good example: 1+rand()%6 is not uniform...

"1 + secrets.randbelow(6)" would be uniform, though. As Tim pointed out, the *lack* of flexibility in randbelow() is a feature here, since it focuses on producing a uniformly random distribution of a given size, which can then be transformed deterministically. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I think the discussion petered out and nobody asked me to approve it yet (or I lost track of it). I'm almost happy to approve it in the current state. My only quibble is with some naming -- I'm not sure that a super-generic name like 'equal' is better than the original ('compare_digest'), and I would have picked a different name for token_url -- probably token_urlsafe. But maybe Steven can convince me that the names currently in the PEP are better. (I also don't like the wishy-washy position of the PEP on the actual specs of the proposed functions. But I'm fine with the actual implementation shown as the spec.) On Thu, Jan 14, 2016 at 10:36 AM, Brett Cannon <brett@python.org> wrote:

I noticed an article about default rand usage in Go <http://blog.sgmansfield.com/2016/01/the-hidden-dangers-of-default-rand/> from the Go Weekly newsletter and it reminded me about PEP 506 and the secrets module. That's when I noticed that the PEP is still open. What is the current blocker on the PEP?

On Thu, 15 Oct 2015 at 17:57 Steven D'Aprano <steve@pearwood.info> wrote:

...

Hi,

As extensively discussed on Python-Ideas, the secrets module and PEP 506 is (I hope) ready for pronouncement.

https://www.python.org/dev/peps/pep-0506/

There is code and tests here:

https://bitbucket.org/sdaprano/secrets

or you can run

hg clone https://sdaprano@bitbucket.org/sdaprano/secrets

The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4.

-- Steve _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

Hi Steven, No probIem with the delay -- it's still before 3.6.0. I do think it's just about a record gap in the PEP review process. :-) I will approve the PEP as soon as you've updated the two function names in the PEP. (If you don't have write access to the peps repo, send the new version to peps@python.org -- or send a link to the new draft somewhere online, e.g. github if you're using that. If you do have peps repo write access, just reply here when it's done.) Regarding the alluded vagueness of the PEP on the specs, I think I was mostly about the phrase "At the time of writing, the following functions have been suggested" which doesn't seem to commit very strongly to a specific API. The later phrase "The following pseudo-code can be taken as a possible starting point for the real implementation" doesn't really do much to take away the feeling that the PEP is non-committal on the actual API it proposes. But I don't want to approve the *idea* of a secrets module -- I want to approve a specific API. Maybe you can just change the words a bit to say something like "this PEP proposes the following API; the implementations given here are not final". None of this will prevent adding more functions to secrets.py before 3.6.0 is released (or, of course, in 3.7, 3.8 etc.), but it should send a clear message that we've agreed on these specific names and signatures, and that those are what I'm approving. If we change our minds about the API of the module before releasing 3.6.0, we should treat it as an amendment to the PEP and take it pretty seriously (but it's happened before so it's not impossible). Hopefully this message isn't drowned in the infinity of pathlib and ~bool threads, and we can proceed to add secrets.py to the 3.6 stdlib. You should be proud of that accomplishment! --Guido On Sat, Apr 9, 2016 at 10:08 PM, Steven D'Aprano <steve@pearwood.info> wrote:

I've just spotted this email from Guido, sorry about the delay in responding.

Further comments below.

On Thu, Jan 14, 2016 at 10:47:09AM -0800, Guido van Rossum wrote:

...

I think the discussion petered out and nobody asked me to approve it yet (or I lost track of it). I'm almost happy to approve it in the current state. My only quibble is with some naming -- I'm not sure that a super-generic name like 'equal' is better than the original ('compare_digest'),

Changed.

...

and I would have picked a different name for token_url -- probably token_urlsafe. But maybe Steven can convince me that the names currently in the PEP are better.

Changed.

...

(I also don't like the wishy-washy position of the PEP on the actual specs of the proposed functions. But I'm fine with the actual implementation shown as the spec.)

I'm not really sure what you want me to do to improve that. Can you be more concrete about what you would like the PEP to say?

I haven't updated the PEP yet, but the newest version of the secrets module with the changes requested is here:

https://bitbucket.org/sdaprano/secrets

-- Steve _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

On Sun, Apr 10, 2016 at 11:43:08AM -0700, Guido van Rossum wrote:

Hi Steven,

No probIem with the delay -- it's still before 3.6.0. I do think it's just about a record gap in the PEP review process. :-)

I will approve the PEP as soon as you've updated the two function names in the PEP. (If you don't have write access to the peps repo, send the new version to peps@python.org -- or send a link to the new draft somewhere online, e.g. github if you're using that. If you do have peps repo write access, just reply here when it's done.)

I have done that, and updated the API and Implementation section to be less wishy-washy and more commital about what exactly will be included. Hope it meets with your approval, and thanks for your guidance! -- Steve

On 04/11/2016 11:35 AM, Guido van Rossum wrote:

Most excellent! PEP 506 is hereby approved. Congrats again.

Congratulations, Steven! -- ~Ethan~

On 15 April 2016 at 19:39, Victor Stinner <victor.stinner@gmail.com> wrote:

Hi,

Would it make sense to add a function to generate a random UUID4 (as a string) in secrets?

The current implement in uuid.py of CPython 3.6 already uses os.urandom():

def uuid4(): """Generate a random UUID.""" return UUID(bytes=os.urandom(16), version=4)

I don't think so, as folks looking to generate a UUID specifically are already likely to end up at the uuid module docs rather than trying to craft their own based on the random module (and the uuid module already does the right thing, and it would be a bug if it didn't). The new secrets module fills the gap for cases where random is otherwise an attractive nuisance by making it easy to say "use this instead". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia