[Python-ideas] Pre-PEP Adding A Secrets Module To The Standard Library

Steven D'Aprano steve at pearwood.info
Sat Sep 19 20:16:12 CEST 2015

Following on to the discussions about changing the default random number 
generator, I would like to propose an alternative: adding a secrets 
module to the standard library.

Attached is a draft PEP. Feedback is requested.

(I'm going to only be intermittently at the keyboard for the next day or 
so, so my responses may be rather slow.)


-------------- next part --------------
PEP: xxx
Title: Adding A Secrets Module To The Standard Library
Version: $Revision$
Last-Modified: $Date$
Author: Steven D'Aprano <steve at pearwood.info>
Status: Draft
Type: Standards Track
Content-Type: text/plain
Created: 19-Sep-2015
Python-Version: 3.6


    This PEP proposes the addition of a module for common security-related
    functions such as generating tokens to the Python standard library.


    Some common abbreviations used in this proposal:

            Pseudo Random Number Generator.  A deterministic algorithm used
            to produce random-looking numbers with certain desirable
            statistical properties.

            Cryptographically Strong Pseudo Random Number Generator.  An
            algorithm used to produce random-looking numbers which are
            resistant to prediction.

            Mersenne Twister.  An extensively studied PRNG which is currently
            used by the ``random`` module as the default.


    This proposal is motivated by concerns that Python's standard library
    makes it too easy for developers to inadvertently make serious security
    errors.  Theo de Raadt, the founder of OpenBSD, contacted Guido van Rossum
    and expressed some concern[1] about the use of MT for generating sensitive
    information such as passwords, secure tokens, session keys and similar.

    Although the documentation for the random module explicitly states that
    the default is not suitable for security purposes[2], it is strongly
    believed that this warning may be missed, ignored or misunderstood by
    many Python developers.  In particular:

    - developers may not have read the documentation and consequently
      not seen the warning;

    - they may not realise that their specific use of it has security
      implications; or

    - not realising that there could be a problem, they have copied code
      (or learned techniques) from websites which don't offer best

    The first[3] hit when searching for "python how to generate passwords" on
    Google is a tutorial that uses the default functions from the ``random``
    module[4].  Although it is not intended for use in web applications, it is
    likely that similar techniques find themselves used in that situation.
    The second hit is to a StackOverflow question about generating
    passwords[5].  Most of the answers given, including the accepted one, use
    the default functions.  When one user warned that the default could be
    easily compromised, they were told "I think you worry too much."[6]

    This strongly suggests that the existing ``random`` module is an attractive
    nuisance when it comes to generating (for example) passwords or secure

    Additional motivation (of a more philosophical bent) can be found in the
    post which first proposed this idea[7].


    Alternative proposals have focused on the default PRNG in the ``random``
    module, with the aim of providing "secure by default" cryptographically
    strong primitives that developers can build upon without thinking about
    security.  (See Alternatives below.)  This proposes a different approach:

    * The standard library already provides cryptographically strong
      primitives, but many users don't know they exist or when to use them.

    * Instead of requiring crypto-naive users to write secure code, the
      standard library should include a set of ready-to-use "batteries" for
      the most common needs, such as generating secure tokens.  This code
      will both directly satisfy a need ("How do I generate a password reset
      token?"), and act as an example of acceptable practises which
      developers can learn from[8].

    To do this, this PEP proposes that we add a new module to the standard
    library, with the suggested name ``secrets``.  This module will contain a
    set of ready-to-use functions for common activities with security
    implications, together with some lower-level primitives.

    The suggestion is that ``secrets`` becomes the go-to module for dealing
    with anything which should remain secret (passwords, tokens, etc.)
    while the ``random`` module remains backward-compatible.

API and Implementation

    The contents of the ``secrets`` module is expected to evolve over time, and
    likely will evolve between the time of writing this PEP and actual release
    in the standard library[9].  At the time of writing, the following functions
    have been suggested:

    * A high-level function for generating secure tokens suitable for use
      in (e.g.) password recovery, as session keys, etc.

    * A limited interface to the system CSPRNG, using either ``os.urandom``
      directly or ``random.SystemRandom``.  Unlike the ``random`` module, this
      does not need to provide methods for seeding, getting or setting the
      state, or any non-uniform distributions.  It should provide the

      - A function for choosing items from a sequence, ``secrets.choice``.
      - A function for generating an integer within some range, such as
        ``secrets.randrange`` or ``secrets.randint``.
      - A function for generating a given number of random bits and/or bytes
        as an integer.
      - A similar function which returns the value as a hex digit string.

    * ``hmac.compare_digest`` under the name ``equal``.

    The consensus appears to be that there is no need to add a new CSPRNG to
    the ``random`` module to support these uses, ``SystemRandom`` will be

    Some illustrative implementations have been given by Nick Coghlan[10].
    This idea has also been discussed on the issue tracker for the
    "cryptography" module[11].

    The ``secrets`` module itself will be pure Python, and other Python
    implementations can easily make use of it unchanged, or adapt it as


    One alternative is to change the default PRNG provided by the ``random``
    module[12].  This received considerable scepticism and outright opposition:

    * There is fear that a CSPRNG may be slower than the current PRNG (which
      in the case of MT is already quite slow).

    * Some applications (such as scientific simulations, and replaying
      gameplay) require the ability to seed the PRNG into a known state,
      which a CSPRNG lacks by design.

    * Another major use of the ``random`` module is for simple "guess a number"
      games written by beginners, and many people are loath to make any
      change to the ``random`` module which may make that harder.

    * Although there is no proposal to remove MT from the ``random`` module,
      there was considerable hostility to the idea of having to opt-in to
      a non-CSPRNG or any backwards-incompatible changes.

    * Demonstrated attacks against MT are typically against PHP applications.
      It is believed that PHP's version of MT is a significantly softer target
      than Python's version, due to a poor seeding technique[13].  Consequently,
      without a proven attack against Python applications, many people object
      to a backwards-incompatible change.

    Nick Coghlan made an earlier suggestion for a globally configurable PRNG
    which uses the system CSPRNG by default[14], but has since hinted that he
    may withdraw it in favour of this proposal[15].

Comparison To Other Languages


        PHP includes a function ``uniqid``[16] which by default returns a
        thirteen character string based on the current time in microseconds.
        Translated into Python syntax, it has the following signature:

            def uniqid(prefix='', more_entropy=False)->str

        The PHP documentation warns that this function is not suitable for
        security purposes.  Nevertheless, various mature, well-known PHP
        applications use it for that purpose (citation needed).

        PHP 5.3 and better also includes a function ``openssl_random_pseudo_bytes``[17].
        Translated into Python syntax, it has roughly the following signature:

            def openssl_random_pseudo_bytes(length:int)->Tuple[str, bool]

        This function returns a pseudo-random string of bytes of the given
        length, and an boolean flag giving whether the string is considered
        cryptographically strong.  The PHP manual suggests that returning
        anything but True should be rare except for old or broken platforms.


        Based on a rather cursory search[18], there doesn't appear to be any
        well-known standard functions for producing strong random values in
        Javascript, although there may be good quality third-party libraries.
        Standard Javascript doesn't seem to include an interface to the
        system CSPRNG either, and people have extensively written about the
        weaknesses of Javascript's Math.random[19].


        The Ruby standard library includes a module ``SecureRandom``[20]
        which includes the following methods:

        * base64 - returns a Base64 encoded random string.

        * hex - returns a random hexadecimal string.

        * random_bytes - returns a random byte string.

        * random_number - depending on the argument, returns either a random
          integer in the range(0, n), or a random float between 0.0 and 1.0.

        * urlsafe_base64 - returns a random URL-safe Base64 encoded string.

        * uuid - return a version 4 random Universally Unique IDentifier.

What Should Be The Name Of The Module?

    There was a proposal to add a "random.safe" submodule, quoting the Zen
    of Python "Namespaces are one honking great idea" koan.  However, the
    author of the Zen, Tim Peters, has come out against this idea[21], and
    recommends a top-level module.

    In discussion on the python-ideas mailing list so far, the name "secrets"
    has received some approval, and no strong opposition.

Frequently Asked Questions

    Q: Is this a real problem? Surely MT is random enough that nobody can
       predict its output.

    A: The consensus among security professionals is that MT is not safe
       in security contexts.  It is not difficult to reconstruct the internal
       state of MT[22][23] and so predict all past and future values.  There
       are a number of known, practical attacks on systems using MT for

       While there are currently no known direct attacks on applications
       written in Python due to the use of MT, there is widespread agreement
       that such usage is unsafe.

    Q: Is this an alternative to specialise cryptographic software such as SSL?

    A: No. This is a "batteries included" solution, not a full-featured
       "nuclear reactor".  It is intended to mitigate against some basic
       security errors, not be a solution to all security-related issues. To
       quote Nick Coghlan referring to his earlier proposal:

            "...folks really are better off learning to use things like
            cryptography.io for security sensitive software, so this change
            is just about harm mitigation given that it's inevitable that a
            non-trivial proportion of the millions of current and future
            Python developers won't do that."[25]


    [1] https://mail.python.org/pipermail/python-ideas/2015-September/035820.html

    [2] https://docs.python.org/3/library/random.html

    [3] As of the date of writing. Also, as Google search terms may be automatically customised for the user without their knowledge, some readers may see different results.

    [4] http://interactivepython.org/runestone/static/everyday/2013/01/3_password.html

    [5] http://stackoverflow.com/questions/3854692/generate-password-in-python

    [6] http://stackoverflow.com/questions/3854692/generate-password-in-python/3854766#3854766

    [7] https://mail.python.org/pipermail/python-ideas/2015-September/036238.html

    [8] At least those who are motivated to read the source code and documentation.

    [9] Tim Peters suggests that bike-shedding the contents of the module will be 10000 times more time consuming than actually implementing the module.  Words do not begin to express how much I am looking forward to this.

    [10] https://mail.python.org/pipermail/python-ideas/2015-September/036271.html

    [11] https://github.com/pyca/cryptography/issues/2347

    [12] Link needed.

    [13] By default PHP seeds the MT PRNG with the time (citation needed), which is exploitable by attackers, while Python seeds the PRNG with output from the system CSPRNG, which is believed to be much harder to exploit.

    [14] http://legacy.python.org/dev/peps/pep-0504/

    [15] https://mail.python.org/pipermail/python-ideas/2015-September/036243.html

    [16] http://php.net/manual/en/function.uniqid.php

    [17] http://php.net/manual/en/function.openssl-random-pseudo-bytes.php

    [18] Volunteers and patches are welcome.

    [19] http://ifsec.blogspot.fr/2012/05/cross-domain-mathrandom-prediction.html

    [20] http://ruby-doc.org/stdlib-2.1.2/libdoc/securerandom/rdoc/SecureRandom.html

    [21] https://mail.python.org/pipermail/python-ideas/2015-September/036254.html

    [22] https://jazzy.id.au/2010/09/22/cracking_random_number_generators_part_3.html

    [23] https://mail.python.org/pipermail/python-ideas/2015-September/036077.html

    [24] https://media.blackhat.com/bh-us-12/Briefings/Argyros/BH_US_12_Argyros_PRNG_WP.pdf

    [25] https://mail.python.org/pipermail/python-ideas/2015-September/036157.html


    This document has been placed in the public domain.

Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8

More information about the Python-ideas mailing list