"Do hash functions such as SHA-2 and SHA-3 output random numbers, and are they therefore good P/RNGs?" https://www.quora.com/Do-hash-functions-such-as-SHA-2-and-SHA-3-output-rando... - ```quote See NIST.SP.800–90A. It details how to use both hash functions and block ciphers as PRNGs. When used properly (see NIST publication), hash functions are perfectly suitable as PRNGs and are used that way frequently. ``` - NIST SP 800-90A Rev. 1 "Recommendation for Random Number Generation Using Deterministic Random Bit Generators" https://csrc.nist.gov/publications/detail/sp/800-90a/rev-1/final PDF: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-90Ar1.pdf Keywords: Deterministic random bit generator (DRBG); entropy; hash function; random number generator. - FWIW, {blockchain xyz(*) SHA-256}'s block hash digits do appear to be conditionally independent in hexadecimal (radix 16) with tx messages and nonces for inputs. - (*) informal study of XBT block hashes - ```quote There are a number of reasons why hash functions should not be used as RNGs, and hash Merkle-Dmagård constructions (including SHA1 and SHA2) should not be used directly for key derivation. That is, something like SHA2(salt+secret) should not be used to generate a key. Instead if you want to use SHA2 to derive a key, you should use it within an HMAC construction, or better still HKDF as a key derivation function (KDF). SHA3 is different. One of the design criteria for the SHA3 is that not have that limitation of its predecessors. So in principle it can be used that way. However, you still should roll your own KDF from SHA3. Instead you should use a professionally designed and implementation KDF. Internally, a KDF that uses SHA3 will be more direct in its use of the hash function than one designed to handle the limitations of SHA2, if you do try to roll your own you are less likely to shoot yourself in the foot with SHA3. But if you are asking the question you are asking, you definitely should not be rolling your own. Now perhaps you weren’t thinking about key derivation. Perhaps your question is more theoretical. In a his case it is important to understand that among other things, a secure RNG must keep its internal state secret. That way, revealing one key does not reveal information about other keys. Hash functions are not really designed with that in mind. (Well, SHA3 is, but it is still not designed to be an RNG.) SHA2 would make a terrible random number generator. SHA3 comes a lot closer, but still isn’t build for the purpose. ``` On Tue, Nov 15, 2022 at 8:58 AM James Johnson <jj126979@gmail.com> wrote:
I am an amateur enthusiast. The random function I found in documentation was in 2014 or so. I do not have the vocabulary to differentiate between various prng's. I only knew of one, and I it returned some version of the bell curve, with three favorite values, and one of THEM a pronounced preference.
The scholars here are referencing white papers and using hardware and software interchangeably (?) I am not a randomization expert, and I do not seek academic recognition. It's a novel hack is all, and I don't know how to invoke the quality algorithms you reference.
Chris A. asked me to "post" a module. I'm pretty sure he means on git-hub, and I don't know how to use it. I attach a script with a randbelow() function.
The kernel of my idea is to use randomness from a hash value, so I declared the hash globally, and called it in the function. I know this may not be correct, but I don't know how to declare one momentarily and then destruct it or release the ram the next minute, between random numbers.
Normally, I would update the hexdigest outside the function, and pass it to randbelow(range, hexdigest_string) as two arguments. Chris A wants it to feed an analysis program that may not agree well with the "permanence" of a hash between random numbers.
I am willing to put in effort, but it's a steep learning curve for me. I DO understand that my first script was a lot of data entry, data checking and elementary reporting that was not germane.
I use my fast computer off-line, and connect to the web with a Chromebook, so online tools would mean changing my setup. I remain interested to help and learn.
I hope this is not too bare-bones.
On Tue, Nov 15, 2022 at 7:17 AM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
James Johnson writes:
I want to be good natured about it, not combative. I tried whatever random function I found in documentation online, and used it. Without analyzing the world, I was simply dissatisfied with the results.
On the Python lists, you'll get much better discussion if you're specific about your "dissatisfaction". (If it's just the repetition problem, that's better handled by composing with a filter than by altering the PRNG itself, which should be as close to uniform as possible to make generating other distributions as simple as possible.)
Your method *as described* is unclear. Don Knuth tried something similar (you have better tools available than he did, but the basic idea seems to be similarly "let's compose a few pseudo-randomizing transformations"), and got an embarrassing result. (Spoiler in footnote [1] if you don't feel like digging through Seminumerical Algorithms.) You may have had more method than Knuth did (cf Dave Mertz's statement that it "looks like Random123"), but again, it would really help communication if you described why you think your PRNG will do better than Python's current one. It's a heavily researched field for many decades now, and there are quite a few core devs with a lot of knowledge and experience in the field (Python apps provide quite of bit of the "attack surface" exposed on the Internet, it would be irresponsible if there weren't!)
By the way, nobody here is going to laugh if it turns out that your thinking was naive. My point is to communicate more effectively, not that your idea is bad (despite having a copy of Seminumerical Algorithms, I'm not competent to evaluate your algorithm itself :-). How you feel about "getting schooled" is a personal thing, but my personal experience has been that there's an excellent education to be had by posting naive ideas to this list. :-)
Sincere regards,
N.B. Deliberately not trimming so you have to work to see the spoiler. :-)
Here’s what I settled on. I actually “put my thumb on the scales,” to
out repetitions for seven questions, so randomness didn’t ultimately answer my question.
I’m not sure that the function I “got” was the Mersenne Twister. I do know Mersenne was a truly great mathematician.
I acknowledge that updating with time.asctime() EVERY time I update the hash would be random; I’m not qualified to know if it would be “more” random. After you ask about the bell curve v square distributions, I am quite out of my depth.
Here’s my “wizard” for anyone who wants to sell him cryptocurrency or overcome his objections to the environmentalist agenda, or query him about foreign policy. I hope you find it better than average.
https://drive.google.com/file/d/1EqQsMfBHDrNpOBQrI7CJxrpbowvKPD2G/
Regards,
James J
On Tue, Nov 15, 2022 at 4:31 AM Wes Turner <wes.turner@gmail.com> wrote:
While FWIU it is advisable to keep seeding an RNG after startup -
what e.g. rngd does - the cryptography.io docs do advise to just use `os.urandom()` (which is not the same as random.SystemRandom()?).
There probably should be better default random in CPython; though I'm personally not at all qualified to assess, TIL about NIST 800-22 and FIPS 140-2 for evaluating sources of entropy.
Differential entropy > Differential entropies for various distributions https://en.wikipedia.org/wiki/Differential_entropy
***
(TIL that quantum information is never destroyed; so which physical processes are actually nonreversible like hashing and RNG are supposed to be is up for consideration as classical information is a subset of quantum information. Do black holes shift gamma radiation may actually be relevant! Perhaps a permanent magnet under a (double-jointed?) bar and ball
on a magnetic bearing would produce enough Brownian motion to get enough uniform random to call it sufficiently entropic for purposes of CSPRNG? Nondeterministic NDE fluid calculations diffract into many possible outcomes, but a brute force combinatorial search of all the possible return values is still deterministically ordered. And the quantum computing folks are working on increasing coherence / reducing error propagation;
- [ ] DOC: The docs could advise regarding which decent enough open
hw RNG would be supported by os.urandom or random.SystemRandom if rngd is not configured to keep nondeterministically seeding with entropy that should presumably be from a Uniform random entropic process
There should be better software random in python.
On Tue, Nov 15, 2022, 1:19 AM James Johnson <jj126979@gmail.com> wrote:
Thank you for replying with such specific assistance. I am made acutely aware that I am only a Python enthusiast, and not an academic.
Hashes are deterministic, not random, but byte by byte, they can be very random. Please accept the attached script as a "hack," that might be novel, or a curiosity. My first script was defective several ways. In
the hash by hashing the uu-8 ENCODED string value of the time - it is not random, but it is rarely the same twice. I used 0-9 because it is decimal, and 0-59, because hours and minutes come in 60's.
I found that taking time.time_nc() gave me far more odds than evens.
I do not know how the scheme I devised of adding hex digits together would scale to larger ranges - it is done by adding hex digits that vary from 0 - 15 repeatedly, but, since the number of hex digits varies with the range, it could actually work for larger ranges.
The random function in Python is not really adequate for a magic eight ball program, so as a consumer I am dissatisfied without examining it at a quantum level, for repeated reference thousands of times each.
Thank you for your kind attention, and I hope the (much improved) hack can be helpful to other enthusiasts.
James J (A.A. Faulkner University)
On Mon, Nov 14, 2022 at 11:23 AM Wes Turner <wes.turner@gmail.com> wrote:
QRNG "Quantum Random Number Generation" -> Hardware random number generator > Physical phenomena with random properties > Quantum random properties
https://en.wikipedia.org/wiki/Hardware_random_number_generator#Quantum_rando...
FWIW, SciPy and SymPy have various non-CSPRNG random distributions:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.uniform.htm...
https://docs.sympy.org/latest/modules/stats.html#sympy.stats.Uniform
-
https://docs.sympy.org/latest/modules/stats.html#sympy.stats.DiscreteUniform
"RFC 8937: Randomness Improvements for Security Protocols" (2020) https://www.rfc-editor.org/rfc/rfc8937.html
***
FWIW, UUIDs and W3C DIDs require entropy e.g. from a CSPRNG or better: https://www.w3.org/TR/did-core/#terminology :
> Universally Unique Identifier (UUID) > A type of globally unique identifier defined by [RFC4122]. UUIDs
are
similar to DIDs in that they do not require a centralized registration authority. UUIDs differ from DIDs in that they are not resolvable or cryptographically-verifiable
On Mon, Nov 14, 2022, 11:54 AM Wes Turner <wes.turner@gmail.com> wrote:
> https://docs.python.org/3/library/random.html : > > > Warning: The pseudo-random generators of this module should not be > used for security purposes. For security or cryptographic uses, see the > secrets module > > https://docs.python.org/3/library/secrets.html#module-secrets > > PEP 506 – Adding A Secrets Module To The Standard Library > https://peps.python.org/pep-0506/#alternatives > https://github.com/python/peps/blob/main/pep-0506.txt > > PEP 12: new PEP template: > https://github.com/python/peps/blob/main/pep-0012/pep-NNNN.rst > > Pseudorandom number generator > Cryptographic PRNGs > > https://en.wikipedia.org/wiki/Pseudorandom_number_generator#Cryptographic_PR... > > Random number generator attack > Defenses > https://en.wikipedia.org/wiki/Random_number_generator_attack#Defenses > > > /? CSPRNG > https://www.google.com/search?q=CSPRNG > > From "THE LINUX CSPRNG IS NOW GOOD!" > https://words.filippo.io/dispatches/linux-csprng/ : > > > [ get random() is from OpenBSD and LibreSSL ] > > > Performance and ChaCha20 > > Some people would say they needed a userspace CSPRNG for PERFORMANCE. > I never really believed most of them, but to be fair Linux was using a > kinda slow SHA-1 extractor back then. However, since Linux 4.8 (2016) the > default getrandom(2) source is a fast ChaCha20-based CSPRNG, with separate > pools per NUMA node to avoid contention. (Just ignore the rude comments in > the code about applications not running their own CSPRNG, this is still > Linux after all.) > > > > There's even a neat trick XOR'ing some of the CSPRGN output back into > the ChaCha20 state to prevent an attacker from recovering any
> from before the time of compromise. > > > > Some of these improvements came along thanks to the Wireguard work by > Jason A. Donenfeld > > "Problems emerge for a unified /dev/*random" (2022) > https://lwn.net/Articles/889452/ > > From > https://www.redhat.com/en/blog/understanding-red-hat-enterprise-linux-random... > : > > """ > How does the kernel initialize its CSPRNG? > The kernel has an “entropy pool,” a place where unpredictable input > observed by the kernel is mixed and stored. That pool serves as a seed to > the internal CSPRNG, and until some threshold of estimated entropy is > reached initially, it is considered uninitialized. > > Let’s now see how the kernel initializes its entropy pool. > > 1. After the kernel takes control on power-on, it starts filling its > entropy pool by mixing interrupt timing and other unpredictable input. > > 2. The kernel gives control to systemd. > > 3. Next, systemd starts and initializes itself. > > 4. Systemd, optionally, loads kernel modules which will improve
> kernel's entropy gathering process on a virtual machine (e.g., virtio-rng). > > 5. Systemd loads the rngd.service which will gather additional input > entropy obtained via a random generator exposed by hardware (e.g., the x86 > RDRAND instruction or similar) and jitter entropy1; this entropy is fed > back into the kernel to initialize its entropy pool, typically in a matter > of milliseconds. > > After the last step, the kernel has its entropy pool initialized, and > any systemd services started can take advantage of the kernel’s random > generator. > > Note that the virtio-rng kernel module loading in step (3), is an > optional step which improves entropy gathering in a virtual machine by > using the host's random generator to initialize the guest systems in KVM. > The rngd.service loading at the final step (4) is what ensures
> kernel entropy pools are initialized on every scenario, and furthermore it > continues mixing additional data in the kernel pool during system runtime. > """ > > https://github.com/nhorman/rng-tools/blob/master/fips.c : > > ```c > /* fips.c -- Performs FIPS 140-1/140-2 RNG tests > ``` > > /? FIPS 140-1/140-2 RNG tests > https://www.google.com/search?q=FIPS+140-1%2F140-2+RNG+tests > > /? CMVP "cprng" > https://www.google.com/search?q=CMVP+%22cprng%22 > https://csrc.nist.gov/publications/detail/fips/140/3/final > > https://www.google.com/search?q=rng+tests > > - https://www.johndcook.com/blog/rng-testing/ : > > > We test RNGs using the standard test suites: PractRand, TestU01 > (BigCrush), DIEHARD(ER), NIST SP 800-22. > > Randomness tests: > > https://en.wikipedia.org/wiki/Randomness_test#Notable_software_implementatio... > : > - https://en.wikipedia.org/wiki/Diehard_tests > - https://en.wikipedia.org/wiki/TestU01 > - /? NIST 800-22 https://www.google.com/search?q=nist+800-22 > > /? nist 800-22 site:github.com > https://www.google.com/search?q=nist+800-22+site%3Agithub.com > > - > https://github.com/google/paranoid_crypto/blob/main/docs/randomness_tests.md > > > From https://cryptography.io/en/latest/random-numbers/ > https://github.com/pyca/cryptography/blob/main/docs/random-numbers.rst > : > > > ```rst > Random number generation > ======================== > > When generating random data for use in cryptographic operations, such > as an > initialization vector for encryption in > :class:`~cryptography.hazmat.primitives.ciphers.modes.CBC` mode, you do > not > want to use the standard :mod:`random` module APIs. This is because > they do not > provide a cryptographically secure random number generator, which can > result in > major security issues depending on the algorithms in use. > > Therefore, it is our recommendation to `always use your operating > system's > provided random number generator`_, which is available as > :func:`os.urandom`. > For example, if you need 16 bytes of random data for an initialization > vector, > you can obtain them with: > > .. doctest:: > > >>> import os > >>> iv = os.urandom(16) > > This will use ``/dev/urandom`` on UNIX platforms, and > ``CryptGenRandom`` on > Windows. > > If you need your random number as an integer (for example, for > :meth:`~cryptography.x509.CertificateBuilder.serial_number`), you can > use > ``int.from_bytes`` to convert the result of ``os.urandom``: > > .. code-block:: pycon > > >>> serial = int.from_bytes(os.urandom(20), byteorder="big") > > In addition, the `Python standard library`_ includes the ``secrets`` > module, > which can be used for generating cryptographically secure random > numbers, with > specific helpers for text-based formats. > > .. _`always use your operating system's provided random number > generator`: > https://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ > .. _`Python standard library`: > https://docs.python.org/3/library/secrets.html > > ``` > > > On Mon, Nov 14, 2022, 10:57 AM Barry <barry@barrys-emacs.org> wrote: > >> >> >> > On 14 Nov 2022, at 14:31, James Johnson <jj126979@gmail.com> wrote: >> > >> > >> > I wrote the attached python (3) code to improve on existing
>> functions. I used the time module for one method, which resulted in >> disproportionate odd values, but agreeable means. >> > >> > I used the hashlib module for the second. It is evident that
rule that's pendulum like ECC.) source this, I seed past output the that the prng the
>> code is amateur, but the program might result in better PRN generation. >> > >> > The "app" lends itself to checking, using statistical tools (see >> comments.) >> >> Have you used any cryptographic tools to prove the quality of your >> PRNG? >> What results did you get? >> How does your PRNG compare to what python already has? >> >> Without that this analysis this will be unlikely to be considered as a >> candidate for python stdlib. >> >> Barry >> >> > >> > I remain a fan, >> > >> > James Johnson >> > <testrandom.py> >> > _______________________________________________ >> > Python-ideas mailing list -- python-ideas@python.org >> > To unsubscribe send an email to python-ideas-leave@python.org >> > https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> > Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/VENAT2... >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> >> _______________________________________________ >> Python-ideas mailing list -- python-ideas@python.org >> To unsubscribe send an email to python-ideas-leave@python.org >> https://mail.python.org/mailman3/lists/python-ideas.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-ideas@python.org/message/HWQV4A... >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- Truth causes consequences; consequences bring pain; pain exorcises guilt!
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ODL773... Code of Conduct: http://python.org/psf/codeofconduct/
Footnotes: [1] He composed several pseudo-random transformations, mostly linear congruential but also including at least one nonlinear one (middle square). The resulting sequence always converged to a short cycle within a handful of iterations.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4BWHIO... Code of Conduct: http://python.org/psf/codeofconduct/