Python-ideas

python-ideas@python.org

20 participants
5473 discussions

Nonuniform PRNG?
by David Lowry-Duda 07 Dec '22

07 Dec '22

Inspired by the recent thread on PRNG, I began to wonder: suppose that I had a pseudorandom number generator that attempted to generate a nonuniform distribution. Suppose for instance that it was to generate a 0 bit 2/3 of the time, and a 1 bit 1/3 of the time. How would one go about testing this PRNG against an idealized (similarly biased) PRNG? - DLD

4 4

Better (?) PRNG
by James Johnson 06 Dec '22

06 Dec '22

I wrote the attached python (3) code to improve on existing prng functions. I used the time module for one method, which resulted in disproportionate odd values, but agreeable means. I used the hashlib module for the second. It is evident that the code is amateur, but the program might result in better PRN generation. The "app" lends itself to checking, using statistical tools (see comments.) I remain a fan, James Johnson

8 23

Re: Better (?) PRNG - follow up
by Steven D'Aprano 06 Dec '22

06 Dec '22

On Tue, Dec 06, 2022 at 07:58:09PM -0500, David Mertz, Ph.D. wrote: > You have an error in the code you posted. You never use R2 after one > call to SystemRandom. Ah so I do, thanks for picking that up! James, see how *easy* it is for experts to notice bugs, at least some of them, in a short piece of code that gets right to the point? The more extraneous and irrelevant code you give, and that includes old dead comments, the more places for bugs to hide and the harder it is for people reading to follow the code and spot the error. Here is the corrected code (only one line needs to be fixed) and its output. Note that the conclusion doesn't change. ```python import random from statistics import mean, stdev R2 = random.SystemRandom() d2 = [R2.randint(1, 9) for i in range(100000)] triples = 0 # Count the number of triples. for i in range(len(d2)-3): a, b, c = d2[i:i+3] if a == b == c: triples += 1 print("mean =", mean(d2)) print("stdev =", stdev(d2)) print("runs of three:", triples) ``` And the output: ``` mean = 4.9922 stdev = 2.5837929328526306 runs of three: 1204 ``` -- Steve

1 0

Re: Better (?) PRNG - follow up
by James Johnson 06 Dec '22

06 Dec '22

I came back to this thread looking for the list of randomness tests, and I keep missing them somehow. The reason for my follow up is this: The random number function in Python has important uses far beyond my personal concerns, and random.randint(0,9) is supposed to be the Mersenne Twister. I used the attached code to generate the attached file. It’s randint(0,9) for n=600, keyed back in by hand, instead of automatically from the script: My question/concern is this: does this dataset look like it came from the Mersenne Twister? In 600 iterations, it generated 8,8,8; 0,0,0; 1,1,1 and 5,5,5, with more than 1% (6) occurrences where the same number repeated. I didn't do the +1/-1 test, but the data is in a (python) list that should be easily cut and pasted into a legit datafile. This leads me to question whether the MTPRNG rubber is meeting the Link Mint Mate 23.x.x road. "Catastrophizing," if randint isn't correct in the random library, what about the more crucial PRNG's? I suppose that feeding the random function back without keyboard intervention would yield similar results - it's not my data entry either: I printed the number suggested and the number entered for every instance, before the report. This is why my magic eight ball program had problems, not the design of Mersenne Twister. It may be amatuer as h*ll, but doesn't it point something? James (I thank you for so articulately showing that my hash isn't a PRNG.) On Wed, Nov 16, 2022 at 10:27 AM James Johnson <jj126979(a)gmail.com> wrote: > These are very many reasons that Python PRNG is indeed adequate. > > Maybe documentation for amateurs and newbies could be improved so that > they get the best results possible without knowing how much work it is. > > How to call the built in PRNG seems to change a good bit. > > As for the hack: I feel obligated to connect to the internet and do > research before I speak further. > > IF it DOES do 0-9 randomly, even if it is not useful for larger ranges, it > may serve by CONCATENATING base 10? > > I feel like it’s the punchline to a lame joke now, but thank you all > again. I will be reading for weeks. > > On Wed, Nov 16, 2022 at 9:36 AM Stephen J. Turnbull < > stephenjturnbull(a)gmail.com> wrote: > >> James Johnson writes: >> >> > The scholars here are referencing white papers and using hardware and >> > software interchangeably (?) I am not a randomization expert, and I do >> not >> > seek academic recognition. It's a novel hack is all, >> >> I'm no expert either, but hash functions and PRNGs have been around a >> long time. I'm sure this approach has been discovered independently. >> >> Besides the Mersenne twister that Python uses, POSIX systems like >> Linux, MacOS, and BSD provide a number of random number generation >> facilities, briefly documented in the man pages. BSD man pages for >> random and arc4random mention the Fortuna algorithm and the arc4 >> algorithm. At least the latter is similar to yours in that in "mixes >> in" additional entropy gradually as more random numbers are generated. >> >> For sources of "entropy" (random bits) the man page on /dev/random >> and /dev/urandom provide nice brief explanations. >> >> >> -- > Truth causes consequences; consequences bring pain; pain exorcises guilt! >

6 8

Re: Better (?) PRNG
by Chris Angelico 06 Dec '22

06 Dec '22

(I'm assuming that you sent this personally by mistake, and am redirecting back to the list. My apologies if you specifically didn't want this to be public.) On Tue, 15 Nov 2022 at 22:34, James Johnson <jj126979(a)gmail.com> wrote: > > It’s been a couple of years ago, but as I recall the duplicates seemed to be of two or three responses, not randomly distributed. > > I looked at my code, and I DID salt the hash at every update. > > At this point, my curiosity is engaged to know if this s/w solution is as good as others. I don’t have the training to test how often 9 follows 5, for example, but I am competitive enough to ask how it holds up against MTprng. I think it’s possibly very good, for s/w, and I’m emboldened to ask you to modify the code (it requires you data enter the numbers back to the machine, allowing time to pass;) to accumulate the results for 2 or 3 million, and see how it holds up. I don’t think the numbers track the bell curve on distribution . I speculate it’s more square. I suppose this is desirable in a PRNG? > I'll get you to do the first step of the modification. Turn your code into a module that has a randbelow() function which will return a random integer from 0 up to the provided argument. (This is equivalent to the standard library's random.randrange() function when given just one argument.) If you like, provide several of them, as randbelow1, randbelow2, etc. Post that code, and then I'll post a test harness that can do some analysis for you. ChrisA

4 10

Enhancing variable scope control
by Anony Mous 06 Dec '22

06 Dec '22

As it stands now, to create a local scope, you must define a function. However, there are many cases where various things can reasonably be done inline. Preparatory code is often done this way. Good coding practice is generally accepted to be that variables are local if at all possible. However, in direct, inline Python code, we're inherently creating variables with a global scope. We don't actually need a function for this kind of code; but a local scope would often be valuable (as we see with lambda.) Moving things off into a function can create a circumstance where we have to go looking for the function. When something is a "one-shot" as in global preparatory code, that's doing more work, and creating more indirection, than needs actually be done. But global operations have their own pitfalls, one of which is variable collisions. So Python sort of drives us to make "one-use" functions where lambdas are insufficient to the case in order to control locality. You can end up writing things like... def PrepOne(): for MyCount in range(0,10): pass PrepOne() ...which achieves the locality desired for variable MyCount, but is clunky. It also clutters the global namespace with prepOne, which has no good reason to exist. So I'm thinking of something along these lines: local: # <== or whatever syntax makes sense for MyCount in range(0,10): pass So in that example, the code is inline - runs as part of the global, sequential code execution - but the variable MyCount is local and goes "poof" once the local: scope is exited. This is the scoping equivalent of a pair of curly braces in c and c++. I would like to see it be nestable, as scopes are in c/c++: local: # <== or whatever syntax makes sense for MyCount in range(0,10): local: for YourCount in range(0,MyCount+1) pass There, MyCount is available in both scopes (the first local: is an enclosing scope) and YourCount is available only in the second local: scope. Or, local: could be very strict, and require further syntax to incorporate a previous scope, something like this: local: # <== or whatever syntax makes sense for MyCount in range(0,10): local: incorporate MyCount # <== or whatever syntax makes sense for YourCount in range(0,MyCount+1) pass Lastly, I suggest that the global keyword be used to expose global scope variables, just as it is within def functions: TheirCount = 10 local: # <== or whatever syntax makes sense global TheirCount for MyCount in range(0,TheirCount): local: for YourCount in range(0,TheirCount) pass This would also be useful within normal functions, and again allows the program flow to be linear rather than calling a function-within-a-function.

10 14

Re: Enhancing variable scope control
by Rob Cliffe 04 Dec '22

04 Dec '22

On 04/12/2022 17:08, Chris Angelico wrote: > On Mon, 5 Dec 2022 at 04:07, Rob Cliffe via Python-ideas > <python-ideas(a)python.org> wrote: >> >> >> On 30/11/2022 20:27, Anony Mous wrote: >>> Danceswithmice wrote: >>> >>> >>> >>> The idea is that YOU write "local:", and the interpreter, without you >>> ever seeing it, promotes that into a hidden function with a hidden >>> name and a hidden call. >>> >>> --- >>> >>> p.f.moore wrote: >>> >>>> That would make "return" in the local scope exit the scope, not the >>> enclosing function. Which is almost certainly not what people would expect >>> from a "local scope" statement. >>> >>> Hence my remark about return and yield requiring attention. These >>> would be disallowed in the context of "local:" UNLESS there is an >>> enclosing scope of "def function():", so there's no issue of what they >>> do if they are simply mainlined, and (I think) a well defined result >>> if they are not. >>> >> What about getting an untrapped exception? Again, your suggestion would >> mean that only the local scope is exited. Again, this would be unexpected. > Why? Wouldn't an uncaught exception bubble up until it is caught? > > ChrisA You're right, I wasn't thinking clearly. Sorry for the noise. Rob Cliffe > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BJHRJ… > Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Re: Enhancing variable scope control
by Rob Cliffe 04 Dec '22

04 Dec '22

On 30/11/2022 20:27, Anony Mous wrote: > Danceswithmice wrote: > > > > The idea is that YOU write "local:", and the interpreter, without you > ever seeing it, promotes that into a hidden function with a hidden > name and a hidden call. > > --- > > p.f.moore wrote: > > > That would make "return" in the local scope exit the scope, not the > enclosing function. Which is almost certainly not what people would expect > from a "local scope" statement. > > Hence my remark about return and yield requiring attention. These > would be disallowed in the context of "local:" UNLESS there is an > enclosing scope of "def function():", so there's no issue of what they > do if they are simply mainlined, and (I think) a well defined result > if they are not. > What about getting an untrapped exception? Again, your suggestion would mean that only the local scope is exited. Again, this would be unexpected. Best wishes Rob Cliffe

2 1

Re: Enhancing variable scope control
by Anony Mous 02 Dec '22

02 Dec '22

These objections -- such as they are -- are all applicable to every instance of "I wrote a function" or even "I named a variable" in any particular namespace: not just within imports, either. Anywhere. Global context. Sub functions. etc. Collisions, etc... not a big deal. Change a name. It'll still work the same. Nothing to it. No matter what the context is in Python. If you were talking about using a compiled library without the source code (and here come the back hats), depending on the linking and permissions, you could probably *still* change the name(s) in the library. But we're talking about an interpreted scripting language. We have the source code for everything. Or should. There is no mystery to be had. You want your own TestDottedQuad? You fear collisions? Fine. Write it as MyTest23DottedQuad87qxy or whatever. Be really random about it if it assuages your concerns. Or just rename your new instance, or the one extending the library (remember,it's an addition, if it's there, something added it. So you *can* change it if that's your call.) if you can't stand to write a new name for yours. Either way, bingo. Your functionality is present in your project. The other functionality is present as well. Both can be used with your strings. Choose. Enjoy your choice. You want to write it the hard way, as a loose function that can't handle class objects like strings? You can do that too. Something immutable is broken or not functional for your purposes? Lift the code, fix it, give it your own name, and use it. As an extension or directly. Whatever floats your boat. There. Done. Easily. Or... like these non-Python Ruby people that keep mysteriously polluting the argument - just tell the team "we're not going to do that." Again, problem solved. You do *not* have to cripple a language to resolve this in any particular direction. But wait.... you told them and they won't/can't comply? That's not a language issue. That's a "your team needs different staffing" issue. And mind you, this is all in the context of "big projects." Doing something yourself? Not even worth a handwave. Write what you need, test it, secure it, make it work, move on. There are myriad ways to get naming issues into compliance with whatever metric you want. Regardless of circumstance. This "we're going to save you from yourself" mentality serves to coddle weak and/or lazy programmers while crippling the rest. The entire idea that these naming issues (and that's all they are) cannot be coped with absolutely trivially is both nonsensical and insupportable. For instance: Obviously if you were to say "extend library with X", and X was already present (remember, these are immutable), such an attempt would fail. Bang. Exception. The very first time you tried it. Therefore, you're already directly on the path of "going to need a non-colliding name to use my particular function." See how easy that was? It's absurd to claim that this is any kind of meaningful obstacle. If it happens, you just fix it. Like any other programming issue. This is what we do. Are you thinking "but what if it wasn't tested so no one saw an exception?" In that case, again, we have a staffing issue. Not a language issue. As for the "big project" argument put forth earlier, that is also invalid. In any big project, if the devs aren't outright newbs anyway, people are given responsibility for specific functionality, and if random person X wants to mod existing function Y, they submit a suggestion, it goes through the appropriate process and gets accepted (possibly altered, such as... the name), or not. Pull requests, etc. Whatever the process is. But should there not be that (obvious and common these days) process, Any function in the project is subject to name collisions, any use of any library with name X / function X is subject to collisions, etc. And how do we get out of such things? if library A and library B have the same names, or prehaps function X and function Y, we change one of them. Now you have your multiple names and/or namespaces and/or names within namespaces. Likewise, if a function is added (at any level) and there is a name collision, we can change that too. We can notify our crew and say "globally edit 'TestDottedQuad' to 'MyTest23DottedQuad87qxy', and by the way, we're having subs for lunch." Trivial. If your crew isn't up for such things... well. That's a different problem entirely. I appreciate that you have strong feelings about this, but you (neither of the respondents here) have not presented any good arguments. Not even one. That doesn't mean there aren't supporters of your positions - of course there are - but every level of computing history is replete with bad ideas supported by many, and good ideas trampled on by just as many. Perceptrons/Minsky. Scrums. Clumsy typing systems. Chiclet keyboards. Freeform formatting. "Comment every line of code." Segment registers (and oh boy did that one cost us progress!) etc., etc., etc. > I concur... Concurring with one another doesn't ensure that you're right. It may also mean you're both wrong in similar or identical ways. To reiterate: I've already implemented this functionality through the auspices of a preprocessor, which also means, no monkey patching. It works very well, does exactly what I wanted, and improved my particular circumstance significantly. And then we have: > I find that implausible. Okay. However, since I know the truth of it, I'll proceed to not worry in the least that your thought process ended up there.

5 4

Re: Enhancing variable scope control
by Anony Mous 01 Dec '22

01 Dec '22

> This is very, very deliberately a design goal of Python. I suspect we're likely not talking about this same "this." Or if we are, I'd love to hear a justification -- any justification -- against what I'm talking about, because to date, I've never run into one. Many have tried, too. :) I'm talking about adding new functionality. NOT modifying existing functions. NOT replacing existing functions. NOT modifying existing, standard class internal variables or states. So If I add TestDottedQuad() to str, and once there it can't be replaced in the running instance, and you were to, unknowingly, end up using my extended string class, tell me where the harm is. No existing str functionality is modified. Anything you ever used will continue to work just as you expect, because it has not been modified in any way. If you don't know that TestDottedQuad() is there, it's essentially non-existent for you. If you do, and you use it... well, that's what it's there for, isn't it? In any case, I wanted that ability because I was doing a lot of interesting things to strings, and str was doing a stellar job of making that more difficult. I made it work WITHOUT monkey patching in any way, and also in a completely invisible fashion to anyone who has to parse my actual .py files, so I'm good. Preprocessors are very useful for such things. I get the extended syntax; everyone else gets standard (much more clumsy) python. > Most large Ruby projects have adopted code standards to explicitly prohibit the behavior you want, because the dangers greatly outweigh the benefits... Again, explain a danger to a nominal class user that arises due to adding NEW, immutable in the current instance, functions/functionality to a class that do not alter the existing base functionality. ANY danger. I'm very interested. In the context of Python, please. If you start talking Ruby, I'm unable to parse. --Ben

3 2

Jump to page:

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas