Re: [Python-ideas] Shuffled
So, why can't you call random.shuffle(all_tests) if you want to run your tests in random order?
I dont randomize test order. People should stop assuming that they know better. I need to randomize some arguments for one particular test and I cannot call shuffle between tests. Its a continous list of declarative tests. Needs to be shuffled(). https://github.com/construct/construct/blob/master/tests/test_all.py See? No way to put imperative code between tests.
sample(container, len(container))
That wont work because I would have to type the expression that is used as argument twice in a test. I need shuffled. Enough said.
I'll reiterate that I don't have a use case for this myself...
I dont have a use case for half of what the std library offers. Or for type annotations. Asynchronous comprehesions, what is that? Do you see me rejecting those?
(sample having default of entire list size)
That would work but would not be pretty. shuffled() is self explanatory and has a nice ring to it. Randomized list is not a sample by definition. pozdrawiam, Arkadiusz Bulski
That wont work because I would have to type the expression that is used as argument twice in a test.
(lambda data: random.sample(data, len(data)))(container) That lambda is actually your "shuffled"... 2016-09-08 6:34 GMT-03:00 Arek Bulski <arek.bulski@gmail.com>:
-- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap)
I also can't really get from "Arek has a need for this one line definition" to "it should be in the standard library". Sure, 'shuffled()' is a more obvious spelling than the slightly odd lambda. But I've yet to see the reason that alias can't just be defined at the top of the test suite. On Sep 8, 2016 3:14 AM, "Danilo J. S. Bellini" <danilo.bellini@gmail.com> wrote:
On 8 September 2016 at 10:34, Arek Bulski <arek.bulski@gmail.com> wrote:
That wont work because I would have to type the expression that is used as argument twice in a test. I need shuffled. Enough said.
You've probably spent way more time debating shuffled here than you would have needed to add a shuffled function to your test file. Your justification for the amount of time you've spent here, and that you've persuaded others to spend (and are further asking to be spent on maintenance) *has* to be more significant than "I needed shuffled() for this one specific case". But it's hard to understand what the *general* use cases you've (presumably) identified are - everyone responding has been saying "but I don't personally have a use for this". So in order to warrant the addition of a function to the stdlib, you need to explain what benefits justify the cost - and you simply haven't done that. The one case you've stated could have been solved in 2 minutes by writing a one-off function. And no-one has said "now that you mention it, I find the problem you had comes up a lot", so we're still at the position where it's a specialised use. The cost is non-trivial. Implementation is a relatively small cost, design is slightly higher (see for example Nick's point that this should be a method on the random instance as well as a module-level function), documentation and tests need to be added, 3rd party training material needs to be updated, bug reports need to be managed and resolved, etc. The benefit? I'm not clear, beyond you not needing to write a custom function in your test suite, and a couple of people who weren't aware they could use PYTHONSTARTUP to add their own personal functions being less frustrated by how random.shuffle behaves. And some theoretical "people might find this useful" comments that no-one has backed up with code samples. Nick contributed some suggestions in terms of benefits around teachability, and the improved design noted above. But the point here is that you shouldn't expect others to justify your proposal for you - you need to justify it yourself, or be prepared to accept "we don't see the benefit" as a response. Paul
This thread is just going around and around in circles, so I will make one final response and then I'm going to bow out. On Thu, Sep 08, 2016 at 11:34:57AM +0200, Arek Bulski wrote:
Okay, okay. Suppose you convince us. You complain and stamp your feet and demand and insist and repeat "I need shuffled" ten thousand times until you wear down everyone and they give in. With the 3.6 feature freeze just a few days away, this new code is going into 3.7, which probably won't reach a stable version until March or April 2018. So what happens between now and then? I see that your package Construct supports Python versions 2.6, 2.7, 3.3, 3.4 and 3.5. Are you ready to abandon ALL of those and support only 3.7 and higher? If not, then you can't even use the std lib implementation of shuffled() **because it won't exist**. At best you can test for a std lib implementation, falling back to your own when it doesn't exist. So you have to write your own implementation regardless of what the std lib does. You cannot rely on the std lib version until you've dropped support for Python 3.6, which you haven't even got support for yet! If you support just *three* releases, the current release plus the two previous, you won't be able to rely on the std lib shuffled() until Python 3.9 comes out, which will probably be in 2021. Maybe you do need shuffled(). But you can't use the one in the std lib until (probably) 2021 or later. What are you going to do for the next five years? - Do without. Then you don't really need it. - Write your own. Then you don't need it to be in the std lib. Adding new functions to the std lib now is an investment for future code, not a necessity for current, existing code. If you need this shuffled function now, you know how to write it. Harsh, but true. -- Steve
[Arek Bulski <arek.bulski@gmail.com>]
I dont randomize test order. People should stop assuming that they know better.
They should, but, to be fair, you've left them little choice because you've been so telegraphic about why you think this is a compelling idea. Others are trying to fill in the blanks, and can only guess.
I'm convinced you want to do that. Fine. The question is whether this is a _common enough_ need to justify all the ongoing costs of building it into a standard library. But in the only actual code you posted a link to, you don't even do it yourself. That just _enhances_ the perception that it's rarely needed. A question: do you know why `product()` isn't built in? Like product([2, 3, 4]) == 24? `sum()` is built in. So are `any()` and `all()` (other kinds of specialized reduction operations). Why does Python pick on multiplication as "unworthy" of the same kind of shortcut? The objections to product() were much the same as the ones being raised against shuffled(): sure, it would be useful at times in some contexts, but it's _rarely_ needed. It's also easy to write your own in the seemingly rare cases it is useful.
https://github.com/construct/construct/blob/master/tests/test_all.py See? No way to put imperative code between tests.
But there's also no example of anything in that code that requires shuffling. "But shuffled() isn't in the std lib!" isn't relevant: if you really _needed_ shuffling in test_all.py, you already know how to write such function, so if you really needed it you would have done so already. "Use cases" are about actual needs, not speculating about "maybe nice to have some day".
sample(container, len(container))
That wont work because I would have to type the expression that is used as argument twice in a test. I need shuffled. Enough said.
shuffled = lambda container: random.sample(container, len(container)) Now you have _a_ way to supply a `shuffled()` function (although I'd strongly recommend writing it in an obvious way building on random.shuffle()). But to justify becoming part of the standard distribution, it's not enough that _you_ "need" it: lots of people have to find it attractive. That's the same bar `sum()`, `any()`, `all()`, `sorted()` and `reversed()` (for examples) had to leap over. They were all popular ideas at the times they were proposed, and they indeed went on to become widely used in all kinds of code. I understand that you're proposing to make `shuffled()` a `Random` method instead of a builtin function, but that doesn't lower the bar much. I expect most people would say there's already "too much" rarely used stuff in Random.
I do not. Do you reject `product()`? You should ;-) But it's not about just you, and there's nothing personal about it. The examples you listed have many strong advocates, and "many" and "strong" both played roles in their acceptance. In contrast, the constituency for "shuffled()" appears small.
(sample having default of entire list size)
That would work but would not be pretty. shuffled() is self explanatory and has a nice ring to it. Randomized list is not a sample by definition.
sample() in this context uses a different spelling of the same algorithm shuffle() uses, but wasn't _intended_ for sorting. Using sample() for this is "a trick", and I wouldn't use it either. But I would use: def shuffled(xs): xs = list(xs) random.shuffle(xs) return xs I don't care that there's an implicit dependence on a shared Random instance, and neither will 99+% of other users. The few who bother with creating their own Random instances can write their own `shuffled()` to use them instead. Assuming the intersection of two small sets isn't empty ;-)
That wont work because I would have to type the expression that is used as argument twice in a test.
(lambda data: random.sample(data, len(data)))(container) That lambda is actually your "shuffled"... 2016-09-08 6:34 GMT-03:00 Arek Bulski <arek.bulski@gmail.com>:
-- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap)
I also can't really get from "Arek has a need for this one line definition" to "it should be in the standard library". Sure, 'shuffled()' is a more obvious spelling than the slightly odd lambda. But I've yet to see the reason that alias can't just be defined at the top of the test suite. On Sep 8, 2016 3:14 AM, "Danilo J. S. Bellini" <danilo.bellini@gmail.com> wrote:
On 8 September 2016 at 10:34, Arek Bulski <arek.bulski@gmail.com> wrote:
That wont work because I would have to type the expression that is used as argument twice in a test. I need shuffled. Enough said.
You've probably spent way more time debating shuffled here than you would have needed to add a shuffled function to your test file. Your justification for the amount of time you've spent here, and that you've persuaded others to spend (and are further asking to be spent on maintenance) *has* to be more significant than "I needed shuffled() for this one specific case". But it's hard to understand what the *general* use cases you've (presumably) identified are - everyone responding has been saying "but I don't personally have a use for this". So in order to warrant the addition of a function to the stdlib, you need to explain what benefits justify the cost - and you simply haven't done that. The one case you've stated could have been solved in 2 minutes by writing a one-off function. And no-one has said "now that you mention it, I find the problem you had comes up a lot", so we're still at the position where it's a specialised use. The cost is non-trivial. Implementation is a relatively small cost, design is slightly higher (see for example Nick's point that this should be a method on the random instance as well as a module-level function), documentation and tests need to be added, 3rd party training material needs to be updated, bug reports need to be managed and resolved, etc. The benefit? I'm not clear, beyond you not needing to write a custom function in your test suite, and a couple of people who weren't aware they could use PYTHONSTARTUP to add their own personal functions being less frustrated by how random.shuffle behaves. And some theoretical "people might find this useful" comments that no-one has backed up with code samples. Nick contributed some suggestions in terms of benefits around teachability, and the improved design noted above. But the point here is that you shouldn't expect others to justify your proposal for you - you need to justify it yourself, or be prepared to accept "we don't see the benefit" as a response. Paul
This thread is just going around and around in circles, so I will make one final response and then I'm going to bow out. On Thu, Sep 08, 2016 at 11:34:57AM +0200, Arek Bulski wrote:
Okay, okay. Suppose you convince us. You complain and stamp your feet and demand and insist and repeat "I need shuffled" ten thousand times until you wear down everyone and they give in. With the 3.6 feature freeze just a few days away, this new code is going into 3.7, which probably won't reach a stable version until March or April 2018. So what happens between now and then? I see that your package Construct supports Python versions 2.6, 2.7, 3.3, 3.4 and 3.5. Are you ready to abandon ALL of those and support only 3.7 and higher? If not, then you can't even use the std lib implementation of shuffled() **because it won't exist**. At best you can test for a std lib implementation, falling back to your own when it doesn't exist. So you have to write your own implementation regardless of what the std lib does. You cannot rely on the std lib version until you've dropped support for Python 3.6, which you haven't even got support for yet! If you support just *three* releases, the current release plus the two previous, you won't be able to rely on the std lib shuffled() until Python 3.9 comes out, which will probably be in 2021. Maybe you do need shuffled(). But you can't use the one in the std lib until (probably) 2021 or later. What are you going to do for the next five years? - Do without. Then you don't really need it. - Write your own. Then you don't need it to be in the std lib. Adding new functions to the std lib now is an investment for future code, not a necessity for current, existing code. If you need this shuffled function now, you know how to write it. Harsh, but true. -- Steve
[Arek Bulski <arek.bulski@gmail.com>]
I dont randomize test order. People should stop assuming that they know better.
They should, but, to be fair, you've left them little choice because you've been so telegraphic about why you think this is a compelling idea. Others are trying to fill in the blanks, and can only guess.
I'm convinced you want to do that. Fine. The question is whether this is a _common enough_ need to justify all the ongoing costs of building it into a standard library. But in the only actual code you posted a link to, you don't even do it yourself. That just _enhances_ the perception that it's rarely needed. A question: do you know why `product()` isn't built in? Like product([2, 3, 4]) == 24? `sum()` is built in. So are `any()` and `all()` (other kinds of specialized reduction operations). Why does Python pick on multiplication as "unworthy" of the same kind of shortcut? The objections to product() were much the same as the ones being raised against shuffled(): sure, it would be useful at times in some contexts, but it's _rarely_ needed. It's also easy to write your own in the seemingly rare cases it is useful.
https://github.com/construct/construct/blob/master/tests/test_all.py See? No way to put imperative code between tests.
But there's also no example of anything in that code that requires shuffling. "But shuffled() isn't in the std lib!" isn't relevant: if you really _needed_ shuffling in test_all.py, you already know how to write such function, so if you really needed it you would have done so already. "Use cases" are about actual needs, not speculating about "maybe nice to have some day".
sample(container, len(container))
That wont work because I would have to type the expression that is used as argument twice in a test. I need shuffled. Enough said.
shuffled = lambda container: random.sample(container, len(container)) Now you have _a_ way to supply a `shuffled()` function (although I'd strongly recommend writing it in an obvious way building on random.shuffle()). But to justify becoming part of the standard distribution, it's not enough that _you_ "need" it: lots of people have to find it attractive. That's the same bar `sum()`, `any()`, `all()`, `sorted()` and `reversed()` (for examples) had to leap over. They were all popular ideas at the times they were proposed, and they indeed went on to become widely used in all kinds of code. I understand that you're proposing to make `shuffled()` a `Random` method instead of a builtin function, but that doesn't lower the bar much. I expect most people would say there's already "too much" rarely used stuff in Random.
I do not. Do you reject `product()`? You should ;-) But it's not about just you, and there's nothing personal about it. The examples you listed have many strong advocates, and "many" and "strong" both played roles in their acceptance. In contrast, the constituency for "shuffled()" appears small.
(sample having default of entire list size)
That would work but would not be pretty. shuffled() is self explanatory and has a nice ring to it. Randomized list is not a sample by definition.
sample() in this context uses a different spelling of the same algorithm shuffle() uses, but wasn't _intended_ for sorting. Using sample() for this is "a trick", and I wouldn't use it either. But I would use: def shuffled(xs): xs = list(xs) random.shuffle(xs) return xs I don't care that there's an implicit dependence on a shared Random instance, and neither will 99+% of other users. The few who bother with creating their own Random instances can write their own `shuffled()` to use them instead. Assuming the intersection of two small sets isn't empty ;-)
participants (7)
-
Alexander Belopolsky -
Arek Bulski -
Danilo J. S. Bellini -
David Mertz -
Paul Moore -
Steven D'Aprano -
Tim Peters