Pass a function as the argument "step" of range()
In languages such as Javascript, the incrementation of a for loop counter can be done by an operation, for instance : for(i=1; i<N; i*=2) would iterate on the powers of 2 lesser than N. To achieve the same thing in Python we currently can't use range() because it increments by an integer (the argument "step"). An option is to build a generator like : def gen(N): i = 1 while i<=N: yield i i *= 2 then we can iterate on gen(N). My proposal is that besides an integer, range() would accept a function as the "step" argument, taking the current counter as its argument and returning the new counter value. Here is a basic pure-Python implementation : import operator class Range: def __init__(self, start, stop, incrementor): self.start, self.stop = start, stop self.incrementor = incrementor # Function to compare current counter and stop value : <= or >= self.comp = operator.ge if self.stop>self.start else operator.le self.counter = None def __iter__(self): return self def __next__(self): if self.counter is None: self.counter = self.start else: self.counter = self.incrementor(self.counter) if self.comp(self.counter, self.stop): raise StopIteration return self.counter Iterating on the powers of 2 below N would be done by : for i in Range(1, N, lambda x:x*2) I haven't seen this discussed before, but I may not have searched enough. Any opinions ?
Pierre Quentel <pierre.quentel@gmail.com> writes:
Generators can be defined in expressions, of course:: ((x * 2) for x in range(n)) So the full function definition above is misleading for this example. Your single example defines the ‘step’ function in-line as a lambda expression::
Iterating on the powers of 2 below N would be done by :
for i in Range(1, N, lambda x:x*2)
So why not define the generator as an expression:: for i in ((x * 2) for x in range(n)): That seems quite clear given existing syntax. Your proposal goes further than that and requires ‘range’ itself to accept a function argument where it currently expects an integer. But your example demonstrates, to me, that it wouldn't improve the code. Do you have some real-world code that would be materially improved by the change you're proposing? -- \ “I don't know anything about music. In my line you don't have | `\ to.” —Elvis Aaron Presley (1935–1977) | _o__) | Ben Finney
On Jul 1, 2015 23:56, "Ben Finney" <ben+python@benfinney.id.au> wrote:
I believe the original example was actually for i in ((2 ** (x + 1) for x in range(int(log2(n)))): or similar... which is clearly making some sort of argument about clarity but I'm not sure what. This isn't going to work for range() anyway though AFAICT because range isn't an iterator, it's an iterable that offers O(1) membership tests. I could see an argument for putting something along these lines in itertools. itertools.orbit, maybe. But I've never felt an urgent need for such a thing myself. -n
LOn Jul 2, 2015, at 00:12, Nathaniel Smith <njs@pobox.com> wrote:
You can already do this with accumulate; you just have to write lambda x, _: x*2. Of course it doesn't include the built-in bounds, but I don't think you'd want that anyway. With accumulate, you can bound on the domain by passing range instead of count for the input, bound on the range with takewhile, or generate an infinite iterator, or anything else you think might be useful. Or one more of the various combinations of things you can trivially build out of these pieces might be useful as a recipe ("irange"?) and/or in the third-party more-iterools.
2015-07-02 9:32 GMT+02:00 Andrew Barnert via Python-ideas < python-ideas@python.org>:
I am not saying that you can't find other ways to get the same result, just that using a function (usually a lambda) is easier to code and to understand. The proposal would bring to Python one of the few options where iteration is more simple in Java or Javascript than with Python - I had the idea from this discussion on reddit : https://www.reddit.com/r/Python/comments/3bj5dh/for_loop_with_iteratively_do...
Pierre Quentel <pierre.quentel@gmail.com> writes:
That's not something I can accept in the abstract. Can you please find and present some existing real-world Python code that you are confident would be improved by the changes you're proposing? So far, the only example you have presented is both contrived (no harm in that, but also not compelling) and does not demonstrate your point. -- \ “You can't have everything; where would you put it?” —Steven | `\ Wright | _o__) | Ben Finney
On Jul 2, 2015, at 03:17, Pierre Quentel <pierre.quentel@gmail.com> wrote:
I don't understand how using a function is easier to code and understand than using a function. Or how passing it to range is any simpler than passing it to accumulate, or to a recipe function built on top of accumulate.
2015-07-03 5:20 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
With the proposed addition to raise, the list of powers of 2 lower than 100 would be : list(range(1, 100, lambda x:x*2)) How do you code the same with accumulate ? I tried, but I'm stuck with "stop when the element is >= 100"
On Jul 2, 2015, at 23:28, Pierre Quentel <pierre.quentel@gmail.com> wrote:
A genexpr, a generator function, or a takewhile call. I already explained how you could write an "irange" function in two lines out of count, accumulate, and takewhile (along with a variety of other useful things). I also suggested that if this isn't obvious enough, it could be a handy recipe in the docs and/or a useful addition to the third-party more-itertools package. So, given that recipe, you'd write it as: list(irange(1, 100, lambda x:x*2)) There's no need to add a new itertools.orbit (with a custom C implementation), much less to change range into something that's sometimes a Sequence and sometimes not, when a two-line recipe (that's also an instructive sample) does it just as well.
2015-07-03 12:17 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
Well, you still haven't given an example with accumulate, have you ? I expected that. There is a good answer in the reddit thread : from itertools import count, takewhile list(takewhile(lambda n: n <= 100, (2**n for n in count()))) My point here is that even with this simple example, it's not clear even for people who know itertools well to remember which function to use.
On Jul 3, 2015, at 03:23, Pierre Quentel <pierre.quentel@gmail.com> wrote:
Well, you still haven't given an example with accumulate, have you ?
If you seriously can't figure out how to put my accumulate(count(1), lambda n, _: n*2) and the takewhile from the reddit example together, then that's a good argument for making it a recipe. But it's still not a good argument for breaking the range type.
On 2 July 2015 at 17:12, Nathaniel Smith <njs@pobox.com> wrote:
This isn't going to work for range() anyway though AFAICT because range isn't an iterator, it's an iterable that offers O(1) membership tests.
Right, Python 3's range() is best thought of as a memory-efficient tuple, rather than as an iterator like Python 2's xrange(). As far as the original request goes, srisyadasti's answer to the Reddit thread highlights one reasonable answer: encapsulating the non-standard iteration logic in a reusable generator, rather than making it easier to define non-standard logic inline. That feature wasn't copied from C into Python for loops, so there's no reason to copy it from Java either. In addition to writing a custom generator, or nesting a generator expression, it's also fairly straightforward to address the OP's request by way of map() and changing the expression of the limiting factor (to be the final input value rather than the final output value): def calc_value(x): return 2 ** (x + 1) for i in map(calc_value, range(10)): ... Depending on the problem being solved, "calc_value" could hopefully be given a more self-documenting name. Given the kinds of options available, the appropriate design is likely to come down to the desired "unit of reusability". * iteration pattern reusable as a whole? Write a custom generator * derivation of iteration value from loop index reusable? Write a derivation function and use map * one-shot operation? Use an inline generator expression or a break inside the loop Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2015-07-02 16:02 GMT+02:00 Nick Coghlan <ncoghlan@gmail.com>:
Again, this does not address the original problem : it produces the first 10 squares of 2, not the squares of 2 lower than a "stop" value. The logic of range(start, stop, step) is to produce the integers starting at "start", incremented by "step", until the integer is >= "stop" (or <= stop if stop<start). In simple cases you can probably compute the number of iterations in advance (that's what some answers in the reddit discussion did with logarithms), but it won't be so easy for more complex series ; you can also test the stop condition in the generator (this is what I did in my contribution as "kervarker") ; but I think it would be better to stick to the logic of range() with a more general incrementation method.
On 07/02/2015 08:16 AM, Pierre Quentel wrote:
Again, this does not address the original problem : it produces the first 10 squares of 2, not the squares of 2 lower than a "stop" value.
The logic of range(start, stop, step) is to produce the integers starting at "start", incremented by "step", until the integer is >= "stop" (or <= stop if stop<start).
The other logic of range is to be able to say: some_value in range(start, stop, step) If step is an integer it is easy to calculate whether some_value is in the range; if step is a function, it becomes impossible to figure out without iterating through (possibly all) the values of range. -- ~Ethan~
2015-07-02 17:23 GMT+02:00 Ethan Furman <ethan@stoneleaf.us>:
It's true, but testing that an integer is a range is very rare : the pattern "if X in range(Y)" is only found once in all the Python 3.4 standard library (in Lib/test/test_genexps.py), and "assert X in range(Y)" nowhere, whereas "for X in range(Y)" is everywhere. So even if __contains__ must iterate on all the items if the argument of step() is a function, I don't see it as a big problem.
On Fri, Jul 3, 2015 at 1:53 AM, Pierre Quentel <pierre.quentel@gmail.com> wrote:
That proves that testing for membership of "range literals" (if I may call them that) is rare - which I would expect. What if the range object is created in one place, and probed in another? Harder to find, but possibly monkey-patching builtins.range to report on __contains__ and then running the Python test suite would show something up. ChrisA
On Jul 2, 2015, at 08:53, Pierre Quentel <pierre.quentel@gmail.com> wrote:
It's true, but testing that an integer is a range is very rare : the pattern "if X in range(Y)" is only found once in all the Python 3.4 standard library
Given that most of the stdlib predates Python 3.2, and modules are rarely rewritten to take advantage of new features just for the hell of it, this isn't very surprising, or very meaningful. Similarly, you'll find that most of the stdlib doesn't use yield from expressions, and many things that could be written in terms of singledispatch instead use type switching, and so on. This doesn't mean yield from or singledispatch are useless or rarely used.
2015-07-03 12:29 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
Then most of the stdlib also predates Python 1.5.2 (the version I started with) : Python 1.5.2 (#0, Apr 13 1999, 10:51:12) [MSC 32 bit (Intel)] on win32 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
4 in range(10) 1
On Jul 5, 2015, at 09:15, Pierre Quentel <pierre.quentel@gmail.com> wrote:
Well, a good chunk of the stdlib does predate 1.5.2, but not nearly as much as predates 3.2... At any rate, as I'm sure you know, that works in 1.5.2 because range returns a list. Try it with range(1000000000) and you may not be quite as happy with the result--but in 3.2+, it returns instantly, without using more than a few dozen bytes of memory.
On 6 July 2015 at 15:00, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
One kinda neat trick with Python 3 ranges is that you can actually work with computed ranges with a size that exceeds 2**64 (and hence can't be handled by len()):
Conveniently, this also means attempting to convert them to a concrete list fails immediately, rather than eating up all your memory before falling over. Those particular bounds are so large they exceed the range of even a C double:
Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 5, 2015, at 22:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
I didn't realize that worked; nifty. Also, I'm not sure why this message triggered this idea but... The OP's example, or any geometric sequence, or anything that can be analytically integrated into an invertible function, can actually provide all the same features as range, without that much code. And that seems like a perfect example for demonstrating how to build a collections.abc.Sequence that's not like a tuple. Maybe a "collections cookbook" would be a useful thing to have in the HOWTO docs? (The OrderedSet recipe could also be moved there from ActiveState; I can't think of any other ideas that aren't overkill like a binary tree as a sorted Mapping or something.)
On 02/07/2015 07:30, Pierre Quentel wrote:
-1 from me. I don't like the idea as it doesn't fit in with my concept of what range() is about. A step is fixed and that's it. Changing it so the output has variable increments is a recipe for confusion in my mind, especially for newbies. I suppose we could have uneven_range() with uneven_step but there must be millions of these implementations in existence in all sorts of applications and libraries with all sorts of names so why implement it in Python now? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
On Thu, Jul 02, 2015 at 08:30:53AM +0200, Pierre Quentel wrote:
Given how simple generators are in Python, that's all you need for the most part. If you find yourself needing to do the above many times, with different expressions, you can write a factory: def gen(start, end, func): def inner(): i = start while i < end: yield i i = func(i) return inner() for i in gen(1, 100, lambda x: 3**x - x**3): for j in gen(-1, 5, lambda x: x**2): for k in gen(1000, 20, lambda x: -(x**3)): pass
Why add this functionality to range? It has little to do with range, except that range happens to sometimes be used as the sequence in for-loops. range has nice simple and clean semantics, for something as uncommon as this request, I think a user-defined generator is fine. range is not a tool for generating arbitrary sequences. It is a tool for generating sequences with equal-spaced values. Let's not complicate it. -- Steven
On 7/2/2015 2:30 AM, Pierre Quentel wrote:
The idea of iterating by non-constant steps is valid. Others have given multiple options for doing so. The idea of stuffing this into range is not valid. It does not fit into what 3.x range actually is. The Range class above is a one-use iterator. This post and your counter-responses seem to miss what others have alluded to. 3.x range class is something quite different -- a reusable collections.abc.Sequence class, with a separate range_iterator class. The range class has the following sequence methods with efficient O(1) implementations: __contains__, __getitem__, __iter__, __len__, __reversed__, count, and index methods. Having such efficient methods is part of the design goal for 3.x range. Your proposal would breaks all of these except __iter__ (and make that slower too) in the sense that the replacements would require the O(n) calculation of list(self), whereas avoiding this is part of the purpose of range. While some of these methods are rarely used, __reversed__ is certainly not rare. People depend on the fact that the often easy to write reversed(up-range(...)) is equivalent in output *and speed* to the often harder to write iter(down-range(...). A trivial case is counting down from n to 0 for i in reversed(range(n+1): versus for i in range(n, -1, -1): People do not always get the latter correct. Now onsider a more general case, such as r = range(11, 44000000000, 1335) r1 = reversed(r) versus the equivalent r2 = iter(range(43999999346, 10, -1335)) 43999999346 is r[-1], which uses __getitem__. Using this is much easier than figuring out the following (which __reversed__ has built in). def reversed_start(start, stop, step): rem = (stop - start) % step return stop - (rem if rem else step) -- Terry Jan Reedy
@Steven, Mark The definition of range() in Python docs says : Python 2.7 : "This is a versatile function to create lists containing arithmetic progressions. It is most often used in for loops." Python 3.4 : "The range type represents an immutable sequence of numbers and is commonly used for looping a specific number of times in for loops." Both stress that range is most often used in a for loop (it doesn't "happens to sometimes be used" in for loops, and is rarely used for membership testing). Python 2.7 limited its definition to arithmetic progressions, but Python 3.4 has a more general definition (an immutable sequence of numbers). I really don't think that the proposal would change the general idea behind range : a suite of integers, where each item is built from the previous following a specific pattern, and stopping when a "stop" value is reached. @Terry If the argument "step" is an integer, all the algorithms used in the mentioned methods would remain the same, so performance would not be affected for existing code. If the argument is a function, you are right, the object returned can't support some of these methods, or with an excessive performance penalty ; it would support __iter__ and not much more. I agree that this is a blocking issue : as far as I know all Python built-in functions return objects of a given type, regardless of its arguments. Thank you all for your time. Pierre 2015-07-02 21:53 GMT+02:00 Terry Reedy <tjreedy@udel.edu>:
On 3 July 2015 at 06:20, Pierre Quentel <pierre.quentel@gmail.com> wrote:
Pierre, I *wrote* the Python 3 range docs. I know what they say. Functionality for generating an arbitrary numeric series isn't going into range(). Now, it may be that there's value in having a way to neatly express a potentially infinite mathematical series, and further value in having a way to terminate iteration of that series based on the values it produces. The key question you need to ask yourself is whether or not you can come up with a proposal that is easier to read than writing an appropriately named custom iterator for whatever iteration problem you need to solve, or using a generator expression with itertools.takewhile and itertools.count: from itertools import takewhile, count for i in takewhile((lambda i: i < N), (2**x for x in count())): ... Outside obfuscated code contests, there aren't any prizes for expressing an idea in the fewest characters possible, but there are plenty of rewards to be found in expressing ideas in such a way that future maintainers can understand not only what the code actually does, but what it was intended to do, and that the computer can also execute at an acceptable speed. Assuming you're able to come up with such a proposal, the second question would then be whether that solution even belongs in the standard library, let alone in the builtins. What are the real world problems that the construct solves that itertools doesn't already cover? Making it easier to translate homework assignments written to explore features of other programming languages rather than features of Python doesn't count.
There isn't a "general idea" behind Python 3's range type, there's a precise, formal definition. For starters, the contents are defined to meet a specific formula: =================== For a positive step, the contents of a range r are determined by the formula r[i] = start + step*i where i >= 0 and r[i] < stop. For a negative step, the contents of the range are still determined by the formula r[i] = start + step*i, but the constraints are i >= 0 and r[i] > stop. =================== If you're tempted to respond "we can change the formula to use an arbitrary element value calculation algorithm", we make some *very* specific performance and behavioural promises for range objects, like: =================== Range objects implement the collections.abc.Sequence ABC, and provide features such as containment tests, element index lookup, slicing and support for negative indices =================== Testing range objects for equality with == and != compares them as sequences. That is, two range objects are considered equal if they represent the same sequence of values. (Note that two range objects that compare equal might have different start, stop and step attributes, for example range(0) == range(2, 1, 3) or range(0, 3, 2) == range(0, 4, 2).) =================== Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2015-07-03 13:23 GMT+02:00 Nick Coghlan <ncoghlan@gmail.com>:
Nick, Thanks for taking the time to explain. I am conscious that my proposal is not well received (understatement), and I respect the opinion of all those who expressed concerns with it. For me Terry's objection is the most serious : with a function instead of a fixed increment, many current methods of range() can't be implemented, or with a serious performance penalty. This pretty much kills the discussion. Nevertheless I will try to answer your questions. The proposal is (was) to extend the incrementation algorithm used to produce items from range() : from an addition of a fixed step to the last item, to an arbitrary function on the last item. The best I can do is rewriting the first lines of the document of range() : ### class range(stop) class range(start, stop[, step]) The arguments start and stop to the range constructor must be integers (either built-in int or any object that implements the __index__ special method). If the start argument is omitted, it defaults to 0. The step argument can be an integer of a function. If it is omitted, it defaults to 1. If step is zero, ValueError is raised. If step is a positive integer, the contents of a range r are determined by the formula r[i] = start + step*i where i >= 0 and r[i] < stop. If step is a negative integer, the contents of the range are still determined by the formula r[i] = start + step*i, but the constraints are i
= 0 and r[i] > stop.
If step is a function, the contents of the range is determined by the formulas r[0] = start, r[i] = step(r[i-1]) where i >= 1. If stop > step, the iteration stops when r[i] >= stop ; if stop < start, when r[i] <= stop. ### The advantage over specific generators or functions in itertools is the generality of the construct. For the example with the sequence of powers of 2, I find that for i in range(1, N, lambda x:x*2): ... is more readable than : from itertools import takewhile, count for i in takewhile((lambda i: i < N), (2**x for x in count())): ... It is not because it is shorter : I hate obscure one-liners as much as anyone. It is for two main reasons : - it makes it clear that we start at 1, we stop at N, and the incrementation is done by multiplying the previous item by 2. - the second form requires mastering the functions in itertools, which is not the case of all Python developers - after all, itertools is a module, its functions are not built-in. Even those who do hesitate between count() and accumulate(). Moreover, the construct applies to any incrementing function ; with itertools, you need to translate the function using the appropriate function(s) in the module. Of course this doesn't solve any problem that can't be solved any other way. But for that matter, there is nothing comprehensions can do that can't be done with loops - not that I compare my proposal to comprehensions in any way, it's just to say that this argument is not an absolute killer. Once again thank you all for your time. Pierre
On Jul 3, 2015, at 12:37, Pierre Quentel <pierre.quentel@gmail.com> wrote:
- the second form requires mastering the functions in itertools, which is not the case of all Python developers - after all, itertools is a module, its functions are not built-in. Even those who do hesitate between count() and accumulate().
Nobody "hesitates" between count and accumulate. They do completely different things. And I think everyone who's answered you, and everyone who's read any of the answers, understands that. It's only because you described "powers of two" analytically in text, but "multiply the last value by two" iteratively in pseudocode, that there's a question of which one to use. That won't happen in any real-life cases. Of course people who haven't "mastered" itertools and aren't used to thinking in higher-level terms might not think of accumulate and takewhile here; they might instead write something like this: def iterate(func, start): while True: yield start start = func(start) def irange(start, stop, stepfunc): for value in iterate(stepfunc, start): if value >= stop: break yield value for powerof2 in irange(1, 1000, lambda n:n*2): print(powerof2) But so what? It's a couple lines longer and maybe a tiny bit slower (at least in CPython; I wouldn't be too surprised if it's actually faster in PyPy...), but it's perfectly readable, and almost certainly efficient enough. And it's abstracted into a pair of simple, reusable functions, which you can always micro-optimize later if that turns out to be necessary. People on places like Reddit or StackOverflow like to debate about what's the absolute best implementation for any idea, but if the naive implementation that a novice would come up with on his own is good enough, those debates aren't relevant except as a fun little challenge, or a way to explore different parts of the language; the good enough code is good enough as-is. So, this just falls into the "not every 3-line function needs to be in the stdlib" category.
Thank you Andrew. This would be a good argument for those who think that there's nothing you can do with itertools that can't be done in a more readable and as efficient way without it. Good argument, but that could be improved with the obvious (and more complete, you forgot the case stop < start) import operator def irange(value, stop, func): comp = operator.ge if stop>value else operator.le while True: if comp(value, stop): break yield value value = func(value) 2015-07-03 22:33 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
On 7/3/2015 7:23 AM, Nick Coghlan wrote:
I think deleting 'arithmetic' was a mistake. Would you mind changing 'immutable sequence' to 'immutable arithmetic sequence'? Also, I think 'numbers' should be narrowed to 'integers' (or whatever is actually accepted). The idea of allowing floats has been proposed at least once, probably more, and rejected. ...
There isn't a "general idea" behind Python 3's range type, there's a precise, formal definition.
'predictable finite increasing or decreasing arithmetic sequence of integers, efficiently implemented' Making step an arbitrary function removes all the adjectives except (maybe) 'of integers', leaving 'sequence of integers'. There are several ways to generate unrestricted or lightly restricted sequences.
For a 0 step, which properly is neither + nor -, a ValueError is raised. range() looks at the value of step to decide whether to raise or return. Something must look as the sign of step to decide whether stop is a max or min, and what comparison to use. Since the sign is constant, this determination need only be done once, though I do not know where or when it is currently done. Given that a function has neither a value nor sign, and each call can have not only a different value, but a different sign. A step function is a very messy fit to an api with a stop parameter whose meaning depends on the sign of step. For many sequences, one would want an explicit max or min or both. Range could have had number-of-items parameter instead of the max-or-min stop parameter. Indeed, this would be easier in some situations, and some languages slice with start and number rather than start and stop. But range is intentionally consistent with python slicing, which uses start, a max-or-min stop, and a + or - step. -- Terry Jan Reedy
On 4 Jul 2015 7:25 am, "Terry Reedy" <tjreedy@udel.edu> wrote:
numbers and
Sure, clarifications aren't a problem - getting "arithmetic progression" back in the docs somewhere will be useful to folks familiar with the mathematical terminology for how range works.
Unfortunately, we don't have a great word for "implements __index__", as "integer" at least arguably implies specifically "int".
Ah, I like that. the sign of step. For many sequences, one would want an explicit max or min or both. Yeah, I started trying to think of how to do this generically, and I think it would need to be considered in terms of upper and lower bounds, rather than a single stop value. That is, there'd be 6 characterising values for a general purpose computed sequence: * domain start * domain stop * domain step * range lower bound * range upper bound * item calculation However, you fundamentally can't make len(obj) an O(1) operation in that model, since you don't know the output range without calling the function. So the general purpose form could be an iterator like: def within_bounds(iterable, lower, upper): for x in iterable: if not lower <= x < upper: break yield x Positive & negative infinity would likely suffice as defaults for the lower & upper bounds. Cheers, Nick.
On Sat, Jul 4, 2015 at 10:17 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm not sure that actually matters - even if the parameters can be any objects that implement __index__, the range still represents a sequence of ints:
Saying "sequence of integers" seems fine to me. ChrisA
On Thu, Jul 02, 2015 at 10:20:16PM +0200, Pierre Quentel wrote:
You have misunderstood me. I'm not saying that range necessarily has many widespread and common uses outside of for-loops, but that for-loops only sometimes use range. Most loops iterate directly over the iterable, they don't use a range object at all. You started this thread with an example from Javascript. For loops in Javascript can be extremely general: js> for(var i=1,j=0,k=2; i < 100; j=2*i, k-=1, i+=j+k){print([i,j,k])} 1,0,2 4,2,1 12,8,0 35,24,-1 Why try to force all that generality into the range function? There are really two issues here: (1) Is there a problem with Python that it cannot easily or reasonable perform certain for-loops that Javascript makes easy? (2) Is modifying range() the right way to solve that problem? I don't think you have actually demonstrated the existence of a problem yet. True, Javascript gives you a nice, compact, readable syntax for some very general loops, but Python has its own way of doing those same loops which may not be quite as compact but are probably still quite acceptable. But even if we accept that Javascript for-loops are more powerful, more readable, and more flexible, and that Python lacks an acceptable way of performing certain for-loops that Javascript makes easy, changing range does not seem to be the right way to fix that lack. -- Steve
I like the idea, but this would be nicer in a language with implicit currying so that your example could be: for i in Range(1, N, (*2)): ... Or in Haskell: range' v a b f | a == b = [] | otherwise = f a:range v a+v b f range a b | a > b = range' -1 a b | a < b = range' 1 a b | a == b = [] Since Python doesn't have this, the generator forms others showed would likely end up more concise. On July 2, 2015 1:30:53 AM CDT, Pierre Quentel <pierre.quentel@gmail.com> wrote:
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
2015-07-02 8:30 GMT+02:00 Pierre Quentel <pierre.quentel@gmail.com>:
With the proposed Range class, here is an implementation of the Fibonacci sequence, limited to 2000 : previous = 0 def fibo(last): global previous _next, previous = previous+last, last return _next print(list(Range(1, 2000, fibo)))
On Fri, Jul 3, 2015 at 8:10 PM, Pierre Quentel <pierre.quentel@gmail.com> wrote:
The whole numbers start with (0, 1) too (pun intended), but you can ask for a range that starts part way into that sequence:
list(range(10,20)) [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
You can create "chunked ranges" by simply migrating your previous second argument into your new first argument, and picking a new second argument. Or you can handle an HTTP parameter "?page=4" by multiplying 4 by your chunk size and using that as your start, adding another chunk and making that your end. While it's fairly easy to ask for Fibonacci numbers up to 2000, it's rather harder to ask for only the ones 1000 and greater. Your range *must* start at the very beginning - a very good place to start, don't get me wrong, but it's a little restrictive if that's _all_ you can do. ChrisA
2015-07-03 12:23 GMT+02:00 Chris Angelico <rosuav@gmail.com>:
No, that's very easy, just rewrite the function : previous = 987 def fibo(last): global previous _next, previous = previous+last, last return _next print(list(Range(1597, 10000, fibo))) and erase the browser history ;-) More seriously, you can't produce this sequence without starting by the first 2 item (0, 1), no matter how you implement it
On Fri, Jul 3, 2015 at 5:30 PM, Pierre Quentel <pierre.quentel@gmail.com> wrote:
Without the proposed Range class, here's an equivalent that doesn't use global state: def fibo(top): a, b = 0, 1 while a < 2000: yield a a, b = a + b, a print(list(fibo(2000))) I'm not seeing this as an argument for a variable-step range func/class, especially since you need to use a global - or to construct a dedicated callable whose sole purpose is to maintain one integer of state. Generators are extremely expressive and flexible. ChrisA
2015-07-03 11:56 GMT+02:00 Chris Angelico <rosuav@gmail.com>:
Of course there are lots of ways to produce the Fibonacci sequence, and generators are perfect for this purpose. This was intended as an example of how to use the proposed range() with always the same logic : build a sequence of integers from a starting point, use a function to build the next item, and stop when a "stop" value is reached.
On 07/02/2015 02:30 AM, Pierre Quentel wrote:
I'm surprised no one mentioned this!?
It looks like map returns a map object which is a generator. You just need to use the power of 2 formula rather than accumulate it. That's actually more flexible solution as your range can start some place other than 1.
Cheers, Ron
On Jul 3, 2015, at 15:24, Ron Adam <ron3200@gmail.com> wrote:
Probably because this map call is equivalent to (2**x for x in range(1, 10)), which someone did mention, and is less readable. If you already have a function lying around that does what you want, passing it to map tends to be more readable than wrapping it in a function call expression with a meaningless argument name just so you can stick in a genexpr. But, by the same token, if you have an expression, and don't have a function lying around, using it in a genexpr tends to be more readable than wrapping it in a lambda expression with a meaningless parameter name just so you can pass it to map. Also, this has the same problem as many of the other proposed solutions, in that it assumes that you can transform the iterative n*2 into an analytic 2**n, and that you can work out the maximum domain value (10) in your head from the maximum range value (1000), and that both of those transformations will be obvious to the readers of the code. In this particular trivial case, that's true, but it's hard to imagine any real-life case where it would be.
On 07/03/2015 07:53 PM, Andrew Barnert via Python-ideas wrote:
I missed that one. But you are correct, it is equivalent.
Which was what I was thinking of. for i in map(fn, range(start, stop): ...
Agree.
Ah.. this is the part I missed. I would most likely rewite it as a while loop myself. Cheers, Ron
On 4 July 2015 at 00:53, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Also, this has the same problem as many of the other proposed solutions, in that it assumes that you can transform the iterative n*2 into an analytic 2**n, and that you can work out the maximum domain value (10) in your head from the maximum range value (1000), and that both of those transformations will be obvious to the readers of the code. In this particular trivial case, that's true, but it's hard to imagine any real-life case where it would be.
One thing that I have kept stumbling over when I've been reading this discussion is that I keep expecting there to be a "simple" (i.e., builtin, or in a relatively obvious module) way of generating repeated applications of a single-argument function: def iterate(fn, start): while True: yield start start = fn)start) ... and yet I can't find one. Am I missing it, or does it not exist? It's not hard to write, as shown above, so I'm not claiming it "needs to be a builtin" necessarily, it just seems like a useful building block. Paul
On 4 July 2015 at 20:56, Paul Moore <p.f.moore@gmail.com> wrote:
It's a particular way of using itertools.accumulate: def infinite_series(fn, start): def series_step(last_output, __): return fn(last_output) return accumulate(repeat(start), series_step) Due to the closure, that can be generalised to tracking multiple past values fairly easily (e.g. using deque), and the use of repeat() means you can adapt it to define finite iterators as well. This is covered in the accumulate docs (https://docs.python.org/3/library/itertools.html#itertools.accumulate) under the name "recurrence relation", but it may be worth extracting the infinite series example as a new recipe in the recipes section. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 4 July 2015 at 14:48, Nick Coghlan <ncoghlan@gmail.com> wrote:
Ah, thanks. I hadn't thought of using accumulate with a function that ignored its second argument. Although the above seems noticeably less obvious than my hand-coded def iterate(fn, start): while True: yield start start = fn(start) Unless the accumulate version is substantially faster (I haven't tested), I can't imagine ever wanting to use it in preference. Paul
Pierre Quentel <pierre.quentel@gmail.com> writes:
Generators can be defined in expressions, of course:: ((x * 2) for x in range(n)) So the full function definition above is misleading for this example. Your single example defines the ‘step’ function in-line as a lambda expression::
Iterating on the powers of 2 below N would be done by :
for i in Range(1, N, lambda x:x*2)
So why not define the generator as an expression:: for i in ((x * 2) for x in range(n)): That seems quite clear given existing syntax. Your proposal goes further than that and requires ‘range’ itself to accept a function argument where it currently expects an integer. But your example demonstrates, to me, that it wouldn't improve the code. Do you have some real-world code that would be materially improved by the change you're proposing? -- \ “I don't know anything about music. In my line you don't have | `\ to.” —Elvis Aaron Presley (1935–1977) | _o__) | Ben Finney
On Jul 1, 2015 23:56, "Ben Finney" <ben+python@benfinney.id.au> wrote:
I believe the original example was actually for i in ((2 ** (x + 1) for x in range(int(log2(n)))): or similar... which is clearly making some sort of argument about clarity but I'm not sure what. This isn't going to work for range() anyway though AFAICT because range isn't an iterator, it's an iterable that offers O(1) membership tests. I could see an argument for putting something along these lines in itertools. itertools.orbit, maybe. But I've never felt an urgent need for such a thing myself. -n
LOn Jul 2, 2015, at 00:12, Nathaniel Smith <njs@pobox.com> wrote:
You can already do this with accumulate; you just have to write lambda x, _: x*2. Of course it doesn't include the built-in bounds, but I don't think you'd want that anyway. With accumulate, you can bound on the domain by passing range instead of count for the input, bound on the range with takewhile, or generate an infinite iterator, or anything else you think might be useful. Or one more of the various combinations of things you can trivially build out of these pieces might be useful as a recipe ("irange"?) and/or in the third-party more-iterools.
2015-07-02 9:32 GMT+02:00 Andrew Barnert via Python-ideas < python-ideas@python.org>:
I am not saying that you can't find other ways to get the same result, just that using a function (usually a lambda) is easier to code and to understand. The proposal would bring to Python one of the few options where iteration is more simple in Java or Javascript than with Python - I had the idea from this discussion on reddit : https://www.reddit.com/r/Python/comments/3bj5dh/for_loop_with_iteratively_do...
Pierre Quentel <pierre.quentel@gmail.com> writes:
That's not something I can accept in the abstract. Can you please find and present some existing real-world Python code that you are confident would be improved by the changes you're proposing? So far, the only example you have presented is both contrived (no harm in that, but also not compelling) and does not demonstrate your point. -- \ “You can't have everything; where would you put it?” —Steven | `\ Wright | _o__) | Ben Finney
On Jul 2, 2015, at 03:17, Pierre Quentel <pierre.quentel@gmail.com> wrote:
I don't understand how using a function is easier to code and understand than using a function. Or how passing it to range is any simpler than passing it to accumulate, or to a recipe function built on top of accumulate.
2015-07-03 5:20 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
With the proposed addition to raise, the list of powers of 2 lower than 100 would be : list(range(1, 100, lambda x:x*2)) How do you code the same with accumulate ? I tried, but I'm stuck with "stop when the element is >= 100"
On Jul 2, 2015, at 23:28, Pierre Quentel <pierre.quentel@gmail.com> wrote:
A genexpr, a generator function, or a takewhile call. I already explained how you could write an "irange" function in two lines out of count, accumulate, and takewhile (along with a variety of other useful things). I also suggested that if this isn't obvious enough, it could be a handy recipe in the docs and/or a useful addition to the third-party more-itertools package. So, given that recipe, you'd write it as: list(irange(1, 100, lambda x:x*2)) There's no need to add a new itertools.orbit (with a custom C implementation), much less to change range into something that's sometimes a Sequence and sometimes not, when a two-line recipe (that's also an instructive sample) does it just as well.
2015-07-03 12:17 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
Well, you still haven't given an example with accumulate, have you ? I expected that. There is a good answer in the reddit thread : from itertools import count, takewhile list(takewhile(lambda n: n <= 100, (2**n for n in count()))) My point here is that even with this simple example, it's not clear even for people who know itertools well to remember which function to use.
On Jul 3, 2015, at 03:23, Pierre Quentel <pierre.quentel@gmail.com> wrote:
Well, you still haven't given an example with accumulate, have you ?
If you seriously can't figure out how to put my accumulate(count(1), lambda n, _: n*2) and the takewhile from the reddit example together, then that's a good argument for making it a recipe. But it's still not a good argument for breaking the range type.
On 2 July 2015 at 17:12, Nathaniel Smith <njs@pobox.com> wrote:
This isn't going to work for range() anyway though AFAICT because range isn't an iterator, it's an iterable that offers O(1) membership tests.
Right, Python 3's range() is best thought of as a memory-efficient tuple, rather than as an iterator like Python 2's xrange(). As far as the original request goes, srisyadasti's answer to the Reddit thread highlights one reasonable answer: encapsulating the non-standard iteration logic in a reusable generator, rather than making it easier to define non-standard logic inline. That feature wasn't copied from C into Python for loops, so there's no reason to copy it from Java either. In addition to writing a custom generator, or nesting a generator expression, it's also fairly straightforward to address the OP's request by way of map() and changing the expression of the limiting factor (to be the final input value rather than the final output value): def calc_value(x): return 2 ** (x + 1) for i in map(calc_value, range(10)): ... Depending on the problem being solved, "calc_value" could hopefully be given a more self-documenting name. Given the kinds of options available, the appropriate design is likely to come down to the desired "unit of reusability". * iteration pattern reusable as a whole? Write a custom generator * derivation of iteration value from loop index reusable? Write a derivation function and use map * one-shot operation? Use an inline generator expression or a break inside the loop Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2015-07-02 16:02 GMT+02:00 Nick Coghlan <ncoghlan@gmail.com>:
Again, this does not address the original problem : it produces the first 10 squares of 2, not the squares of 2 lower than a "stop" value. The logic of range(start, stop, step) is to produce the integers starting at "start", incremented by "step", until the integer is >= "stop" (or <= stop if stop<start). In simple cases you can probably compute the number of iterations in advance (that's what some answers in the reddit discussion did with logarithms), but it won't be so easy for more complex series ; you can also test the stop condition in the generator (this is what I did in my contribution as "kervarker") ; but I think it would be better to stick to the logic of range() with a more general incrementation method.
On 07/02/2015 08:16 AM, Pierre Quentel wrote:
Again, this does not address the original problem : it produces the first 10 squares of 2, not the squares of 2 lower than a "stop" value.
The logic of range(start, stop, step) is to produce the integers starting at "start", incremented by "step", until the integer is >= "stop" (or <= stop if stop<start).
The other logic of range is to be able to say: some_value in range(start, stop, step) If step is an integer it is easy to calculate whether some_value is in the range; if step is a function, it becomes impossible to figure out without iterating through (possibly all) the values of range. -- ~Ethan~
2015-07-02 17:23 GMT+02:00 Ethan Furman <ethan@stoneleaf.us>:
It's true, but testing that an integer is a range is very rare : the pattern "if X in range(Y)" is only found once in all the Python 3.4 standard library (in Lib/test/test_genexps.py), and "assert X in range(Y)" nowhere, whereas "for X in range(Y)" is everywhere. So even if __contains__ must iterate on all the items if the argument of step() is a function, I don't see it as a big problem.
On Fri, Jul 3, 2015 at 1:53 AM, Pierre Quentel <pierre.quentel@gmail.com> wrote:
That proves that testing for membership of "range literals" (if I may call them that) is rare - which I would expect. What if the range object is created in one place, and probed in another? Harder to find, but possibly monkey-patching builtins.range to report on __contains__ and then running the Python test suite would show something up. ChrisA
On Jul 2, 2015, at 08:53, Pierre Quentel <pierre.quentel@gmail.com> wrote:
It's true, but testing that an integer is a range is very rare : the pattern "if X in range(Y)" is only found once in all the Python 3.4 standard library
Given that most of the stdlib predates Python 3.2, and modules are rarely rewritten to take advantage of new features just for the hell of it, this isn't very surprising, or very meaningful. Similarly, you'll find that most of the stdlib doesn't use yield from expressions, and many things that could be written in terms of singledispatch instead use type switching, and so on. This doesn't mean yield from or singledispatch are useless or rarely used.
2015-07-03 12:29 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
Then most of the stdlib also predates Python 1.5.2 (the version I started with) : Python 1.5.2 (#0, Apr 13 1999, 10:51:12) [MSC 32 bit (Intel)] on win32 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
4 in range(10) 1
On Jul 5, 2015, at 09:15, Pierre Quentel <pierre.quentel@gmail.com> wrote:
Well, a good chunk of the stdlib does predate 1.5.2, but not nearly as much as predates 3.2... At any rate, as I'm sure you know, that works in 1.5.2 because range returns a list. Try it with range(1000000000) and you may not be quite as happy with the result--but in 3.2+, it returns instantly, without using more than a few dozen bytes of memory.
On 6 July 2015 at 15:00, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
One kinda neat trick with Python 3 ranges is that you can actually work with computed ranges with a size that exceeds 2**64 (and hence can't be handled by len()):
Conveniently, this also means attempting to convert them to a concrete list fails immediately, rather than eating up all your memory before falling over. Those particular bounds are so large they exceed the range of even a C double:
Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 5, 2015, at 22:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
I didn't realize that worked; nifty. Also, I'm not sure why this message triggered this idea but... The OP's example, or any geometric sequence, or anything that can be analytically integrated into an invertible function, can actually provide all the same features as range, without that much code. And that seems like a perfect example for demonstrating how to build a collections.abc.Sequence that's not like a tuple. Maybe a "collections cookbook" would be a useful thing to have in the HOWTO docs? (The OrderedSet recipe could also be moved there from ActiveState; I can't think of any other ideas that aren't overkill like a binary tree as a sorted Mapping or something.)
On 02/07/2015 07:30, Pierre Quentel wrote:
-1 from me. I don't like the idea as it doesn't fit in with my concept of what range() is about. A step is fixed and that's it. Changing it so the output has variable increments is a recipe for confusion in my mind, especially for newbies. I suppose we could have uneven_range() with uneven_step but there must be millions of these implementations in existence in all sorts of applications and libraries with all sorts of names so why implement it in Python now? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
On Thu, Jul 02, 2015 at 08:30:53AM +0200, Pierre Quentel wrote:
Given how simple generators are in Python, that's all you need for the most part. If you find yourself needing to do the above many times, with different expressions, you can write a factory: def gen(start, end, func): def inner(): i = start while i < end: yield i i = func(i) return inner() for i in gen(1, 100, lambda x: 3**x - x**3): for j in gen(-1, 5, lambda x: x**2): for k in gen(1000, 20, lambda x: -(x**3)): pass
Why add this functionality to range? It has little to do with range, except that range happens to sometimes be used as the sequence in for-loops. range has nice simple and clean semantics, for something as uncommon as this request, I think a user-defined generator is fine. range is not a tool for generating arbitrary sequences. It is a tool for generating sequences with equal-spaced values. Let's not complicate it. -- Steven
On 7/2/2015 2:30 AM, Pierre Quentel wrote:
The idea of iterating by non-constant steps is valid. Others have given multiple options for doing so. The idea of stuffing this into range is not valid. It does not fit into what 3.x range actually is. The Range class above is a one-use iterator. This post and your counter-responses seem to miss what others have alluded to. 3.x range class is something quite different -- a reusable collections.abc.Sequence class, with a separate range_iterator class. The range class has the following sequence methods with efficient O(1) implementations: __contains__, __getitem__, __iter__, __len__, __reversed__, count, and index methods. Having such efficient methods is part of the design goal for 3.x range. Your proposal would breaks all of these except __iter__ (and make that slower too) in the sense that the replacements would require the O(n) calculation of list(self), whereas avoiding this is part of the purpose of range. While some of these methods are rarely used, __reversed__ is certainly not rare. People depend on the fact that the often easy to write reversed(up-range(...)) is equivalent in output *and speed* to the often harder to write iter(down-range(...). A trivial case is counting down from n to 0 for i in reversed(range(n+1): versus for i in range(n, -1, -1): People do not always get the latter correct. Now onsider a more general case, such as r = range(11, 44000000000, 1335) r1 = reversed(r) versus the equivalent r2 = iter(range(43999999346, 10, -1335)) 43999999346 is r[-1], which uses __getitem__. Using this is much easier than figuring out the following (which __reversed__ has built in). def reversed_start(start, stop, step): rem = (stop - start) % step return stop - (rem if rem else step) -- Terry Jan Reedy
@Steven, Mark The definition of range() in Python docs says : Python 2.7 : "This is a versatile function to create lists containing arithmetic progressions. It is most often used in for loops." Python 3.4 : "The range type represents an immutable sequence of numbers and is commonly used for looping a specific number of times in for loops." Both stress that range is most often used in a for loop (it doesn't "happens to sometimes be used" in for loops, and is rarely used for membership testing). Python 2.7 limited its definition to arithmetic progressions, but Python 3.4 has a more general definition (an immutable sequence of numbers). I really don't think that the proposal would change the general idea behind range : a suite of integers, where each item is built from the previous following a specific pattern, and stopping when a "stop" value is reached. @Terry If the argument "step" is an integer, all the algorithms used in the mentioned methods would remain the same, so performance would not be affected for existing code. If the argument is a function, you are right, the object returned can't support some of these methods, or with an excessive performance penalty ; it would support __iter__ and not much more. I agree that this is a blocking issue : as far as I know all Python built-in functions return objects of a given type, regardless of its arguments. Thank you all for your time. Pierre 2015-07-02 21:53 GMT+02:00 Terry Reedy <tjreedy@udel.edu>:
On 3 July 2015 at 06:20, Pierre Quentel <pierre.quentel@gmail.com> wrote:
Pierre, I *wrote* the Python 3 range docs. I know what they say. Functionality for generating an arbitrary numeric series isn't going into range(). Now, it may be that there's value in having a way to neatly express a potentially infinite mathematical series, and further value in having a way to terminate iteration of that series based on the values it produces. The key question you need to ask yourself is whether or not you can come up with a proposal that is easier to read than writing an appropriately named custom iterator for whatever iteration problem you need to solve, or using a generator expression with itertools.takewhile and itertools.count: from itertools import takewhile, count for i in takewhile((lambda i: i < N), (2**x for x in count())): ... Outside obfuscated code contests, there aren't any prizes for expressing an idea in the fewest characters possible, but there are plenty of rewards to be found in expressing ideas in such a way that future maintainers can understand not only what the code actually does, but what it was intended to do, and that the computer can also execute at an acceptable speed. Assuming you're able to come up with such a proposal, the second question would then be whether that solution even belongs in the standard library, let alone in the builtins. What are the real world problems that the construct solves that itertools doesn't already cover? Making it easier to translate homework assignments written to explore features of other programming languages rather than features of Python doesn't count.
There isn't a "general idea" behind Python 3's range type, there's a precise, formal definition. For starters, the contents are defined to meet a specific formula: =================== For a positive step, the contents of a range r are determined by the formula r[i] = start + step*i where i >= 0 and r[i] < stop. For a negative step, the contents of the range are still determined by the formula r[i] = start + step*i, but the constraints are i >= 0 and r[i] > stop. =================== If you're tempted to respond "we can change the formula to use an arbitrary element value calculation algorithm", we make some *very* specific performance and behavioural promises for range objects, like: =================== Range objects implement the collections.abc.Sequence ABC, and provide features such as containment tests, element index lookup, slicing and support for negative indices =================== Testing range objects for equality with == and != compares them as sequences. That is, two range objects are considered equal if they represent the same sequence of values. (Note that two range objects that compare equal might have different start, stop and step attributes, for example range(0) == range(2, 1, 3) or range(0, 3, 2) == range(0, 4, 2).) =================== Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
2015-07-03 13:23 GMT+02:00 Nick Coghlan <ncoghlan@gmail.com>:
Nick, Thanks for taking the time to explain. I am conscious that my proposal is not well received (understatement), and I respect the opinion of all those who expressed concerns with it. For me Terry's objection is the most serious : with a function instead of a fixed increment, many current methods of range() can't be implemented, or with a serious performance penalty. This pretty much kills the discussion. Nevertheless I will try to answer your questions. The proposal is (was) to extend the incrementation algorithm used to produce items from range() : from an addition of a fixed step to the last item, to an arbitrary function on the last item. The best I can do is rewriting the first lines of the document of range() : ### class range(stop) class range(start, stop[, step]) The arguments start and stop to the range constructor must be integers (either built-in int or any object that implements the __index__ special method). If the start argument is omitted, it defaults to 0. The step argument can be an integer of a function. If it is omitted, it defaults to 1. If step is zero, ValueError is raised. If step is a positive integer, the contents of a range r are determined by the formula r[i] = start + step*i where i >= 0 and r[i] < stop. If step is a negative integer, the contents of the range are still determined by the formula r[i] = start + step*i, but the constraints are i
= 0 and r[i] > stop.
If step is a function, the contents of the range is determined by the formulas r[0] = start, r[i] = step(r[i-1]) where i >= 1. If stop > step, the iteration stops when r[i] >= stop ; if stop < start, when r[i] <= stop. ### The advantage over specific generators or functions in itertools is the generality of the construct. For the example with the sequence of powers of 2, I find that for i in range(1, N, lambda x:x*2): ... is more readable than : from itertools import takewhile, count for i in takewhile((lambda i: i < N), (2**x for x in count())): ... It is not because it is shorter : I hate obscure one-liners as much as anyone. It is for two main reasons : - it makes it clear that we start at 1, we stop at N, and the incrementation is done by multiplying the previous item by 2. - the second form requires mastering the functions in itertools, which is not the case of all Python developers - after all, itertools is a module, its functions are not built-in. Even those who do hesitate between count() and accumulate(). Moreover, the construct applies to any incrementing function ; with itertools, you need to translate the function using the appropriate function(s) in the module. Of course this doesn't solve any problem that can't be solved any other way. But for that matter, there is nothing comprehensions can do that can't be done with loops - not that I compare my proposal to comprehensions in any way, it's just to say that this argument is not an absolute killer. Once again thank you all for your time. Pierre
On Jul 3, 2015, at 12:37, Pierre Quentel <pierre.quentel@gmail.com> wrote:
- the second form requires mastering the functions in itertools, which is not the case of all Python developers - after all, itertools is a module, its functions are not built-in. Even those who do hesitate between count() and accumulate().
Nobody "hesitates" between count and accumulate. They do completely different things. And I think everyone who's answered you, and everyone who's read any of the answers, understands that. It's only because you described "powers of two" analytically in text, but "multiply the last value by two" iteratively in pseudocode, that there's a question of which one to use. That won't happen in any real-life cases. Of course people who haven't "mastered" itertools and aren't used to thinking in higher-level terms might not think of accumulate and takewhile here; they might instead write something like this: def iterate(func, start): while True: yield start start = func(start) def irange(start, stop, stepfunc): for value in iterate(stepfunc, start): if value >= stop: break yield value for powerof2 in irange(1, 1000, lambda n:n*2): print(powerof2) But so what? It's a couple lines longer and maybe a tiny bit slower (at least in CPython; I wouldn't be too surprised if it's actually faster in PyPy...), but it's perfectly readable, and almost certainly efficient enough. And it's abstracted into a pair of simple, reusable functions, which you can always micro-optimize later if that turns out to be necessary. People on places like Reddit or StackOverflow like to debate about what's the absolute best implementation for any idea, but if the naive implementation that a novice would come up with on his own is good enough, those debates aren't relevant except as a fun little challenge, or a way to explore different parts of the language; the good enough code is good enough as-is. So, this just falls into the "not every 3-line function needs to be in the stdlib" category.
Thank you Andrew. This would be a good argument for those who think that there's nothing you can do with itertools that can't be done in a more readable and as efficient way without it. Good argument, but that could be improved with the obvious (and more complete, you forgot the case stop < start) import operator def irange(value, stop, func): comp = operator.ge if stop>value else operator.le while True: if comp(value, stop): break yield value value = func(value) 2015-07-03 22:33 GMT+02:00 Andrew Barnert <abarnert@yahoo.com>:
On 7/3/2015 7:23 AM, Nick Coghlan wrote:
I think deleting 'arithmetic' was a mistake. Would you mind changing 'immutable sequence' to 'immutable arithmetic sequence'? Also, I think 'numbers' should be narrowed to 'integers' (or whatever is actually accepted). The idea of allowing floats has been proposed at least once, probably more, and rejected. ...
There isn't a "general idea" behind Python 3's range type, there's a precise, formal definition.
'predictable finite increasing or decreasing arithmetic sequence of integers, efficiently implemented' Making step an arbitrary function removes all the adjectives except (maybe) 'of integers', leaving 'sequence of integers'. There are several ways to generate unrestricted or lightly restricted sequences.
For a 0 step, which properly is neither + nor -, a ValueError is raised. range() looks at the value of step to decide whether to raise or return. Something must look as the sign of step to decide whether stop is a max or min, and what comparison to use. Since the sign is constant, this determination need only be done once, though I do not know where or when it is currently done. Given that a function has neither a value nor sign, and each call can have not only a different value, but a different sign. A step function is a very messy fit to an api with a stop parameter whose meaning depends on the sign of step. For many sequences, one would want an explicit max or min or both. Range could have had number-of-items parameter instead of the max-or-min stop parameter. Indeed, this would be easier in some situations, and some languages slice with start and number rather than start and stop. But range is intentionally consistent with python slicing, which uses start, a max-or-min stop, and a + or - step. -- Terry Jan Reedy
On 4 Jul 2015 7:25 am, "Terry Reedy" <tjreedy@udel.edu> wrote:
numbers and
Sure, clarifications aren't a problem - getting "arithmetic progression" back in the docs somewhere will be useful to folks familiar with the mathematical terminology for how range works.
Unfortunately, we don't have a great word for "implements __index__", as "integer" at least arguably implies specifically "int".
Ah, I like that. the sign of step. For many sequences, one would want an explicit max or min or both. Yeah, I started trying to think of how to do this generically, and I think it would need to be considered in terms of upper and lower bounds, rather than a single stop value. That is, there'd be 6 characterising values for a general purpose computed sequence: * domain start * domain stop * domain step * range lower bound * range upper bound * item calculation However, you fundamentally can't make len(obj) an O(1) operation in that model, since you don't know the output range without calling the function. So the general purpose form could be an iterator like: def within_bounds(iterable, lower, upper): for x in iterable: if not lower <= x < upper: break yield x Positive & negative infinity would likely suffice as defaults for the lower & upper bounds. Cheers, Nick.
On Sat, Jul 4, 2015 at 10:17 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm not sure that actually matters - even if the parameters can be any objects that implement __index__, the range still represents a sequence of ints:
Saying "sequence of integers" seems fine to me. ChrisA
On Thu, Jul 02, 2015 at 10:20:16PM +0200, Pierre Quentel wrote:
You have misunderstood me. I'm not saying that range necessarily has many widespread and common uses outside of for-loops, but that for-loops only sometimes use range. Most loops iterate directly over the iterable, they don't use a range object at all. You started this thread with an example from Javascript. For loops in Javascript can be extremely general: js> for(var i=1,j=0,k=2; i < 100; j=2*i, k-=1, i+=j+k){print([i,j,k])} 1,0,2 4,2,1 12,8,0 35,24,-1 Why try to force all that generality into the range function? There are really two issues here: (1) Is there a problem with Python that it cannot easily or reasonable perform certain for-loops that Javascript makes easy? (2) Is modifying range() the right way to solve that problem? I don't think you have actually demonstrated the existence of a problem yet. True, Javascript gives you a nice, compact, readable syntax for some very general loops, but Python has its own way of doing those same loops which may not be quite as compact but are probably still quite acceptable. But even if we accept that Javascript for-loops are more powerful, more readable, and more flexible, and that Python lacks an acceptable way of performing certain for-loops that Javascript makes easy, changing range does not seem to be the right way to fix that lack. -- Steve
I like the idea, but this would be nicer in a language with implicit currying so that your example could be: for i in Range(1, N, (*2)): ... Or in Haskell: range' v a b f | a == b = [] | otherwise = f a:range v a+v b f range a b | a > b = range' -1 a b | a < b = range' 1 a b | a == b = [] Since Python doesn't have this, the generator forms others showed would likely end up more concise. On July 2, 2015 1:30:53 AM CDT, Pierre Quentel <pierre.quentel@gmail.com> wrote:
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
2015-07-02 8:30 GMT+02:00 Pierre Quentel <pierre.quentel@gmail.com>:
With the proposed Range class, here is an implementation of the Fibonacci sequence, limited to 2000 : previous = 0 def fibo(last): global previous _next, previous = previous+last, last return _next print(list(Range(1, 2000, fibo)))
On Fri, Jul 3, 2015 at 8:10 PM, Pierre Quentel <pierre.quentel@gmail.com> wrote:
The whole numbers start with (0, 1) too (pun intended), but you can ask for a range that starts part way into that sequence:
list(range(10,20)) [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
You can create "chunked ranges" by simply migrating your previous second argument into your new first argument, and picking a new second argument. Or you can handle an HTTP parameter "?page=4" by multiplying 4 by your chunk size and using that as your start, adding another chunk and making that your end. While it's fairly easy to ask for Fibonacci numbers up to 2000, it's rather harder to ask for only the ones 1000 and greater. Your range *must* start at the very beginning - a very good place to start, don't get me wrong, but it's a little restrictive if that's _all_ you can do. ChrisA
2015-07-03 12:23 GMT+02:00 Chris Angelico <rosuav@gmail.com>:
No, that's very easy, just rewrite the function : previous = 987 def fibo(last): global previous _next, previous = previous+last, last return _next print(list(Range(1597, 10000, fibo))) and erase the browser history ;-) More seriously, you can't produce this sequence without starting by the first 2 item (0, 1), no matter how you implement it
On Fri, Jul 3, 2015 at 5:30 PM, Pierre Quentel <pierre.quentel@gmail.com> wrote:
Without the proposed Range class, here's an equivalent that doesn't use global state: def fibo(top): a, b = 0, 1 while a < 2000: yield a a, b = a + b, a print(list(fibo(2000))) I'm not seeing this as an argument for a variable-step range func/class, especially since you need to use a global - or to construct a dedicated callable whose sole purpose is to maintain one integer of state. Generators are extremely expressive and flexible. ChrisA
2015-07-03 11:56 GMT+02:00 Chris Angelico <rosuav@gmail.com>:
Of course there are lots of ways to produce the Fibonacci sequence, and generators are perfect for this purpose. This was intended as an example of how to use the proposed range() with always the same logic : build a sequence of integers from a starting point, use a function to build the next item, and stop when a "stop" value is reached.
On 07/02/2015 02:30 AM, Pierre Quentel wrote:
I'm surprised no one mentioned this!?
It looks like map returns a map object which is a generator. You just need to use the power of 2 formula rather than accumulate it. That's actually more flexible solution as your range can start some place other than 1.
Cheers, Ron
On Jul 3, 2015, at 15:24, Ron Adam <ron3200@gmail.com> wrote:
Probably because this map call is equivalent to (2**x for x in range(1, 10)), which someone did mention, and is less readable. If you already have a function lying around that does what you want, passing it to map tends to be more readable than wrapping it in a function call expression with a meaningless argument name just so you can stick in a genexpr. But, by the same token, if you have an expression, and don't have a function lying around, using it in a genexpr tends to be more readable than wrapping it in a lambda expression with a meaningless parameter name just so you can pass it to map. Also, this has the same problem as many of the other proposed solutions, in that it assumes that you can transform the iterative n*2 into an analytic 2**n, and that you can work out the maximum domain value (10) in your head from the maximum range value (1000), and that both of those transformations will be obvious to the readers of the code. In this particular trivial case, that's true, but it's hard to imagine any real-life case where it would be.
On 07/03/2015 07:53 PM, Andrew Barnert via Python-ideas wrote:
I missed that one. But you are correct, it is equivalent.
Which was what I was thinking of. for i in map(fn, range(start, stop): ...
Agree.
Ah.. this is the part I missed. I would most likely rewite it as a while loop myself. Cheers, Ron
On 4 July 2015 at 00:53, Andrew Barnert via Python-ideas <python-ideas@python.org> wrote:
Also, this has the same problem as many of the other proposed solutions, in that it assumes that you can transform the iterative n*2 into an analytic 2**n, and that you can work out the maximum domain value (10) in your head from the maximum range value (1000), and that both of those transformations will be obvious to the readers of the code. In this particular trivial case, that's true, but it's hard to imagine any real-life case where it would be.
One thing that I have kept stumbling over when I've been reading this discussion is that I keep expecting there to be a "simple" (i.e., builtin, or in a relatively obvious module) way of generating repeated applications of a single-argument function: def iterate(fn, start): while True: yield start start = fn)start) ... and yet I can't find one. Am I missing it, or does it not exist? It's not hard to write, as shown above, so I'm not claiming it "needs to be a builtin" necessarily, it just seems like a useful building block. Paul
On 4 July 2015 at 20:56, Paul Moore <p.f.moore@gmail.com> wrote:
It's a particular way of using itertools.accumulate: def infinite_series(fn, start): def series_step(last_output, __): return fn(last_output) return accumulate(repeat(start), series_step) Due to the closure, that can be generalised to tracking multiple past values fairly easily (e.g. using deque), and the use of repeat() means you can adapt it to define finite iterators as well. This is covered in the accumulate docs (https://docs.python.org/3/library/itertools.html#itertools.accumulate) under the name "recurrence relation", but it may be worth extracting the infinite series example as a new recipe in the recipes section. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 4 July 2015 at 14:48, Nick Coghlan <ncoghlan@gmail.com> wrote:
Ah, thanks. I hadn't thought of using accumulate with a function that ignored its second argument. Although the above seems noticeably less obvious than my hand-coded def iterate(fn, start): while True: yield start start = fn(start) Unless the accumulate version is substantially faster (I haven't tested), I can't imagine ever wanting to use it in preference. Paul
participants (14)
-
Andrew Barnert -
Ben Finney -
Chris Angelico -
Ethan Furman -
Mark Lawrence -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Peter Otten -
Pierre Quentel -
Ron Adam -
Ryan Gonzalez -
Steven D'Aprano -
Terry Reedy