Datetime operation in itertools count
I wanted to generate all the dates between two date ranges for which I was using count function in the itertools and to my surprise count function doesn't support datetime operation. For example
Why is count function only limited to numbers shouldn't we make it generic that it should support operation like datetime where addition between the objects is possible. Would like to hear thoughts from you people.
On Wed, 9 Feb 2022 at 01:00, Aman Pandey <amanpandey5319@gmail.com> wrote:
This sounds like a perfect place for a custom generator function. def timecount(start, **delta): delta = datetime.timedelta(**delta) while True: yield start start += delta timecount(datetime.date.today(), days=1) The default itertools.count is highly optimized for its job, but when you need flexibility, it's easy enough to handroll something to your needs. ChrisA
On Tue, 8 Feb 2022 at 14:00, Aman Pandey <amanpandey5319@gmail.com> wrote:
Because it's implemented in C for speed, and limiting it to numbers makes it both easier to implement (in C) and faster.
Would like to hear thoughts from you people.
start = date.datetime.today() (start + datetime.timedelta(n) for n in count()) does exactly the same as your code does, so it's not *that* hard to get the functionality you want already. Paul
I get your point. I checked the code which is in C and implement the solution which was something similar to yours. Should we have some function like that in the datetime module that can generate date, and time as well between two ranges? This looks like a feature to me that can be helpful. Yesterday I found Pandas Library has this feature <https://pandas.pydata.org/docs/reference/api/pandas.date_range.html>. What do you think? On Tue, Feb 8, 2022 at 9:23 PM Paul Moore <p.f.moore@gmail.com> wrote:
I think a datetime.range object could be useful. Perhaps someone can write one and then see if the core devs would accept it in the stdlib. It would be na interesting exercise in any case :-) -CHB On Tue, Feb 8, 2022 at 10:38 PM Aman Pandey <amanpandey5319@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I can give it a try, but I have never done it before and anything which I should be knowing beforehand because it will be my first time. I would like to listen from other fellow developers what they think about this? On Wed, Feb 9, 2022 at 1:09 PM Christopher Barker <pythonchb@gmail.com> wrote:
As I said, it’s an interesting exercise in any case. I have no idea if the core devs would be interested in adding it. There have been a couple recipes on this thread for an iterator, but I envision something like the range() object — it’s a lazy Sequence, not just an iterator. I think that would be pretty cool. -CHB On Wed, Feb 9, 2022 at 3:45 AM Aman Pandey <amanpandey5319@gmail.com> wrote:
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Christopher Barker writes:
I would be strongly against an extension to range() itself (I think that was the original proposal), and I think we haven't even started to mix the batter let alone bake the cake for a range method on datetime (or time or date). I see the following issues that would need to be discussed: 1. clock monotonicity 2. "daylight saving" monotonicity 3. handling of "naive" datetimes (probably easy, but that's what everybody says about everything) 4. The "anywhere on earth" (AOE) "timezone" and other similar not- exactly-what-you-teach-1st-graders-about-time practices. There's probably more, but given the above, but I don't see a good reason to say anything but """what's wrong with dt = datetime(*startdate) while dt <= datetime(*enddate): do_something_with(dt) dt += timedelta(*deltaspec) """ yet. It's possible that all of the above are straightforward to deal with (2 would be messy at best with an indexable dateteime_range, though), but the monotonic clock PEP was quite a bit of work. They need to be addressed though.
Hello Stephen, I think you misunderstood it. we are not talking about modifying the existing built-in function range rather adding a new function in datetime module that will generate the date and time between two periods. if we add a function like that in the datetime module the code will be exactly like the one you gave. Christopher was mentioning that he visualizes that function in the datetime module as something similar in syntax and execution like the built-in range function. what do you think if we add some function like that in datetime module wouldn't it be helpful? On Thu, Feb 10, 2022 at 4:56 PM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
On Thu, 10 Feb 2022 at 15:21, Aman Pandey <amanpandey5319@gmail.com> wrote:
I think something like this might occasionally be helpful. However, for simple examples, it's pretty easy to write your own, sufficiently so that people would expect more of a stdlib version. And understanding the edge cases, correctly handling them, and designing an API which makes the function easy to use while still having the power to cover everyone's use cases, is a *lot* harder. It's that part of the problem that is receiving the pushback here. To give a concrete example, consider a range with a start of 1:30am on a particular day, and a step of 1 day. If the range extends over a DST change, then there's one day in the range where either there are no valid 1:30am times, or there are *two*, equally valid 1:30am times (depending on whether the clock is "going forwards" or "going backwards"). What should the stdlib function do in this case? How should the function deal with the fact that people can legitimately expect different behaviours (for some applications "raise an exception" could be the only safe option, for others, "use the nearest valid time" might be)? Having flags that pick between different behaviours scales badly, and is typically not a good design. For your own code, you can write the behaviour that works best for you, and for your application. For the stdlib, the behaviour has to address the needs of *everyone*. My feeling is that the complexity of a "does the right thing for everyone" stdlib implementation would outweigh any benefit gained from not having to write your own short function, tailored to your particular needs. Paul
Since a started this:-) 1) the complexities are exactly why this *might* be a candidate for the stdlib. Those are the things that someone is likely to get wrong, or not think of, when they write something quick themselves, and only lightly test. 2) it doesn’t have to be everything for everyone- it has to be many things for many (most) people. 3) DST is a nightmare, plain and simple. The datetime module was useful for years even before it gained the ability to deal with it at all.[*] An educational thing to explore, in any case. -CHB [*] now that I think about it, I would probably only do this without DST — including DST would require a DST database, which the stdlib doesn’t include. On Thu, Feb 10, 2022 at 7:48 AM Paul Moore <p.f.moore@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Feb 10, 2022 at 8:57 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
Don't forget leap seconds! :-)
TAI only :-) -CHB On Thu, Feb 10, 2022, 11:45 AM Christopher Barker <pythonchb@gmail.com>
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, 10 Feb 2022 at 16:44, Christopher Barker <pythonchb@gmail.com> wrote:
Since a started this:-)
1) the complexities are exactly why this *might* be a candidate for the stdlib. Those are the things that someone is likely to get wrong, or not think of, when they write something quick themselves, and only lightly test.
Fair. But no-one has come up with a use case to justify doing the work to handle those complexities. All the discussion has tended to be around how it "wouldn't be too hard".
2) it doesn’t have to be everything for everyone- it has to be many things for many (most) people.
Agreed, but that's mostly just a matter of degree. At a minimum, if it deliberately omits certain situations, those need to be documented. Otherwise there's a maintenance cost from people asking for such support and debating whether it's "implied" by the existence of the function in the stdlib.
3) DST is a nightmare, plain and simple. The datetime module was useful for years even before it gained the ability to deal with it at all.[*] [*] now that I think about it, I would probably only do this without DST — including DST would require a DST database, which the stdlib doesn’t include.
https://docs.python.org/3/library/zoneinfo.html Having a stdlib function that handles range-type sequences for dates, but does not handle the various timezone-aware datetime types is bound to cause some level of confusion. Having said that, maybe it's simple to do this right. That's my point, someone needs to do the work to think all this through. I'm not *against* the idea, I just don't think it's worth the effort given the (fairly trivial) benefits that have been claimed so far. In particular, I have no use for it personally - the one-liners already posted in this thread would be perfectly fine for any use I can imagine, and I see no reason to add such one-liners to the stdlib. Paul
On Thu, Feb 10, 2022 at 9:02 AM Paul Moore <p.f.moore@gmail.com> wrote:
My Bad -- I forgot about that -- I know there was a lot of debate about including something that was going to be out of data at some point :( -- so yes, a daterange could allow DST :-) But at the end of the day, it is never going to be better than horrible :-(
Exactly.
agreed. But we seem to be getting a mixed message here: 1) it's too coplex with too many variable that have to be thought through and won't satisfy everyone anyway. and 2) It's just a simple one-liner Those are kind of incompatible ideas ;-) This feels to me not so different then math.isclose() -- not that complex, but complex enough that it's good to put a well thought through implementation in the stdlib. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, 10 Feb 2022 at 17:11, Christopher Barker <pythonchb@gmail.com> wrote:
Indeed, so which is being proposed here? My answer is the same in both cases (I don't think it's worth adding to the stdlib) but which justification I need to express depends on which option is being proposed :-)
This feels to me not so different then math.isclose() -- not that complex, but complex enough that it's good to put a well thought through implementation in the stdlib.
Maybe. It still needs someone to get down to details and propose something specific. And I'll still be between -0.5 and -1 on it, but you don't need to convince me personally, you just need some level of support from *some* of the core devs, and/or broad community support sufficient to persuade at least one core dev to merge a PR implementing the idea. Paul
On Thu, Feb 10, 2022 at 9:25 AM Paul Moore <p.f.moore@gmail.com> wrote:
Exactly—I think it would be an interesting exercise— so worth doing even if no one finds it compelling. I’d likely work in it myself if I didn’t have 100 other interesting things I don’t have time for :-) But I’m always happy to kibitz…. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Christopher Barker writes:
agreed. But we seem to be getting a mixed message here:
1) it's too coplex with too many variable that have to be thought through
s/too//g please. I haven't seen any explicit -1s yet, just "I'm not gonna do it myself for sure". @Aman Paul's discussion mirrors my own thought (or vice versa if you like, Paul -- you wrote it better than I could have).
and won't satisfy everyone anyway.
You've already admitted that.
No, they aren't incompatible at all, because the "it" that refers to to a stdlib patch is a singleton that has to do it all, and the "it" that refers to one-liners is a fairly large plurality of one offs that do one thing well.
In outline, yes, but math.isclose() is a single function with literally centuries of math thought behind it, and at least a century of numerical analysis. I don't think a projected indexable datetime.range object has quite the same literature backing up the discussion. So I want to see the discussion. Steve
On Wed, 9 Feb 2022 at 01:00, Aman Pandey <amanpandey5319@gmail.com> wrote:
This sounds like a perfect place for a custom generator function. def timecount(start, **delta): delta = datetime.timedelta(**delta) while True: yield start start += delta timecount(datetime.date.today(), days=1) The default itertools.count is highly optimized for its job, but when you need flexibility, it's easy enough to handroll something to your needs. ChrisA
On Tue, 8 Feb 2022 at 14:00, Aman Pandey <amanpandey5319@gmail.com> wrote:
Because it's implemented in C for speed, and limiting it to numbers makes it both easier to implement (in C) and faster.
Would like to hear thoughts from you people.
start = date.datetime.today() (start + datetime.timedelta(n) for n in count()) does exactly the same as your code does, so it's not *that* hard to get the functionality you want already. Paul
I get your point. I checked the code which is in C and implement the solution which was something similar to yours. Should we have some function like that in the datetime module that can generate date, and time as well between two ranges? This looks like a feature to me that can be helpful. Yesterday I found Pandas Library has this feature <https://pandas.pydata.org/docs/reference/api/pandas.date_range.html>. What do you think? On Tue, Feb 8, 2022 at 9:23 PM Paul Moore <p.f.moore@gmail.com> wrote:
I think a datetime.range object could be useful. Perhaps someone can write one and then see if the core devs would accept it in the stdlib. It would be na interesting exercise in any case :-) -CHB On Tue, Feb 8, 2022 at 10:38 PM Aman Pandey <amanpandey5319@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
I can give it a try, but I have never done it before and anything which I should be knowing beforehand because it will be my first time. I would like to listen from other fellow developers what they think about this? On Wed, Feb 9, 2022 at 1:09 PM Christopher Barker <pythonchb@gmail.com> wrote:
As I said, it’s an interesting exercise in any case. I have no idea if the core devs would be interested in adding it. There have been a couple recipes on this thread for an iterator, but I envision something like the range() object — it’s a lazy Sequence, not just an iterator. I think that would be pretty cool. -CHB On Wed, Feb 9, 2022 at 3:45 AM Aman Pandey <amanpandey5319@gmail.com> wrote:
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Christopher Barker writes:
I would be strongly against an extension to range() itself (I think that was the original proposal), and I think we haven't even started to mix the batter let alone bake the cake for a range method on datetime (or time or date). I see the following issues that would need to be discussed: 1. clock monotonicity 2. "daylight saving" monotonicity 3. handling of "naive" datetimes (probably easy, but that's what everybody says about everything) 4. The "anywhere on earth" (AOE) "timezone" and other similar not- exactly-what-you-teach-1st-graders-about-time practices. There's probably more, but given the above, but I don't see a good reason to say anything but """what's wrong with dt = datetime(*startdate) while dt <= datetime(*enddate): do_something_with(dt) dt += timedelta(*deltaspec) """ yet. It's possible that all of the above are straightforward to deal with (2 would be messy at best with an indexable dateteime_range, though), but the monotonic clock PEP was quite a bit of work. They need to be addressed though.
Hello Stephen, I think you misunderstood it. we are not talking about modifying the existing built-in function range rather adding a new function in datetime module that will generate the date and time between two periods. if we add a function like that in the datetime module the code will be exactly like the one you gave. Christopher was mentioning that he visualizes that function in the datetime module as something similar in syntax and execution like the built-in range function. what do you think if we add some function like that in datetime module wouldn't it be helpful? On Thu, Feb 10, 2022 at 4:56 PM Stephen J. Turnbull < stephenjturnbull@gmail.com> wrote:
On Thu, 10 Feb 2022 at 15:21, Aman Pandey <amanpandey5319@gmail.com> wrote:
I think something like this might occasionally be helpful. However, for simple examples, it's pretty easy to write your own, sufficiently so that people would expect more of a stdlib version. And understanding the edge cases, correctly handling them, and designing an API which makes the function easy to use while still having the power to cover everyone's use cases, is a *lot* harder. It's that part of the problem that is receiving the pushback here. To give a concrete example, consider a range with a start of 1:30am on a particular day, and a step of 1 day. If the range extends over a DST change, then there's one day in the range where either there are no valid 1:30am times, or there are *two*, equally valid 1:30am times (depending on whether the clock is "going forwards" or "going backwards"). What should the stdlib function do in this case? How should the function deal with the fact that people can legitimately expect different behaviours (for some applications "raise an exception" could be the only safe option, for others, "use the nearest valid time" might be)? Having flags that pick between different behaviours scales badly, and is typically not a good design. For your own code, you can write the behaviour that works best for you, and for your application. For the stdlib, the behaviour has to address the needs of *everyone*. My feeling is that the complexity of a "does the right thing for everyone" stdlib implementation would outweigh any benefit gained from not having to write your own short function, tailored to your particular needs. Paul
Since a started this:-) 1) the complexities are exactly why this *might* be a candidate for the stdlib. Those are the things that someone is likely to get wrong, or not think of, when they write something quick themselves, and only lightly test. 2) it doesn’t have to be everything for everyone- it has to be many things for many (most) people. 3) DST is a nightmare, plain and simple. The datetime module was useful for years even before it gained the ability to deal with it at all.[*] An educational thing to explore, in any case. -CHB [*] now that I think about it, I would probably only do this without DST — including DST would require a DST database, which the stdlib doesn’t include. On Thu, Feb 10, 2022 at 7:48 AM Paul Moore <p.f.moore@gmail.com> wrote:
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Feb 10, 2022 at 8:57 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
Don't forget leap seconds! :-)
TAI only :-) -CHB On Thu, Feb 10, 2022, 11:45 AM Christopher Barker <pythonchb@gmail.com>
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, 10 Feb 2022 at 16:44, Christopher Barker <pythonchb@gmail.com> wrote:
Since a started this:-)
1) the complexities are exactly why this *might* be a candidate for the stdlib. Those are the things that someone is likely to get wrong, or not think of, when they write something quick themselves, and only lightly test.
Fair. But no-one has come up with a use case to justify doing the work to handle those complexities. All the discussion has tended to be around how it "wouldn't be too hard".
2) it doesn’t have to be everything for everyone- it has to be many things for many (most) people.
Agreed, but that's mostly just a matter of degree. At a minimum, if it deliberately omits certain situations, those need to be documented. Otherwise there's a maintenance cost from people asking for such support and debating whether it's "implied" by the existence of the function in the stdlib.
3) DST is a nightmare, plain and simple. The datetime module was useful for years even before it gained the ability to deal with it at all.[*] [*] now that I think about it, I would probably only do this without DST — including DST would require a DST database, which the stdlib doesn’t include.
https://docs.python.org/3/library/zoneinfo.html Having a stdlib function that handles range-type sequences for dates, but does not handle the various timezone-aware datetime types is bound to cause some level of confusion. Having said that, maybe it's simple to do this right. That's my point, someone needs to do the work to think all this through. I'm not *against* the idea, I just don't think it's worth the effort given the (fairly trivial) benefits that have been claimed so far. In particular, I have no use for it personally - the one-liners already posted in this thread would be perfectly fine for any use I can imagine, and I see no reason to add such one-liners to the stdlib. Paul
On Thu, Feb 10, 2022 at 9:02 AM Paul Moore <p.f.moore@gmail.com> wrote:
My Bad -- I forgot about that -- I know there was a lot of debate about including something that was going to be out of data at some point :( -- so yes, a daterange could allow DST :-) But at the end of the day, it is never going to be better than horrible :-(
Exactly.
agreed. But we seem to be getting a mixed message here: 1) it's too coplex with too many variable that have to be thought through and won't satisfy everyone anyway. and 2) It's just a simple one-liner Those are kind of incompatible ideas ;-) This feels to me not so different then math.isclose() -- not that complex, but complex enough that it's good to put a well thought through implementation in the stdlib. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, 10 Feb 2022 at 17:11, Christopher Barker <pythonchb@gmail.com> wrote:
Indeed, so which is being proposed here? My answer is the same in both cases (I don't think it's worth adding to the stdlib) but which justification I need to express depends on which option is being proposed :-)
This feels to me not so different then math.isclose() -- not that complex, but complex enough that it's good to put a well thought through implementation in the stdlib.
Maybe. It still needs someone to get down to details and propose something specific. And I'll still be between -0.5 and -1 on it, but you don't need to convince me personally, you just need some level of support from *some* of the core devs, and/or broad community support sufficient to persuade at least one core dev to merge a PR implementing the idea. Paul
On Thu, Feb 10, 2022 at 9:25 AM Paul Moore <p.f.moore@gmail.com> wrote:
Exactly—I think it would be an interesting exercise— so worth doing even if no one finds it compelling. I’d likely work in it myself if I didn’t have 100 other interesting things I don’t have time for :-) But I’m always happy to kibitz…. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Christopher Barker writes:
agreed. But we seem to be getting a mixed message here:
1) it's too coplex with too many variable that have to be thought through
s/too//g please. I haven't seen any explicit -1s yet, just "I'm not gonna do it myself for sure". @Aman Paul's discussion mirrors my own thought (or vice versa if you like, Paul -- you wrote it better than I could have).
and won't satisfy everyone anyway.
You've already admitted that.
No, they aren't incompatible at all, because the "it" that refers to to a stdlib patch is a singleton that has to do it all, and the "it" that refers to one-liners is a fairly large plurality of one offs that do one thing well.
In outline, yes, but math.isclose() is a single function with literally centuries of math thought behind it, and at least a century of numerical analysis. I don't think a projected indexable datetime.range object has quite the same literature backing up the discussion. So I want to see the discussion. Steve
participants (6)
-
Aman Pandey -
Chris Angelico -
Christopher Barker -
David Mertz, Ph.D. -
Paul Moore -
Stephen J. Turnbull