proposal: add basic time type to the standard library

I propose adding a basic time type (or time base type ;-) to the standard library, which can be subclassed by more elaborate date/time/timestamp implementations, such as mxDateTime, custom types provided by DB-API drivers, etc. The goal is to make it easy to extract the year, month, day, hour, minute, and second from any given time object. Or to put it another way, I want the following to work for any time object, including mxDateTime objects, any date/timestamp returned by a DB-API driver, and weird date/time-like types I've developed myself: if isinstance(t, basetime): # yay! it's a timestamp print t.timetuple() The goal is not to standardize any behaviour beyond this; anything else should be provided by subtypes. More details here: http://effbot.org/ideas/time-type.htm I can produce PEP and patch if necessary. </F>

http://effbot.org/ideas/time-type.htm
I can produce PEP and patch if necessary.
Yes, a PEP, please! Jim Fulton has been asking for this for a long time too. His main requirement is that timestamp objects are small, both in memory and as pickles, because Zope keeps a lot of these around. They are currently represented either as long ints (with a little under 64 bits) or as 8-byte strings. A dedicated timestamp object could be smaller than that. Your idea of a base type (which presumably standarizes at least one form of representation) sounds like a breakthrough that can help satisfy different other needs. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Sounds like a plan :-) In order to make mxDateTime subtypes of this new type we'd need to make sure that the datetime type uses a true struct subset of what I have in DateTime objects now: typedef struct { PyObject_HEAD /* Representation used to do calculations */ long absdate; /* number of days since 31.12. in the year 1 BC calculated in the Gregorian calendar. */ double abstime; /* seconds since 0:00:00.00 (midnight) on the day pointed to by absdate */ ...lots of broken down values needed to assure roundtrip safety... } Depending on the size of PyObject_HEAD, this should meet Jim Fultons requirements (the base type would of course not implement the "..." part :-). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

"M.-A. Lemburg" wrote:
I forgot to mention that there is another object type in mxDateTime too: DateTimeDelta. That's the type needed to represent the time difference between two DateTime instances, or what people usually call "time" :-) It has the following type "signature": typedef struct { PyObject_HEAD double seconds; /* number of delta seconds */ ...some broken down values needed to assure roundtrip safety... } -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

mal wrote:
as Tim has pointed out, what I have in mind is: typedef struct { PyObject_HEAD /* nothing here: subtypes should implement timetuple and, if possible, utctimetuple */ } basetimeObject; /* maybe: */ PyObject* basetime_timetuple(PyObject* self, PyObject* args) { PyErr_SetString(PyExc_NotImplementedError, "must override"); return NULL; } (to adapt mxDateTime, all you should have to do is to inherit from baseObject, and add an alias for your "tuple" method) ::: since it's really easy to do, we should probably also add a simpletime type to the standard library, which wraps the standard time_t: typedef struct { PyObject_HEAD time_t time; /* maybe: int timezone; */ } simpletimeObject; ::: What I'm looking for is "decoupling", and making it easier for people to experiment with different implementations. Things like xmlrpclib, the logging system, database adapters, etc can look for basetime instances, and use the standard protocol to extract time information from any time object implementation. (I can imagine similar "abstract" basetypes for money/decimal data -- a basetype plus standardized behaviour for __int__, __float__, __str__ -- and possibly some other data types: baseimage, base- sound, basedomnode, ...) Hopefully, such base types can be converted to "interfaces" when- ever we get that. But I don't want to wait for a datetime working group to solve everything that MAL has already solved in mxDate- Time, and then everything he hasn't addressed. Nor do I want to wait for an interface working group to sort that thing out. Let's do something really simple instead. </F>

Fredrik Lundh wrote:
Ok. Sounds like you are inventing something like a set of abstract types here. I'm very much +1 on that idea, provided the interfaces we define for these types are simple enough (I think the DB SIG has shown that simple interface can go a looong way). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

[/F]
http://effbot.org/ideas/time-type.htm
I can produce PEP and patch if necessary.
[Guido]
Are you sure Jim is looking to replace the TimeStamp object? All the complaints I've seen aren't about the relatively tiny TimeStamp object, but about Zope's relatively huge DateTime class (note that you won't have source for that if you're looking at a StandaloneZODB checkout -- DateTime is used at higher Zope levels), which is a Python class with a couple dozen(!) instance attributes. See, e.g., http://dev.zope.org/Wikis/DevSite/Proposals/ReplacingDateTime It seems clear from the source code that TimeStamp is exactly what Jim intended it to be <wink>.
Best I can make out, /F is only proposing what Jim would call an Interface: the existence of two methods, timetuple() and utctimetuple(). In a comment on his page, /F calls it an "abstract" base class, which is more C++-ish terminology, and the sample implementation makes clear it's a "pure" abstract base class, so same thing as a Jim Interface in the end.

[Tim]
I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object. I think the mistake of DateTime is that it stores the broken-out info, rather than computing it on request.
I'll show the PEP to Jim when it appears. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]
I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object.
Given that most mallocs align to 8-byte boundaries these days (also true of pymalloc), it's impossible in reality to define a smaller object than TimeStamp, provided it needs at least one byte of info beyond PyObject_HEAD.
I think the mistake of DateTime is that it stores the broken-out info, rather than computing it on request.
Possibly, but hard to say, since speed of display is also an issue, and I imagine also speed of range searches. At least 2.2 makes it easy to define computed attributes, any of which could choose to cache their ultimate value, but none of which would need to be stored in pickles.

"TP" == Tim Peters <tim.one@comcast.net> writes:
TP> [Guido]
I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object.
TP> Given that most mallocs align to 8-byte boundaries these days TP> (also true of pymalloc), it's impossible in reality to define a TP> smaller object than TimeStamp, provided it needs at least one TP> byte of info beyond PyObject_HEAD. Also, it may not be necessary to have a TimeStamp object in ZODB 4. There are three uses for the timestamp: tracking how recently an object was used for cache evication, providing a last modified time to users, and as a simple version number. In ZODB 4, the cache eviction may be done quite differently. The version number may be a simple int. The last mod time will not be provided for each object; instead, users will need to define this themselves if they care about it. If they define it themselves, they'd probably use a DateTime object, but we'd care much less about how small it is. Jeremy

In that case, I take back everything I've said about Jim Fulton's requirements. I'm quite sure that in the past he said he needed a very lightweight date/time object, but from what you say it appears this need has disappeared. --Guido van Rossum (home page: http://www.python.org/~guido/)

Is comparison the same what Tim mentioned as range searches? I guess a representation like current Zope timestamps or what time.time() returns is fine for that -- it is monononous even if not necessarily continuous. I guess a broken-out time tuple is much harder to compare. --Guido van Rossum (home page: http://www.python.org/~guido/)

"Fred L. Drake, Jr." wrote:
Uhm... I think this thread is heading in the wrong direction. Fredrik wasn't proposing a solution to Jim's particular problem (whatever it was ;-), but instead opting for a solution of a large number of Python users out there. While mxDateTime probably works for most of them (and is used by pretty much all major database modules out there), some may feel that they don't want to rely on external libs for their software to run on. I would be willing to make the mxDateTime types subtypes of whatever Fredrik comes up with. The only requirement I have is that the binary footprint of the types needs to match todays layout of mxDateTime types since I need to maintain binary compatibility. The other possibility would be adding a set of new types to mxDateTime which focus on low memory requirements rather than data roundtrip safety and speed. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

[M.-A. Lemburg]
Uhm... I think this thread is heading in the wrong direction.
Maybe from your POV, but from our POV the only way we can get time to work on Python is talk all of you into doing Zope work for Jim <wink>.
I believe all /F is asking for is that all datetime types supply two specific methods, so that he can get the year etc out of anybody's datetime object via a uniform spelling. It's a fine idea.
If /F is asking more than that datetime types implement a specific interface, he's got some major rewriting to do <wink>. Python doesn't have a good way to spell "interface" now, so think of it as a do-nothing base class, inheriting from which means absolutely nothing except that (a) you promise to supply the methods /F specified, and (b) /F can use isinstance to determine whether or not a given object supports this interface.
That's getting back to what Jim wants. Maybe someone should ask him what that is <wink>.

"M.-A. Lemburg" wrote:
Right. ;)
I have no problem relying on external libraries.
The binary footprint of your types, not the standard base class, right? I don't see a problem with that,
What is data roundtrip safety? I rarely do date-time arithmetic, but I often do date-time-part extraction. I think that mxDateTime is optimized for arithmetic, whereas, I'd prefer a type more focussed on extraction efficiency, and memory usage, and that effciently supports time zones. This is, of course, no knock on mxDateTime. I also want fast comparisons, which I presume mxDateTime provides. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

WRT RAM usage, a Python int is no smaller than a TimeStamp object.
Wrong, unless TimeStamps also use a custom allocator. The custom allocator uses 12 bytes per int (on a 32-bit machine) and incurs malloc overhead + 8 bytes of additional overhead for every 82 ints. That's about 12.2 bytes per int object; using malloc it would probably be 24 bytes. (PyMalloc would probably do a little better, except it would still round up to 16 bytes.) If TimeStamp objects were to use a similar allocation scheme, they could be pushed down to 16.2 bytes. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim]
WRT RAM usage, a Python int is no smaller than a TimeStamp object.
[Guido[
Wrong, unless TimeStamps also use a custom allocator.
Good point, and it doesn't (it uses PyObject_NEW). I don't think counting fractions of bytes is of great interest here, though, since I (still) believe it's the massive Zope DateTime type that's the focus of complaints.
pymalloc overhead is a few percent; would work out to 16+f bytes per int object, for some f < 1.0. A difference is that "total memory dedicated to ints" never shrinks using the custom allocator, but can get reused for other objects under pymalloc.

Jeremy Hylton wrote:
"TP" == Tim Peters <tim.one@comcast.net> writes:
...
Time-stamp isn't used for this.
providing a last modified time to users, and as a simple version number.
This is certainly a hack.
In ZODB 4, the cache eviction may be done quite differently.
Yup, or, with Toby Dickenson's patches, in ZODB 3. :)
The TimeStamp type will still be useful for storage implementations that want compact time strings. I could imagine al alternate implementation that conformed to the new interface and retained the compact representation. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Guido van Rossum wrote:
I don't think the mistake was so much to store broken-out info, but to store too much data in general. It stores redundant data, which makes it's implementation a bit difficult to understand and maintain. Note that scalability was not a goal of Zope's DateTime type. We meant to replace it with something much tighter a long time ago, but never got around to it.
Yup.
I'll show the PEP to Jim when it appears.
Thanks. ;) BTW, has there been any progress on this? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

jim wrote:
BTW, has there been any progress on this?
the current pepoid proposal is here: http://effbot.org/ideas/time-type.htm major issues right now is 1. should the type know about timezones (probably not), and 2. should it support basic arithmetics (probably yes). (and 3. find time to write a html-to-pep converter ;-) cheers /F

Fredrik Lundh wrote:
:( -1 Doesn't the proposal sort of imply time-zone awareness of some kind? Or does it simply imply UT storage?
and 2. should it support basic arithmetics (probably yes).
Does this imply leap second hell, or will we simply be vague about expectations? I'd also like to see simple access methods for year, month, day, hours, minutes, and seconds, with date parts being one based and time parts being zero based. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Jim Fulton wrote:
I tried that in early version of mxDateTime -- it fails badly. I switched to the local time assumption very early in the development. Note that Fredrik's type is an abstract type; it doesn't even store anything -- that's up to subtypes which of course can implement timezones at their liking.
The type will store a fixed point in time, so why worry about leap seconds (most system's don't support these anyway and if they do, the support is usually switched off per default) ?
In the abstract base type ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

"M.-A. Lemburg" wrote:
That's a really bad assumption if times are used in different time zones.
No, it does store something, it just doesn't say how. There are methods for returning localtime and gmtime, so there is (abstract) storage.
There are a lot of semantic issues with date-time math. Leap seconds is an example. If you store local time, are date-time subtractions affected by daylight-savings time? Do the calculations depend on the calendar? Do you take into account the lost days in the switch from the Julean to the Gregorian calendar? I'm not really opposed to doing math, but we need to at least recognize the fuzzyness of the semantics.
Yes. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

mal wrote:
can you elaborate?
the updated proposal adds __hash__ and __cmp__, and the following (optional?) operations: deltaobject = timeobject - timeobject floatobject = float(deltaobject) # fractional seconds timeobject = timeobject + integerobject timeobject = timeobject + floatobject timeobject = timeobject + deltaobject note that "deltaobject" can be anything; the abstract type only says that if you manage to subtract one time object from another one of the same type, you get some object that you can 1) convert to a float, and 2) add to another time object. vague, but pretty useful.
Q. does mxDateTime provide separate accessors for individual members? </F>

Fredrik Lundh wrote:
First of all, the C lib only support UTC and local time, so you don't really have a chance of correctly converting a non-local time using a different time zone in either local time or UTC: there simply are no C APIs you could use and the problems which DST and leap seconds introduces are no fun at all (they are fun to read though: figuring out the various DST switch times is an adventure -- just have a look at the C lib's DST files). The next problem is that the C lib only provides APIs for conversion from local time to UTC, but not UTC to local time. There is an API called timegm() for this on some platforms, but its non-standard. As a result, making UTC the default won't allow you to safely represent the datetime value in local time. A third obstacle is typcial user assumptions: users simply assume local time and it's hard to tell them otherwise (people have very personal feelings about date and time for some reason...).
Indeed :-)
Yes, it provides access to these in form of attributes. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Does this imply leap second hell, or will we simply be vague about expectations?
IMO, leap seconds should be ignored. Time stands still during a leap second. Consider this a BDFL pronouncement if you wish. :-)
I'd also like to see simple access methods for year, month, day, hours, minutes, and seconds,
The timetuple() method provides access to all of these simultaneously. Isn't that enough? t.year() could be spelled as t.timetuple()[0]. I expect that usually you'd request several of these together anyway, in order to do some fancy formatting, so the timetuple() approach makes sense.
with date parts being one based and time parts being zero based.
I'm not sure what you mean here. --Guido van Rossum (home page: http://www.python.org/~guido/)

When do you ever use this in isolation? I'd expect in 99% of the cases you hand it off to a formatting routine, and guess what -- strftime() takes a time tuple. I worry about the time wasted by calling all of t.year(), t.month(), t.day() (etc.) -- given that they do so little, the call overhead is probably near 100%. I wonder how often this is needed. The only occurrences of year() in the entire Zope source that I found are in various test routines.
Serious question: what do you tend to do with time values? I imagine that once we change strftime() to accept an abstract time object, you'll never need to call either timetuple() or year() -- strftime() will do it for you.
That's why /F's pre-PEP allows the implementation to leaves these three set to -1.
I guess that confusion is yours alone. In Python, of course month and day start from 1. Whether years start from 1 is a theological question. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
These methods and others are used a lot in presentation code, which tends to be expressed in DTML or ZPT. It's not uncommon to select/catagorize things by year or month. I think most people would find individual date-part methods a lot more natural than tuples.
I format them in various ways and I sort them.
Maybe, if I use strftime, but I don't use strftime all that much. I can certainly think of even formatting cases (e.g. internationalized dates) where it's not adequate.
I missed these. Still, providing -1s seems, uh, vestigial.
I doubt the confusion is mine alone, but I'll take your word for it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

[me]
I wonder how often this is needed. The only occurrences of year() in the entire Zope source that I found are in various test routines.
[Jim]
OK, that explains a lot. For this context I agree, although I think they should probably appear as (computed) attributes rather than methods. Properties seem perfect.
Maybe you should. :-)
I can certainly think of even formatting cases (e.g. internationalized dates) where it's not adequate.
Then a super-strftime() should be invented that *is* enough, rather than fumbling with hand-coded solutions. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
That's fine with me.
I do when I can. But it often doesn't meet my needs.
I think we don't need a one-size-fits-all all-powerful date-time formating solution. ;) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

I think we don't need a one-size-fits-all all-powerful date-time formating solution. ;)
It's probably impossible to create one, but I think there's also no reason to require people to invent the wheel over and over. I've seen enough broken code attempting to do date/time formatting that I strongly prefer the creation of a few standard solutions that will work for most people, rather than only giving people the low-level bits to work with. Another thing to consider is that for most apps, the choice of the date/time format should be taken out of the hands of the programmer and placed into the hands of the user, through some kind of preference setting. I18n and L10n also strongly suggests to take this route. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
FWIW, mxDateTime exposes these values as attributes -- there is no call overhead.
I actually use the attributes quite a bit in my stuff and hardly ever use .strftime(). The mxDateTime is different though, e.g. I sometimes get questions about how to make strftime() output fractions of a seconds (doesn't work, because strftime() doesn't support it).
Depends on the application space. Database applications will call .timetuple() very often and use strftime() hardly ever.
It's not really a question: the year 0 simply does not exist in reality ! (Christians didn't have a 0 available at the time ;-) Still, historic dates are usually referenced by making year 0 == 1 b.c., -1 == 2 b.c., etc. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

FWIW, mxDateTime exposes these values as attributes -- there is no call overhead.
Good, I think this is the way to go. (Of course there will be some C-level call overhead if we make these properties.)
What does a database app with the resulting tuple? --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Right.
It puts the values into struct fields for year, month, day, etc. (Databases usually avoid using Unix ticks since these cause the known problems with dates prior to 1970) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Hm, I thought that databases have their own date/time types? Aren't these used? --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
At C level, interfacing is usually done using structs (ISO SQL/CLI defines these). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Ah, there's another requirement that we (or at least I) almost forgot. There should be an efficient C-level interface for the abstract date/time type. This stuff is hairier than it seems! I think the main tension is between improving upon the Unix time_t type, and improving upon the Unix "struct tm" type. Improving upon time_t could mean to extend the range beyond 1970-2038, and/or extend the precision to milliseconds or microseconds. Improving upon struct tm is hard (it has all the necessary fields and no others), unless you want to add operations (just add methods) or make the representation more compact (several of the fields can be packed in 4-6 bits each). A third dimension might be to provide better date/time arithmetic, but I'm not sure if there's much of a market for that, given all the fuzzy semantics (leap seconds, differences across DST changes, timezones). --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]
... This stuff is hairier than it seems!
You're just getting your toes wet: it's impossible to make any two of {astronomers, businessfolk, Jim} happy at the same time, even if they're all American and live in the same city.
I'm suprised you say "all the necessary fields", because a tm contains no info about timezone. C99 introduces a struct tmx that does. The initial segment of a struct tmx must be identical to a struct tm, but the meaning of tmx.tm_isdst differs from tm.tm_isdst. tmx.tm_isdst is the positive number of minutes of offset if Daylight Saving Time is in effect, zero if Daylight Saving Time is not in effect, and -1 if the information is not available. Then it adds some fields not present in a struct tm: int tm_version; // version number int tm_zone; // time zone offset in minutes from UTC [-1439, +1439] int tm_leapsecs; // number of leap seconds applied void *tm_ext; // extension block size_t tm_extlen;// size of the extension block The existence of tm_version, tm_ext and tm_extlen can be fairly viewed as a committee's inability to say "no" <wink>.
I don't think we can get off that easy. Time computation is critical for businesses and astronomers, and leap seconds etc are a PITA independent of time computations. Time computations seem to me to be the easiest of all, provided we've already "done something" intelligible about the rest: any calculation boils down to factoring away leap seconds etc in conversion to a canonical form, doing the computing there, then injecting leap seconds etc back in to the result when converting out of canonical form again. The ECMAScript std (nee Javascript) has, I think, a good example of a usable facility that refused to get mired down in impossible details; e.g., it flat-out refuses to recognize leap seconds. mxDateTime is similarly sane, but MAL keeps threatening to flirt with insanity <wink>. BTW, I doubt there'd be any discussion of leap seconds in the C std if some astronomers hadn't been early Unix users. It's never a net win in the end to try to make a scientist happy <0.9 wink>.

Tim Peters wrote:
FYI, mxDateTime test the C lib for leap second support; if leap seconds are used, then it has to support these too in conversions from and to Unix ticks.
What strange about leap seconds is that they don't fit well with the idea of counting seconds since some fixed point in history. They are only useful for conversions from this count to a broken down date and time representation.... time simply doesn't leap.

Since AFAIK POSIX doesn't admit the existence of leap seconds, how do you ask the C library for leap seconds?
Yeah, we learned that the hard way by adding complex numbers. :-)
I think an important (but so far unvoiced) requirement is that datetime objects can be stored in a database. Since the database may be read by systems that may or may not support leap seconds, we should be independent of the leap second support in the C library. As I've said before, we should ignore leap seconds. Even if we end up expressing times deltas as a number of seconds, that should be understood to be calendar seconds and not astronomical seconds. Let the astronomers deal with leap seconds themselves -- they should know how to. BTW, this means that we can't use the C calls mktime(), timegm(), localtime(), and gmtime(), or their Python wrappers in the time module! That's fine by me. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
See below (the quoted C comment).
SQL databases don't deal with leap seconds. They store the broken down value (in some way) without time zone information and that's it, fortunately :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

On Wed, 27 Feb 2002, M.-A. Lemburg wrote:
Er... SQL99 (and I believe SQL92) have native support for time with and without time zones, and neither say nothing about how databases are to "store" those values. I don't have a copy in front of me, so I can't tell you what they say about leap-seconds. Of course, few implementations support this yet, though it worth being forward-looking. For my own uses, I have a base time class that encapsulates either mxDateTime objects or unix time-since-epoch, and implements the basic time and date accessors and simple arithmetic. A subclass of that type then adds awareness of timezones and daylight savings time. My first effort at trying to do all of those things in one big monolithic class was a nightmare. This layering does result in some (relative) inefficiency, but correctness and maintainability is vastly more important to me. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

Kevin Jacobs wrote:
True, SQL-92 defines data types "TIME WITH TIME ZONE" and "TIMESTAMP WITH TIME ZONE". The standard is only available as book, but here's a draft which has all the details: http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt Still, only Oracle and PostgreSQL seem to actually implement these and ODBC (SQL/CLI), the defacto standard for database interfacing, doesn't even provide interfaces to query or store time zone information (you can put the information directly in the SQL string, but not use it in bound variables). Basically, you should not store local time in databases, but instead use UTC. If you need the original time zone information for reference, you'd keep this in separate DB columns (e.g. as strings). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

On Wed, 27 Feb 2002, M.-A. Lemburg wrote:
Strangely enough I use TIMESPAMP WITH TIMEZONE quite a bit on both Oracle and PostgreSQL using native drivers. I'm also fairly sure that Sybase and MS-SQL store timestamps with timezone somehow, though my memory on the project that did so is a little fuzzy.
Why not minute offset from UTC like C99? Anyhow, everyone knows that time zones and daylight savings time are a pain to deal with. However, lets provide work toward a sane implementation that can relieve the end-user from having to smack their head against this particular brick wall every time. (even if it means smacking our collective heads against the brick wall until we're happy, or reduced to unintelligible ranting, or possibly both). Regards, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

Guido van Rossum wrote:
As for us astronomers, we're suppose to represent time in Julian days and fractions thereof since the beginning of time (about 6714 years ago). Today is day 2452346. In practice we use whole days and represent the fractional part in seconds, because floating point numbers don't have a sufficient number of bits to represent Julian days to nanosecond precision. A typical day contains 86400 seconds. In essence we use Julian days as our reference point and seconds of a day as our delta time. From these two values you can theoretically calculate any time past, present, or future with or without leap seconds (if known). Just thought you might like to know, if you didn't already. -- Paul -- Paul Barrett, PhD Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Group FAX: 410-338-4767 Baltimore, MD 21218

[me]
[Tim]
I'm suprised you say "all the necessary fields", because a tm contains no info about timezone.
Oops. My mistake. I thought it had timezone. [tmx details snipped]
I'd be happy to support time computations, provided we keep the leap seconds out. I propose a representation that resembles a compressed struct tm (or tmx), with appropriately-sized bit fields for year, month, day, hour, minute, second, millisecond, and microsecond, and timezone and DST info. Since the most likely situation is extraction in local time, these should be stored as local time with an explicit timezone. (I don't want to store these things in a database without an explicit timezone, even if it costs another 12 bit field.) An app extracting the local time without checking the timezone could be fooled by a time stored with a different timezone. Do we care? Time computations are only slightly complex because they have to be calendar-aware, but at least they don't have to be DST-aware -- they can just thake the timezone offset in minutes and apply it. The DST info should probably be two bits: one telling whether DST is in effect at the given time, one telling whether DST is honored in the given timezone. Maybe it should also allow "missing info" for either. Details, details. --Guido van Rossum (home page: http://www.python.org/~guido/)

The discussion is going astray again: Fredrik proposed an abstract base type, i.e. a type providing only the name and an interface which is defined as convention. I am all for adding such an abstract base type (and others as well, e.g. for numbers, sequences, money, decimal, etc.) with minimal interfaces, but not for fixing a complex interface on top of these. What you are currently discussing is heading in the direction of imlementing one or more time subclasses. That's two steps ahead of what Fredrik was proposing. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

On Wednesday, February 27, 2002, at 03:21 , M.-A. Lemburg wrote:
Oops, I had missed that bit as well, that adding an *abstract* base type was the intention. I'm all for that as well. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Good point. The two discussions are both useful, but should be separated. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido> The timetuple() method provides access to all of these Guido> simultaneously. Isn't that enough? t.year() could be spelled as Guido> t.timetuple()[0]. Since we're discussing an abstract type it probably doesn't apply directly, but perhaps timetuple() could be specified to return a super-tuple like os.stat() does... Skip

jim wrote:
as written, it still implies time-zone awareness. the question is whether to remove that constraint (and the utc* methods). *all* early reviewers argued that time zones are a representation thingie, and doesn't belong in the abstract type. I'm tempted to agree, but I'm not sure I can explain why...
vague.
use timetuple(). (I rather not add too much stuff to the abstract interface; the goal is to let MAL turn mxDateTime into a basetime sub- type without breaking any application code...) </F>

*all* early reviewers argued that time zones are a representation thingie, and doesn't belong in the abstract type.
Then maybe the reviewers didn't have a sufficiently wide range of applications in mind? If I were to create a database of email messages, I'd be seriously annoyed if it normalized the timezone info away. A message sent at 10pm EST has a different feel to it than one sent at 4am MET. It should *sort* on UTC, but it should use the original timezone to display the dates. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum writes:
Sounds like a user preference, not a universal truth. Is it important that the timezone is part of the date/time type, though? Is it important that it be part of the abstract base date/time? Specific implementations should certainly be able to add support for timezones, and perhaps some hypothetical default date/time type should include it for convenience, but that doesn't tell me it's fundamental. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

[Fred]
Sounds like a user preference, not a universal truth.
Fair enough. Even some of my well-traveled friends cannot do timezone arithmetic in their head... :-)
I guess I want it to be possible to have an implementation that keeps track of the timezone as entered. It's true that time deltas are a nightmare when dealing with different timezones. --Guido van Rossum (home page: http://www.python.org/~guido/)

and while I'm at it: I propose adding an "abstract" money base type to the standard library, to be subclassed by real money/decimal implementations. if isinstance(v, basemoney): # yay! it's money print float(money) # let's hope it's not too much The goal is not to standardize any behaviour beyond this; anything else should be provided by subtypes. More details here: http://effbot.org/ideas/money-type.htm I can produce PEP and patch if necessary. </F>

I propose adding an "abstract" money base type to the standard library, to be subclassed by real money/decimal implementations.
Why do we need this? I guess that would be Question #1... --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
For databases ?! The DB API has long had a monetary or at least decimal type on its plate... never got around to implementing one, though :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

I can only find one reference to money or decimal in the DB API PEP, and that's as a future task. I guess that's what you mean by "on its plate". Since I'm not a database expert, maybe you can explain the use of this in more detail? And why would we need a monetary type rather than a fixed-point decimal type? --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Exactly :-)
A decimal would do as well, I suppose, at least in terms of storing the raw value. The reason for trying to come up with a monetary type is to make operations between monetary values having two different currencies illegal. Coercion between two of those would always have to be made explicit (for obvious reasons). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Are you sure you're trying to solve a real problem here? There are lots of operations on monetary values that make no sense (try multiplying two amounts of money). --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Indeed, monetary types solve different problems than decimal types. Financial applications do have a need for these kind of implicit error checks. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

But this is easily done by creating a custom class -- which has the advantage that the set of constraints can be specialized to the needs of a specific application. When we add a monetary type to the language we'll never get it right for all apps. OTOH, I think we could get a fixed point type right. How many other languages have a monetary type? What support for money does SQL have? --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
True. A real implementation of a good working decimal type with adjustable rounding rules would certainly go a long way and the money type could be built on top of it.
What support for money does SQL have?
SQL-92 doesn't have support for it, but some modern database engines do, e.g. MS SQL Server, PostgreSQL (even though it's deprecated there, now), MS Access. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

M.-A. Lemburg wrote:
I'm about 1/3 of the way there. Now that I'm recovered from my cochlear implant surgery and have a Linux box set up, I expect to get back to it Real Soon Now. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many.

Fredrik Lundh wrote:
Sounds like a plan. One thing though: the RE "[+|-]?\d+(.\d+)?" should be extended to allow for currency symbols and names in front or after the monetary value. Currency for money is a bit like timezones for datetime, so it's a good idea, not to add it to the base type interface. However, the interface should be extendable to include currency information. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

mal wrote:
isn't this better handled by a separate method/attribute? (otherwise, I fear that we'll end up adding all possible currency notations to the abstract type. but maybe there is a standard for this, somewhere?) </F>

Currency is much worse than timezones -- once you are interested in exchange rates, you need to know *when* to calculate the exchange rate (as well as other parameters such as which exchange rate). So please let's keep the currency out of the money type; it's utterly application dependent what to do with that information. --Guido van Rossum (home page: http://www.python.org/~guido/)

Fredrik Lundh wrote:
-1 on any proposal for a built-in money type that gets *anywhere* near floats. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many.

Guido van Rossum wrote:
I also need time-zone support.
They are currently represented either as long ints (with a little under 64 bits) or as 8-byte strings.
ZODB has a TimeStamp type that uses a 32-bit unsigned integer to store year, month,, day, hour, and minute in a way that makes it dirt simple to extract a component. It uses a 32-bit integer to store seconds in units of 60/2**32 seconds. This type isn't appropriate for general use because it only allows dates later than Dec 31, 1899.
A dedicated timestamp object could be smaller than that.
A type that only needed minute precision could easily be expressed with 32-bits. Of course, the two-word object overhead makes the difference between 32-bits and 64-bits rather unexciting.
I agree. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

[Jim Fulton]
You really think so? It's a mixed-radix scheme: v=((((y-1900)*12+mo-1)*31+d-1)*24+h)*60+m; so requires lots of expensive integer division and remainder operations to pick apart again (the trend in CPUs is to make these relatively more expensive, not less, and e.g. Itanium doesn't even have an integer division instruction). If we had this to do over again, I'd strongly suggest assigning 12 bits to the year, 4 to the month, 5 each to day and hour, and 6 to the minute. The components would then be truly dirt simple and dirt cheap to extract, and we wouldn't even have to bother switching between 0-based and 1-based for the months and days (let 'em stay 1-based). They would still sort and compare correctly in packed format. The only downside I can see is that not pursuing every last drop of potential compression would shrink the dynamic range from 8000+ years to 4000+ years, but we're likely to have much worse problems in Zope by the year 5900 anyway <wink>.

Tim Peters wrote:
Compared to storing date-times as offsets from an epoch, this is much simpler and cheaper.
Sounds good to me. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

[Jim Fulton]
[Tim]
[Jim]
Compared to storing date-times as offsets from an epoch, this is much simpler and cheaper.
OK, as with most things, it boils down to the definition of dirt: you're contrasting hard-packed dirt with a 21%-dirt 79%-concrete mix, and I'm constrasting hard-packed dirt with household dust. I'm sure you'll agree that's a rigorously correct summary <wink>.

Fredrik Lundh wrote:
Looks good! I'd prefer None to -1, though, for the last three items of the tuple. Also, the raise on utctime() should be NotImplementedError, maybe? -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many.

Aahz Maruch wrote:
None would be better from an interface design POV, but for historic reasons (compatibility to localtime()) -1 is better.
Also, the raise on utctime() should be NotImplementedError, maybe?
In the DB API we let the implementors decide: if the functionality cannot be provided per design, then it should not be implemented; if it can be implemented, but only works under certain conditions, a DB API NotSupportedError is raised instead. For mxDateTime I would implement both methods since mxDateTime does not store a timezone with the value but instead defines methods (and other operations) based on assumptions about the value. Time zones are on the plate, though, and the parser already knows about them. The C lib only provides APIs for local time and UTC; if you ever tried to convert a non-local time value into UTC, you'll know that this is not easy at all (mostly because of the troubles caused by DST and sometimes also due to leap seconds getting in the way). About the proposed interface: I'd rename the type to datetimebase and the methods to .tuple() and .gmtuple(). y,m,d = datetime.tuple()[:3] h,m,s = datetime.utctuple()[3:6] IMHO, it looks better :-) One thing I'm missing is a definition for a constructor (type objects are callable, so it'll have to do something, I guess...) and there should also be a datetimedeltabase type (this one is needed for dealing with the difference between two datetime values). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

http://effbot.org/ideas/time-type.htm
I can produce PEP and patch if necessary.
Yes, a PEP, please! Jim Fulton has been asking for this for a long time too. His main requirement is that timestamp objects are small, both in memory and as pickles, because Zope keeps a lot of these around. They are currently represented either as long ints (with a little under 64 bits) or as 8-byte strings. A dedicated timestamp object could be smaller than that. Your idea of a base type (which presumably standarizes at least one form of representation) sounds like a breakthrough that can help satisfy different other needs. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Sounds like a plan :-) In order to make mxDateTime subtypes of this new type we'd need to make sure that the datetime type uses a true struct subset of what I have in DateTime objects now: typedef struct { PyObject_HEAD /* Representation used to do calculations */ long absdate; /* number of days since 31.12. in the year 1 BC calculated in the Gregorian calendar. */ double abstime; /* seconds since 0:00:00.00 (midnight) on the day pointed to by absdate */ ...lots of broken down values needed to assure roundtrip safety... } Depending on the size of PyObject_HEAD, this should meet Jim Fultons requirements (the base type would of course not implement the "..." part :-). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

"M.-A. Lemburg" wrote:
I forgot to mention that there is another object type in mxDateTime too: DateTimeDelta. That's the type needed to represent the time difference between two DateTime instances, or what people usually call "time" :-) It has the following type "signature": typedef struct { PyObject_HEAD double seconds; /* number of delta seconds */ ...some broken down values needed to assure roundtrip safety... } -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

mal wrote:
as Tim has pointed out, what I have in mind is: typedef struct { PyObject_HEAD /* nothing here: subtypes should implement timetuple and, if possible, utctimetuple */ } basetimeObject; /* maybe: */ PyObject* basetime_timetuple(PyObject* self, PyObject* args) { PyErr_SetString(PyExc_NotImplementedError, "must override"); return NULL; } (to adapt mxDateTime, all you should have to do is to inherit from baseObject, and add an alias for your "tuple" method) ::: since it's really easy to do, we should probably also add a simpletime type to the standard library, which wraps the standard time_t: typedef struct { PyObject_HEAD time_t time; /* maybe: int timezone; */ } simpletimeObject; ::: What I'm looking for is "decoupling", and making it easier for people to experiment with different implementations. Things like xmlrpclib, the logging system, database adapters, etc can look for basetime instances, and use the standard protocol to extract time information from any time object implementation. (I can imagine similar "abstract" basetypes for money/decimal data -- a basetype plus standardized behaviour for __int__, __float__, __str__ -- and possibly some other data types: baseimage, base- sound, basedomnode, ...) Hopefully, such base types can be converted to "interfaces" when- ever we get that. But I don't want to wait for a datetime working group to solve everything that MAL has already solved in mxDate- Time, and then everything he hasn't addressed. Nor do I want to wait for an interface working group to sort that thing out. Let's do something really simple instead. </F>

Fredrik Lundh wrote:
Ok. Sounds like you are inventing something like a set of abstract types here. I'm very much +1 on that idea, provided the interfaces we define for these types are simple enough (I think the DB SIG has shown that simple interface can go a looong way). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

[/F]
http://effbot.org/ideas/time-type.htm
I can produce PEP and patch if necessary.
[Guido]
Are you sure Jim is looking to replace the TimeStamp object? All the complaints I've seen aren't about the relatively tiny TimeStamp object, but about Zope's relatively huge DateTime class (note that you won't have source for that if you're looking at a StandaloneZODB checkout -- DateTime is used at higher Zope levels), which is a Python class with a couple dozen(!) instance attributes. See, e.g., http://dev.zope.org/Wikis/DevSite/Proposals/ReplacingDateTime It seems clear from the source code that TimeStamp is exactly what Jim intended it to be <wink>.
Best I can make out, /F is only proposing what Jim would call an Interface: the existence of two methods, timetuple() and utctimetuple(). In a comment on his page, /F calls it an "abstract" base class, which is more C++-ish terminology, and the sample implementation makes clear it's a "pure" abstract base class, so same thing as a Jim Interface in the end.

[Tim]
I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object. I think the mistake of DateTime is that it stores the broken-out info, rather than computing it on request.
I'll show the PEP to Jim when it appears. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]
I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object.
Given that most mallocs align to 8-byte boundaries these days (also true of pymalloc), it's impossible in reality to define a smaller object than TimeStamp, provided it needs at least one byte of info beyond PyObject_HEAD.
I think the mistake of DateTime is that it stores the broken-out info, rather than computing it on request.
Possibly, but hard to say, since speed of display is also an issue, and I imagine also speed of range searches. At least 2.2 makes it easy to define computed attributes, any of which could choose to cache their ultimate value, but none of which would need to be stored in pickles.

"TP" == Tim Peters <tim.one@comcast.net> writes:
TP> [Guido]
I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object.
TP> Given that most mallocs align to 8-byte boundaries these days TP> (also true of pymalloc), it's impossible in reality to define a TP> smaller object than TimeStamp, provided it needs at least one TP> byte of info beyond PyObject_HEAD. Also, it may not be necessary to have a TimeStamp object in ZODB 4. There are three uses for the timestamp: tracking how recently an object was used for cache evication, providing a last modified time to users, and as a simple version number. In ZODB 4, the cache eviction may be done quite differently. The version number may be a simple int. The last mod time will not be provided for each object; instead, users will need to define this themselves if they care about it. If they define it themselves, they'd probably use a DateTime object, but we'd care much less about how small it is. Jeremy

In that case, I take back everything I've said about Jim Fulton's requirements. I'm quite sure that in the past he said he needed a very lightweight date/time object, but from what you say it appears this need has disappeared. --Guido van Rossum (home page: http://www.python.org/~guido/)

Is comparison the same what Tim mentioned as range searches? I guess a representation like current Zope timestamps or what time.time() returns is fine for that -- it is monononous even if not necessarily continuous. I guess a broken-out time tuple is much harder to compare. --Guido van Rossum (home page: http://www.python.org/~guido/)

"Fred L. Drake, Jr." wrote:
Uhm... I think this thread is heading in the wrong direction. Fredrik wasn't proposing a solution to Jim's particular problem (whatever it was ;-), but instead opting for a solution of a large number of Python users out there. While mxDateTime probably works for most of them (and is used by pretty much all major database modules out there), some may feel that they don't want to rely on external libs for their software to run on. I would be willing to make the mxDateTime types subtypes of whatever Fredrik comes up with. The only requirement I have is that the binary footprint of the types needs to match todays layout of mxDateTime types since I need to maintain binary compatibility. The other possibility would be adding a set of new types to mxDateTime which focus on low memory requirements rather than data roundtrip safety and speed. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

[M.-A. Lemburg]
Uhm... I think this thread is heading in the wrong direction.
Maybe from your POV, but from our POV the only way we can get time to work on Python is talk all of you into doing Zope work for Jim <wink>.
I believe all /F is asking for is that all datetime types supply two specific methods, so that he can get the year etc out of anybody's datetime object via a uniform spelling. It's a fine idea.
If /F is asking more than that datetime types implement a specific interface, he's got some major rewriting to do <wink>. Python doesn't have a good way to spell "interface" now, so think of it as a do-nothing base class, inheriting from which means absolutely nothing except that (a) you promise to supply the methods /F specified, and (b) /F can use isinstance to determine whether or not a given object supports this interface.
That's getting back to what Jim wants. Maybe someone should ask him what that is <wink>.

"M.-A. Lemburg" wrote:
Right. ;)
I have no problem relying on external libraries.
The binary footprint of your types, not the standard base class, right? I don't see a problem with that,
What is data roundtrip safety? I rarely do date-time arithmetic, but I often do date-time-part extraction. I think that mxDateTime is optimized for arithmetic, whereas, I'd prefer a type more focussed on extraction efficiency, and memory usage, and that effciently supports time zones. This is, of course, no knock on mxDateTime. I also want fast comparisons, which I presume mxDateTime provides. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

WRT RAM usage, a Python int is no smaller than a TimeStamp object.
Wrong, unless TimeStamps also use a custom allocator. The custom allocator uses 12 bytes per int (on a 32-bit machine) and incurs malloc overhead + 8 bytes of additional overhead for every 82 ints. That's about 12.2 bytes per int object; using malloc it would probably be 24 bytes. (PyMalloc would probably do a little better, except it would still round up to 16 bytes.) If TimeStamp objects were to use a similar allocation scheme, they could be pushed down to 16.2 bytes. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim]
WRT RAM usage, a Python int is no smaller than a TimeStamp object.
[Guido[
Wrong, unless TimeStamps also use a custom allocator.
Good point, and it doesn't (it uses PyObject_NEW). I don't think counting fractions of bytes is of great interest here, though, since I (still) believe it's the massive Zope DateTime type that's the focus of complaints.
pymalloc overhead is a few percent; would work out to 16+f bytes per int object, for some f < 1.0. A difference is that "total memory dedicated to ints" never shrinks using the custom allocator, but can get reused for other objects under pymalloc.

Jeremy Hylton wrote:
"TP" == Tim Peters <tim.one@comcast.net> writes:
...
Time-stamp isn't used for this.
providing a last modified time to users, and as a simple version number.
This is certainly a hack.
In ZODB 4, the cache eviction may be done quite differently.
Yup, or, with Toby Dickenson's patches, in ZODB 3. :)
The TimeStamp type will still be useful for storage implementations that want compact time strings. I could imagine al alternate implementation that conformed to the new interface and retained the compact representation. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Guido van Rossum wrote:
I don't think the mistake was so much to store broken-out info, but to store too much data in general. It stores redundant data, which makes it's implementation a bit difficult to understand and maintain. Note that scalability was not a goal of Zope's DateTime type. We meant to replace it with something much tighter a long time ago, but never got around to it.
Yup.
I'll show the PEP to Jim when it appears.
Thanks. ;) BTW, has there been any progress on this? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

jim wrote:
BTW, has there been any progress on this?
the current pepoid proposal is here: http://effbot.org/ideas/time-type.htm major issues right now is 1. should the type know about timezones (probably not), and 2. should it support basic arithmetics (probably yes). (and 3. find time to write a html-to-pep converter ;-) cheers /F

Fredrik Lundh wrote:
:( -1 Doesn't the proposal sort of imply time-zone awareness of some kind? Or does it simply imply UT storage?
and 2. should it support basic arithmetics (probably yes).
Does this imply leap second hell, or will we simply be vague about expectations? I'd also like to see simple access methods for year, month, day, hours, minutes, and seconds, with date parts being one based and time parts being zero based. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Jim Fulton wrote:
I tried that in early version of mxDateTime -- it fails badly. I switched to the local time assumption very early in the development. Note that Fredrik's type is an abstract type; it doesn't even store anything -- that's up to subtypes which of course can implement timezones at their liking.
The type will store a fixed point in time, so why worry about leap seconds (most system's don't support these anyway and if they do, the support is usually switched off per default) ?
In the abstract base type ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

"M.-A. Lemburg" wrote:
That's a really bad assumption if times are used in different time zones.
No, it does store something, it just doesn't say how. There are methods for returning localtime and gmtime, so there is (abstract) storage.
There are a lot of semantic issues with date-time math. Leap seconds is an example. If you store local time, are date-time subtractions affected by daylight-savings time? Do the calculations depend on the calendar? Do you take into account the lost days in the switch from the Julean to the Gregorian calendar? I'm not really opposed to doing math, but we need to at least recognize the fuzzyness of the semantics.
Yes. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

mal wrote:
can you elaborate?
the updated proposal adds __hash__ and __cmp__, and the following (optional?) operations: deltaobject = timeobject - timeobject floatobject = float(deltaobject) # fractional seconds timeobject = timeobject + integerobject timeobject = timeobject + floatobject timeobject = timeobject + deltaobject note that "deltaobject" can be anything; the abstract type only says that if you manage to subtract one time object from another one of the same type, you get some object that you can 1) convert to a float, and 2) add to another time object. vague, but pretty useful.
Q. does mxDateTime provide separate accessors for individual members? </F>

Fredrik Lundh wrote:
First of all, the C lib only support UTC and local time, so you don't really have a chance of correctly converting a non-local time using a different time zone in either local time or UTC: there simply are no C APIs you could use and the problems which DST and leap seconds introduces are no fun at all (they are fun to read though: figuring out the various DST switch times is an adventure -- just have a look at the C lib's DST files). The next problem is that the C lib only provides APIs for conversion from local time to UTC, but not UTC to local time. There is an API called timegm() for this on some platforms, but its non-standard. As a result, making UTC the default won't allow you to safely represent the datetime value in local time. A third obstacle is typcial user assumptions: users simply assume local time and it's hard to tell them otherwise (people have very personal feelings about date and time for some reason...).
Indeed :-)
Yes, it provides access to these in form of attributes. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Does this imply leap second hell, or will we simply be vague about expectations?
IMO, leap seconds should be ignored. Time stands still during a leap second. Consider this a BDFL pronouncement if you wish. :-)
I'd also like to see simple access methods for year, month, day, hours, minutes, and seconds,
The timetuple() method provides access to all of these simultaneously. Isn't that enough? t.year() could be spelled as t.timetuple()[0]. I expect that usually you'd request several of these together anyway, in order to do some fancy formatting, so the timetuple() approach makes sense.
with date parts being one based and time parts being zero based.
I'm not sure what you mean here. --Guido van Rossum (home page: http://www.python.org/~guido/)

When do you ever use this in isolation? I'd expect in 99% of the cases you hand it off to a formatting routine, and guess what -- strftime() takes a time tuple. I worry about the time wasted by calling all of t.year(), t.month(), t.day() (etc.) -- given that they do so little, the call overhead is probably near 100%. I wonder how often this is needed. The only occurrences of year() in the entire Zope source that I found are in various test routines.
Serious question: what do you tend to do with time values? I imagine that once we change strftime() to accept an abstract time object, you'll never need to call either timetuple() or year() -- strftime() will do it for you.
That's why /F's pre-PEP allows the implementation to leaves these three set to -1.
I guess that confusion is yours alone. In Python, of course month and day start from 1. Whether years start from 1 is a theological question. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
These methods and others are used a lot in presentation code, which tends to be expressed in DTML or ZPT. It's not uncommon to select/catagorize things by year or month. I think most people would find individual date-part methods a lot more natural than tuples.
I format them in various ways and I sort them.
Maybe, if I use strftime, but I don't use strftime all that much. I can certainly think of even formatting cases (e.g. internationalized dates) where it's not adequate.
I missed these. Still, providing -1s seems, uh, vestigial.
I doubt the confusion is mine alone, but I'll take your word for it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

[me]
I wonder how often this is needed. The only occurrences of year() in the entire Zope source that I found are in various test routines.
[Jim]
OK, that explains a lot. For this context I agree, although I think they should probably appear as (computed) attributes rather than methods. Properties seem perfect.
Maybe you should. :-)
I can certainly think of even formatting cases (e.g. internationalized dates) where it's not adequate.
Then a super-strftime() should be invented that *is* enough, rather than fumbling with hand-coded solutions. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
That's fine with me.
I do when I can. But it often doesn't meet my needs.
I think we don't need a one-size-fits-all all-powerful date-time formating solution. ;) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

I think we don't need a one-size-fits-all all-powerful date-time formating solution. ;)
It's probably impossible to create one, but I think there's also no reason to require people to invent the wheel over and over. I've seen enough broken code attempting to do date/time formatting that I strongly prefer the creation of a few standard solutions that will work for most people, rather than only giving people the low-level bits to work with. Another thing to consider is that for most apps, the choice of the date/time format should be taken out of the hands of the programmer and placed into the hands of the user, through some kind of preference setting. I18n and L10n also strongly suggests to take this route. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
FWIW, mxDateTime exposes these values as attributes -- there is no call overhead.
I actually use the attributes quite a bit in my stuff and hardly ever use .strftime(). The mxDateTime is different though, e.g. I sometimes get questions about how to make strftime() output fractions of a seconds (doesn't work, because strftime() doesn't support it).
Depends on the application space. Database applications will call .timetuple() very often and use strftime() hardly ever.
It's not really a question: the year 0 simply does not exist in reality ! (Christians didn't have a 0 available at the time ;-) Still, historic dates are usually referenced by making year 0 == 1 b.c., -1 == 2 b.c., etc. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

FWIW, mxDateTime exposes these values as attributes -- there is no call overhead.
Good, I think this is the way to go. (Of course there will be some C-level call overhead if we make these properties.)
What does a database app with the resulting tuple? --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
Right.
It puts the values into struct fields for year, month, day, etc. (Databases usually avoid using Unix ticks since these cause the known problems with dates prior to 1970) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Hm, I thought that databases have their own date/time types? Aren't these used? --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
At C level, interfacing is usually done using structs (ISO SQL/CLI defines these). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Ah, there's another requirement that we (or at least I) almost forgot. There should be an efficient C-level interface for the abstract date/time type. This stuff is hairier than it seems! I think the main tension is between improving upon the Unix time_t type, and improving upon the Unix "struct tm" type. Improving upon time_t could mean to extend the range beyond 1970-2038, and/or extend the precision to milliseconds or microseconds. Improving upon struct tm is hard (it has all the necessary fields and no others), unless you want to add operations (just add methods) or make the representation more compact (several of the fields can be packed in 4-6 bits each). A third dimension might be to provide better date/time arithmetic, but I'm not sure if there's much of a market for that, given all the fuzzy semantics (leap seconds, differences across DST changes, timezones). --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]
... This stuff is hairier than it seems!
You're just getting your toes wet: it's impossible to make any two of {astronomers, businessfolk, Jim} happy at the same time, even if they're all American and live in the same city.
I'm suprised you say "all the necessary fields", because a tm contains no info about timezone. C99 introduces a struct tmx that does. The initial segment of a struct tmx must be identical to a struct tm, but the meaning of tmx.tm_isdst differs from tm.tm_isdst. tmx.tm_isdst is the positive number of minutes of offset if Daylight Saving Time is in effect, zero if Daylight Saving Time is not in effect, and -1 if the information is not available. Then it adds some fields not present in a struct tm: int tm_version; // version number int tm_zone; // time zone offset in minutes from UTC [-1439, +1439] int tm_leapsecs; // number of leap seconds applied void *tm_ext; // extension block size_t tm_extlen;// size of the extension block The existence of tm_version, tm_ext and tm_extlen can be fairly viewed as a committee's inability to say "no" <wink>.
I don't think we can get off that easy. Time computation is critical for businesses and astronomers, and leap seconds etc are a PITA independent of time computations. Time computations seem to me to be the easiest of all, provided we've already "done something" intelligible about the rest: any calculation boils down to factoring away leap seconds etc in conversion to a canonical form, doing the computing there, then injecting leap seconds etc back in to the result when converting out of canonical form again. The ECMAScript std (nee Javascript) has, I think, a good example of a usable facility that refused to get mired down in impossible details; e.g., it flat-out refuses to recognize leap seconds. mxDateTime is similarly sane, but MAL keeps threatening to flirt with insanity <wink>. BTW, I doubt there'd be any discussion of leap seconds in the C std if some astronomers hadn't been early Unix users. It's never a net win in the end to try to make a scientist happy <0.9 wink>.

Tim Peters wrote:
FYI, mxDateTime test the C lib for leap second support; if leap seconds are used, then it has to support these too in conversions from and to Unix ticks.
What strange about leap seconds is that they don't fit well with the idea of counting seconds since some fixed point in history. They are only useful for conversions from this count to a broken down date and time representation.... time simply doesn't leap.

Since AFAIK POSIX doesn't admit the existence of leap seconds, how do you ask the C library for leap seconds?
Yeah, we learned that the hard way by adding complex numbers. :-)
I think an important (but so far unvoiced) requirement is that datetime objects can be stored in a database. Since the database may be read by systems that may or may not support leap seconds, we should be independent of the leap second support in the C library. As I've said before, we should ignore leap seconds. Even if we end up expressing times deltas as a number of seconds, that should be understood to be calendar seconds and not astronomical seconds. Let the astronomers deal with leap seconds themselves -- they should know how to. BTW, this means that we can't use the C calls mktime(), timegm(), localtime(), and gmtime(), or their Python wrappers in the time module! That's fine by me. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
See below (the quoted C comment).
SQL databases don't deal with leap seconds. They store the broken down value (in some way) without time zone information and that's it, fortunately :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

On Wed, 27 Feb 2002, M.-A. Lemburg wrote:
Er... SQL99 (and I believe SQL92) have native support for time with and without time zones, and neither say nothing about how databases are to "store" those values. I don't have a copy in front of me, so I can't tell you what they say about leap-seconds. Of course, few implementations support this yet, though it worth being forward-looking. For my own uses, I have a base time class that encapsulates either mxDateTime objects or unix time-since-epoch, and implements the basic time and date accessors and simple arithmetic. A subclass of that type then adds awareness of timezones and daylight savings time. My first effort at trying to do all of those things in one big monolithic class was a nightmare. This layering does result in some (relative) inefficiency, but correctness and maintainability is vastly more important to me. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

Kevin Jacobs wrote:
True, SQL-92 defines data types "TIME WITH TIME ZONE" and "TIMESTAMP WITH TIME ZONE". The standard is only available as book, but here's a draft which has all the details: http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt Still, only Oracle and PostgreSQL seem to actually implement these and ODBC (SQL/CLI), the defacto standard for database interfacing, doesn't even provide interfaces to query or store time zone information (you can put the information directly in the SQL string, but not use it in bound variables). Basically, you should not store local time in databases, but instead use UTC. If you need the original time zone information for reference, you'd keep this in separate DB columns (e.g. as strings). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

On Wed, 27 Feb 2002, M.-A. Lemburg wrote:
Strangely enough I use TIMESPAMP WITH TIMEZONE quite a bit on both Oracle and PostgreSQL using native drivers. I'm also fairly sure that Sybase and MS-SQL store timestamps with timezone somehow, though my memory on the project that did so is a little fuzzy.
Why not minute offset from UTC like C99? Anyhow, everyone knows that time zones and daylight savings time are a pain to deal with. However, lets provide work toward a sane implementation that can relieve the end-user from having to smack their head against this particular brick wall every time. (even if it means smacking our collective heads against the brick wall until we're happy, or reduced to unintelligible ranting, or possibly both). Regards, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
participants (14)
-
aahz@rahul.net
-
Fred L. Drake, Jr.
-
Fredrik Lundh
-
Guido van Rossum
-
Jack Jansen
-
Jeremy Hylton
-
Jim Fulton
-
Kevin Jacobs
-
M.-A. Lemburg
-
Paul Barrett
-
Skip Montanaro
-
Tim Peters
-
Tim Peters
-
Tim Peters