"M.-A. Lemburg" wrote:

In order to make mxDateTime subtypes of this new type we'd need to make sure that the datetime type uses a true struct subset of what I have in DateTime objects now:

typedef struct { PyObject_HEAD

/* Representation used to do calculations */ long absdate; /* number of days since 31.12. in the year 1 BC calculated in the Gregorian calendar. */ double abstime; /* seconds since 0:00:00.00 (midnight) on the day pointed to by absdate */

...lots of broken down values needed to assure roundtrip safety...

}

Depending on the size of PyObject_HEAD, this should meet Jim Fultons requirements (the base type would of course not implement the "..." part :-).

I forgot to mention that there is another object type in mxDateTime too: DateTimeDelta. That's the type needed to represent the time difference between two DateTime instances, or what people usually call "time" :-) It has the following type "signature": typedef struct { PyObject_HEAD double seconds; /* number of delta seconds */ ...some broken down values needed to assure roundtrip safety... } -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Fredrik Lundh wrote:

mal wrote:

...

In order to make mxDateTime subtypes of this new type we'd need to make sure that the datetime type uses a true struct subset of what I have in DateTime objects now:

typedef struct { PyObject_HEAD

/* Representation used to do calculations */ long absdate; /* number of days since 31.12. in the year 1 BC calculated in the Gregorian calendar. */ double abstime; /* seconds since 0:00:00.00 (midnight) on the day pointed to by absdate */

...lots of broken down values needed to assure roundtrip safety...

}

as Tim has pointed out, what I have in mind is:

typedef struct { PyObject_HEAD /* nothing here: subtypes should implement timetuple and, if possible, utctimetuple */ } basetimeObject;

/* maybe: */

PyObject* basetime_timetuple(PyObject* self, PyObject* args) { PyErr_SetString(PyExc_NotImplementedError, "must override"); return NULL; }

(to adapt mxDateTime, all you should have to do is to inherit from baseObject, and add an alias for your "tuple" method)

Ok. Sounds like you are inventing something like a set of abstract types here. I'm very much +1 on that idea, provided the interfaces we define for these types are simple enough (I think the DB SIG has shown that simple interface can go a looong way). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

[Tim]

Are you sure Jim is looking to replace the TimeStamp object? All the complaints I've seen aren't about the relatively tiny TimeStamp object, but about Zope's relatively huge DateTime class (note that you won't have source for that if you're looking at a StandaloneZODB checkout -- DateTime is used at higher Zope levels), which is a Python class with a couple dozen(!) instance attributes. See, e.g.,

http://dev.zope.org/Wikis/DevSite/Proposals/ReplacingDateTime

It seems clear from the source code that TimeStamp is exactly what Jim intended it to be <wink>.

I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object. I think the mistake of DateTime is that it stores the broken-out info, rather than computing it on request.

...

Your idea of a base type (which presumably standarizes at least one form of representation) sounds like a breakthrough that can help satisfy different other needs.

Best I can make out, /F is only proposing what Jim would call an Interface: the existence of two methods, timetuple() and utctimetuple(). In a comment on his page, /F calls it an "abstract" base class, which is more C++-ish terminology, and the sample implementation makes clear it's a "pure" abstract base class, so same thing as a Jim Interface in the end.

I'll show the PEP to Jim when it appears. --Guido van Rossum (home page: http://www.python.org/~guido/)

...

...

...

...

"TP" == Tim Peters writes:

TP> [Guido]

...

I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object.

TP> Given that most mallocs align to 8-byte boundaries these days TP> (also true of pymalloc), it's impossible in reality to define a TP> smaller object than TimeStamp, provided it needs at least one TP> byte of info beyond PyObject_HEAD. Also, it may not be necessary to have a TimeStamp object in ZODB 4. There are three uses for the timestamp: tracking how recently an object was used for cache evication, providing a last modified time to users, and as a simple version number. In ZODB 4, the cache eviction may be done quite differently. The version number may be a simple int. The last mod time will not be provided for each object; instead, users will need to define this themselves if they care about it. If they define it themselves, they'd probably use a DateTime object, but we'd care much less about how small it is. Jeremy

Guido van Rossum writes:

In that case, I take back everything I've said about Jim Fulton's requirements. I'm quite sure that in the past he said he needed a very lightweight date/time object, but from what you say it appears this need has disappeared.

He wanted this for the catalog, and I suspect he still does. Both size and performance (of comparisons) were important, not rendering time. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

Guido van Rossum writes:

Is comparison the same what Tim mentioned as range searches? I guess a representation like current Zope timestamps or what time.time() returns is fine for that -- it is monononous even if not necessarily continuous. I guess a broken-out time tuple is much harder to compare.

Yes; as long as ordering is easy to check, we're fine with a long int or some such thing. The range search is indeed the specific application Jim has in mind. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

[M.-A. Lemburg]

Uhm... I think this thread is heading in the wrong direction.

Maybe from your POV, but from our POV the only way we can get time to work on Python is talk all of you into doing Zope work for Jim <wink>.

Fredrik wasn't proposing a solution to Jim's particular problem (whatever it was ;-), but instead opting for a solution of a large number of Python users out there.

I believe all /F is asking for is that all datetime types supply two specific methods, so that he can get the year etc out of anybody's datetime object via a uniform spelling. It's a fine idea.

... I would be willing to make the mxDateTime types subtypes of whatever Fredrik comes up with. The only requirement I have is that the binary footprint of the types needs to match todays layout of mxDateTime types since I need to maintain binary compatibility.

If /F is asking more than that datetime types implement a specific interface, he's got some major rewriting to do <wink>. Python doesn't have a good way to spell "interface" now, so think of it as a do-nothing base class, inheriting from which means absolutely nothing except that (a) you promise to supply the methods /F specified, and (b) /F can use isinstance to determine whether or not a given object supports this interface.

The other possibility would be adding a set of new types to mxDateTime which focus on low memory requirements rather than data roundtrip safety and speed.

That's getting back to what Jim wants. Maybe someone should ask him what that is <wink>.

[Jeremy Hylton]

Also, it may not be necessary to have a TimeStamp object in ZODB 4. There are three uses for the timestamp: tracking how recently an object was used for cache evication, providing a last modified time to users, and as a simple version number.

In ZODB 4, the cache eviction may be done quite differently. The version number may be a simple int.

WRT RAM usage, a Python int is no smaller than a TimeStamp object. An int pickle is likely much smaller, though.

The last mod time will not be provided for each object; instead, users will need to define this themselves if they care about it. If they define it themselves, they'd probably use a DateTime object, but we'd care much less about how small it is.

Unclear that we'd care less, if the catalog remains full of DateTime objects, and Fred is channeling more faithfully than the rest of us <wink>.

[Tim]

WRT RAM usage, a Python int is no smaller than a TimeStamp object.

[Guido[

Wrong, unless TimeStamps also use a custom allocator.

Good point, and it doesn't (it uses PyObject_NEW). I don't think counting fractions of bytes is of great interest here, though, since I (still) believe it's the massive Zope DateTime type that's the focus of complaints.

The custom allocator uses 12 bytes per int (on a 32-bit machine) and incurs malloc overhead + 8 bytes of additional overhead for every 82 ints. That's about 12.2 bytes per int object; using malloc it would probably be 24 bytes. (PyMalloc would probably do a little better, except it would still round up to 16 bytes.)

pymalloc overhead is a few percent; would work out to 16+f bytes per int object, for some f < 1.0. A difference is that "total memory dedicated to ints" never shrinks using the custom allocator, but can get reused for other objects under pymalloc.

Guido van Rossum wrote:

[Tim]

...

Are you sure Jim is looking to replace the TimeStamp object? All the complaints I've seen aren't about the relatively tiny TimeStamp object, but about Zope's relatively huge DateTime class (note that you won't have source for that if you're looking at a StandaloneZODB checkout -- DateTime is used at higher Zope levels), which is a Python class with a couple dozen(!) instance attributes. See, e.g.,

http://dev.zope.org/Wikis/DevSite/Proposals/ReplacingDateTime

It seems clear from the source code that TimeStamp is exactly what Jim intended it to be <wink>.

I'm notoriously bad at channeling Jim. Nevertheless, I do recall him saying he wanted a lightweight time object. I think the mistake of DateTime is that it stores the broken-out info, rather than computing it on request.

I don't think the mistake was so much to store broken-out info, but to store too much data in general. It stores redundant data, which makes it's implementation a bit difficult to understand and maintain. Note that scalability was not a goal of Zope's DateTime type. We meant to replace it with something much tighter a long time ago, but never got around to it.

...

...

Your idea of a base type (which presumably standarizes at least one form of representation) sounds like a breakthrough that can help satisfy different other needs.

Best I can make out, /F is only proposing what Jim would call an Interface: the existence of two methods, timetuple() and utctimetuple(). In a comment on his page, /F calls it an "abstract" base class, which is more C++-ish terminology, and the sample implementation makes clear it's a "pure" abstract base class, so same thing as a Jim Interface in the end.

Yup.

I'll show the PEP to Jim when it appears.

Thanks. ;) BTW, has there been any progress on this? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Fredrik Lundh wrote:

jim wrote:

...

BTW, has there been any progress on this?

the current pepoid proposal is here:

http://effbot.org/ideas/time-type.htm

major issues right now is 1. should the type know about timezones (probably not),

:( -1 Doesn't the proposal sort of imply time-zone awareness of some kind? Or does it simply imply UT storage?

and 2. should it support basic arithmetics (probably yes).

Does this imply leap second hell, or will we simply be vague about expectations? I'd also like to see simple access methods for year, month, day, hours, minutes, and seconds, with date parts being one based and time parts being zero based. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

"M.-A. Lemburg" wrote:

Jim Fulton wrote:

...

...

major issues right now is 1. should the type know about timezones (probably not),

:(

-1

Doesn't the proposal sort of imply time-zone awareness of some kind? Or does it simply imply UT storage?

I tried that in early version of mxDateTime -- it fails badly. I switched to the local time assumption very early in the development.

That's a really bad assumption if times are used in different time zones.

Note that Fredrik's type is an abstract type; it doesn't even store anything -- that's up to subtypes which of course can implement timezones at their liking.

No, it does store something, it just doesn't say how. There are methods for returning localtime and gmtime, so there is (abstract) storage.

...

...

and 2. should it support basic arithmetics (probably yes).

Does this imply leap second hell, or will we simply be vague about expectations?

The type will store a fixed point in time, so why worry about leap seconds (most system's don't support these anyway and if they do, the support is usually switched off per default) ?

There are a lot of semantic issues with date-time math. Leap seconds is an example. If you store local time, are date-time subtractions affected by daylight-savings time? Do the calculations depend on the calendar? Do you take into account the lost days in the switch from the Julean to the Gregorian calendar? I'm not really opposed to doing math, but we need to at least recognize the fuzzyness of the semantics.

...

I'd also like to see simple access methods for year, month, day, hours, minutes, and seconds, with date parts being one based and time parts being zero based.

In the abstract base type ?

Yes. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Fredrik Lundh wrote:

mal wrote:

...

...

Doesn't the proposal sort of imply time-zone awareness of some kind? Or does it simply imply UT storage?

I tried that in early version of mxDateTime -- it fails badly.

can you elaborate?

First of all, the C lib only support UTC and local time, so you don't really have a chance of correctly converting a non-local time using a different time zone in either local time or UTC: there simply are no C APIs you could use and the problems which DST and leap seconds introduces are no fun at all (they are fun to read though: figuring out the various DST switch times is an adventure -- just have a look at the C lib's DST files). The next problem is that the C lib only provides APIs for conversion from local time to UTC, but not UTC to local time. There is an API called timegm() for this on some platforms, but its non-standard. As a result, making UTC the default won't allow you to safely represent the datetime value in local time. A third obstacle is typcial user assumptions: users simply assume local time and it's hard to tell them otherwise (people have very personal feelings about date and time for some reason...).

...

...

Does this imply leap second hell, or will we simply be vague about expectations?

The type will store a fixed point in time, so why worry about leap seconds (most system's don't support these anyway and if they do, the support is usually switched off per default) ?

the updated proposal adds __hash__ and __cmp__, and the following (optional?) operations:

deltaobject = timeobject - timeobject floatobject = float(deltaobject) # fractional seconds timeobject = timeobject + integerobject timeobject = timeobject + floatobject timeobject = timeobject + deltaobject

note that "deltaobject" can be anything; the abstract type only says that if you manage to subtract one time object from another one of the same type, you get some object that you can 1) convert to a float, and 2) add to another time object.

vague, but pretty useful.

Indeed :-)

...

...

I'd also like to see simple access methods for year, month, day, hours, minutes, and seconds, with date parts being one based and time parts being zero based.

In the abstract base type ?

Q. does mxDateTime provide separate accessors for individual members?

Yes, it provides access to these in form of attributes. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Guido van Rossum wrote:

...

Does this imply leap second hell, or will we simply be vague about expectations?

IMO, leap seconds should be ignored. Time stands still during a leap second. Consider this a BDFL pronouncement if you wish. :-)

...

I'd also like to see simple access methods for year, month, day, hours, minutes, and seconds,

The timetuple() method provides access to all of these simultaneously. Isn't that enough?

Guido van Rossum wrote:

...

...

The timetuple() method provides access to all of these simultaneously. Isn't that enough?

From a minimalist point of view, yet, but from a usability point of view, no.

...

t.year() could be spelled as t.timetuple()[0].

Yes, but t.year() is a lot more readable.

When do you ever use this in isolation? I'd expect in 99% of the cases you hand it off to a formatting routine, and guess what -- strftime() takes a time tuple.

I worry about the time wasted by calling all of t.year(), t.month(), t.day() (etc.) -- given that they do so little, the call overhead is probably near 100%.

I wonder how often this is needed. The only occurrences of year() in the entire Zope source that I found are in various test routines.

These methods and others are used a lot in presentation code, which tends to be expressed in DTML or ZPT. It's not uncommon to select/catagorize things by year or month. I think most people would find individual date-part methods a lot more natural than tuples.

...

...

I expect that usually you'd request several of these together anyway, in order to do some fancy formatting, so the timetuple() approach makes sense.

I find the time tuples to be really inconvenient. I *always* have to slice off the parts I don't want, which I find annoying.

Serious question: what do you tend to do with time values?

I format them in various ways and I sort them.

I imagine that once we change strftime() to accept an abstract time object, you'll never need to call either timetuple() or year() -- strftime() will do it for you.

Maybe, if I use strftime, but I don't use strftime all that much. I can certainly think of even formatting cases (e.g. internationalized dates) where it's not adequate.

...

Hm, now that I mention the extra parts, it seems kind of silly to make implementors of the type come up with weekday, julian day, and a daylight-savings flag. This time format is really biased by the C time library, which is fine for a module that wraps the C library but seems a bit silly for a standard date-time interface.

That's why /F's pre-PEP allows the implementation to leaves these three set to -1.

I missed these. Still, providing -1s seems, uh, vestigial.

...

...

...

with date parts being one based and time parts being zero based.

I'm not sure what you mean here.

Years, months, and days should start from 1. Hours, minutes and seconds should start from 0.

One confusion I often have with time tuples is that I know too much about C's time struct, which numbers months from 0 and which has years since 1900.

I guess that confusion is yours alone. In Python, of course month and day start from 1. Whether years start from 1 is a theological question. :-)

I doubt the confusion is mine alone, but I'll take your word for it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Guido van Rossum wrote:

[me]

...

...

I wonder how often this is needed. The only occurrences of year() in the entire Zope source that I found are in various test routines.

[Jim]

...

These methods and others are used a lot in presentation code, which tends to be expressed in DTML or ZPT.

It's not uncommon to select/catagorize things by year or month.

I think most people would find individual date-part methods a lot more natural than tuples.

OK, that explains a lot. For this context I agree, although I think they should probably appear as (computed) attributes rather than methods. Properties seem perfect.

That's fine with me.

...

...

I imagine that once we change strftime() to accept an abstract time object, you'll never need to call either timetuple() or year() -- strftime() will do it for you.

Maybe, if I use strftime, but I don't use strftime all that much.

Maybe you should. :-)

I do when I can. But it often doesn't meet my needs.

...

I can certainly think of even formatting cases (e.g. internationalized dates) where it's not adequate.

Then a super-strftime() should be invented that *is* enough, rather than fumbling with hand-coded solutions.

I think we don't need a one-size-fits-all all-powerful date-time formating solution. ;) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Guido van Rossum wrote:

...

...

The timetuple() method provides access to all of these simultaneously. Isn't that enough?

From a minimalist point of view, yet, but from a usability point of view, no.

...

t.year() could be spelled as t.timetuple()[0].

Yes, but t.year() is a lot more readable.

When do you ever use this in isolation? I'd expect in 99% of the cases you hand it off to a formatting routine, and guess what -- strftime() takes a time tuple.

FWIW, mxDateTime exposes these values as attributes -- there is no call overhead.

I worry about the time wasted by calling all of t.year(), t.month(), t.day() (etc.) -- given that they do so little, the call overhead is probably near 100%.

I wonder how often this is needed. The only occurrences of year() in the entire Zope source that I found are in various test routines.

I actually use the attributes quite a bit in my stuff and hardly ever use .strftime(). The mxDateTime is different though, e.g. I sometimes get questions about how to make strftime() output fractions of a seconds (doesn't work, because strftime() doesn't support it).

...

...

I expect that usually you'd request several of these together anyway, in order to do some fancy formatting, so the timetuple() approach makes sense.

I find the time tuples to be really inconvenient. I *always* have to slice off the parts I don't want, which I find annoying.

Serious question: what do you tend to do with time values? I imagine that once we change strftime() to accept an abstract time object, you'll never need to call either timetuple() or year() -- strftime() will do it for you.

Depends on the application space. Database applications will call .timetuple() very often and use strftime() hardly ever.

...

...

...

with date parts being one based and time parts being zero based.

I'm not sure what you mean here.

Years, months, and days should start from 1. Hours, minutes and seconds should start from 0.

One confusion I often have with time tuples is that I know too much about C's time struct, which numbers months from 0 and which has years since 1900.

I guess that confusion is yours alone. In Python, of course month and day start from 1. Whether years start from 1 is a theological question. :-)

It's not really a question: the year 0 simply does not exist in reality ! (Christians didn't have a 0 available at the time ;-) Still, historic dates are usually referenced by making year 0 == 1 b.c., -1 == 2 b.c., etc. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Guido van Rossum wrote:

...

FWIW, mxDateTime exposes these values as attributes -- there is no call overhead.

Good, I think this is the way to go. (Of course there will be some C-level call overhead if we make these properties.)

Right.

...

...

Serious question: what do you tend to do with time values? I imagine that once we change strftime() to accept an abstract time object, you'll never need to call either timetuple() or year() -- strftime() will do it for you.

Depends on the application space. Database applications will call .timetuple() very often and use strftime() hardly ever.

What does a database app with the resulting tuple?

It puts the values into struct fields for year, month, day, etc. (Databases usually avoid using Unix ticks since these cause the known problems with dates prior to 1970) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Guido van Rossum wrote:

...

...

What does a database app with the resulting tuple?

It puts the values into struct fields for year, month, day, etc. (Databases usually avoid using Unix ticks since these cause the known problems with dates prior to 1970)

Hm, I thought that databases have their own date/time types? Aren't these used?

At C level, interfacing is usually done using structs (ISO SQL/CLI defines these). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

[Guido]

... This stuff is hairier than it seems!

You're just getting your toes wet: it's impossible to make any two of {astronomers, businessfolk, Jim} happy at the same time, even if they're all American and live in the same city.

I think the main tension is between improving upon the Unix time_t type, and improving upon the Unix "struct tm" type. Improving upon time_t could mean to extend the range beyond 1970-2038, and/or extend the precision to milliseconds or microseconds.

Improving upon struct tm is hard (it has all the necessary fields and no others), unless you want to add operations (just add methods) or make the representation more compact (several of the fields can be packed in 4-6 bits each).

I'm suprised you say "all the necessary fields", because a tm contains no info about timezone. C99 introduces a struct tmx that does. The initial segment of a struct tmx must be identical to a struct tm, but the meaning of tmx.tm_isdst differs from tm.tm_isdst. tmx.tm_isdst is the positive number of minutes of offset if Daylight Saving Time is in effect, zero if Daylight Saving Time is not in effect, and -1 if the information is not available. Then it adds some fields not present in a struct tm: int tm_version; // version number int tm_zone; // time zone offset in minutes from UTC [-1439, +1439] int tm_leapsecs; // number of leap seconds applied void *tm_ext; // extension block size_t tm_extlen;// size of the extension block The existence of tm_version, tm_ext and tm_extlen can be fairly viewed as a committee's inability to say "no" <wink>.

A third dimension might be to provide better date/time arithmetic, but I'm not sure if there's much of a market for that, given all the fuzzy semantics (leap seconds, differences across DST changes, timezones).

I don't think we can get off that easy. Time computation is critical for businesses and astronomers, and leap seconds etc are a PITA independent of time computations. Time computations seem to me to be the easiest of all, provided we've already "done something" intelligible about the rest: any calculation boils down to factoring away leap seconds etc in conversion to a canonical form, doing the computing there, then injecting leap seconds etc back in to the result when converting out of canonical form again. The ECMAScript std (nee Javascript) has, I think, a good example of a usable facility that refused to get mired down in impossible details; e.g., it flat-out refuses to recognize leap seconds. mxDateTime is similarly sane, but MAL keeps threatening to flirt with insanity <wink>. BTW, I doubt there'd be any discussion of leap seconds in the C std if some astronomers hadn't been early Unix users. It's never a net win in the end to try to make a scientist happy <0.9 wink>.

FYI, mxDateTime test the C lib for leap second support; if leap seconds are used, then it has to support these too in conversions from and to Unix ticks.

Since AFAIK POSIX doesn't admit the existence of leap seconds, how do you ask the C library for leap seconds?

...

BTW, I doubt there'd be any discussion of leap seconds in the C std if some astronomers hadn't been early Unix users. It's never a net win in the end to try to make a scientist happy <0.9 wink>.

Yeah, we learned that the hard way by adding complex numbers. :-)

What strange about leap seconds is that they don't fit well with the idea of counting seconds since some fixed point in history. They are only useful for conversions from this count to a broken down date and time representation.... time simply doesn't leap.

...

From a comment in mxDateTime: /* This function checks whether the system uses the POSIX time_t rules (which do not support leap seconds) or a time package with leap second support enabled. Return 1 if it uses POSIX time_t values, 0 otherwise.

POSIX: 1986-12-31 23:59:59 UTC == 536457599

With leap seconds: == 536457612

(since there were 13 leap seconds in the years 1972-1985 according to the tz package available from ftp://elsie.nci.nih.gov/pub/)

*/

I think an important (but so far unvoiced) requirement is that datetime objects can be stored in a database. Since the database may be read by systems that may or may not support leap seconds, we should be independent of the leap second support in the C library. As I've said before, we should ignore leap seconds. Even if we end up expressing times deltas as a number of seconds, that should be understood to be calendar seconds and not astronomical seconds. Let the astronomers deal with leap seconds themselves -- they should know how to. BTW, this means that we can't use the C calls mktime(), timegm(), localtime(), and gmtime(), or their Python wrappers in the time module! That's fine by me. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wed, 27 Feb 2002, M.-A. Lemburg wrote:

SQL databases don't deal with leap seconds. They store the broken down value (in some way) without time zone information and that's it, fortunately :-)

Er... SQL99 (and I believe SQL92) have native support for time with and without time zones, and neither say nothing about how databases are to "store" those values. I don't have a copy in front of me, so I can't tell you what they say about leap-seconds. Of course, few implementations support this yet, though it worth being forward-looking. For my own uses, I have a base time class that encapsulates either mxDateTime objects or unix time-since-epoch, and implements the basic time and date accessors and simple arithmetic. A subclass of that type then adds awareness of timezones and daylight savings time. My first effort at trying to do all of those things in one big monolithic class was a nightmare. This layering does result in some (relative) inefficiency, but correctness and maintainability is vastly more important to me. -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

On Wed, 27 Feb 2002, M.-A. Lemburg wrote:

True, SQL-92 defines data types "TIME WITH TIME ZONE" and "TIMESTAMP WITH TIME ZONE". The standard is only available as book, but here's a draft which has all the details:

http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt

Still, only Oracle and PostgreSQL seem to actually implement these and ODBC (SQL/CLI), the defacto standard for database interfacing, doesn't even provide interfaces to query or store time zone information (you can put the information directly in the SQL string, but not use it in bound variables).

Strangely enough I use TIMESPAMP WITH TIMEZONE quite a bit on both Oracle and PostgreSQL using native drivers. I'm also fairly sure that Sybase and MS-SQL store timestamps with timezone somehow, though my memory on the project that did so is a little fuzzy.

Basically, you should not store local time in databases, but instead use UTC. If you need the original time zone information for reference, you'd keep this in separate DB columns (e.g. as strings).

Why not minute offset from UTC like C99? Anyhow, everyone knows that time zones and daylight savings time are a pain to deal with. However, lets provide work toward a sane implementation that can relieve the end-user from having to smack their head against this particular brick wall every time. (even if it means smacking our collective heads against the brick wall until we're happy, or reduced to unintelligible ranting, or possibly both). Regards, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

[me]

...

Improving upon struct tm is hard (it has all the necessary fields and no others), unless you want to add operations (just add methods) or make the representation more compact (several of the fields can be packed in 4-6 bits each).

[Tim]

I'm suprised you say "all the necessary fields", because a tm contains no info about timezone.

Oops. My mistake. I thought it had timezone. [tmx details snipped]

...

A third dimension might be to provide better date/time arithmetic, but I'm not sure if there's much of a market for that, given all the fuzzy semantics (leap seconds, differences across DST changes, timezones).

I don't think we can get off that easy. Time computation is critical for businesses and astronomers, and leap seconds etc are a PITA independent of time computations. Time computations seem to me to be the easiest of all, provided we've already "done something" intelligible about the rest: any calculation boils down to factoring away leap seconds etc in conversion to a canonical form, doing the computing there, then injecting leap seconds etc back in to the result when converting out of canonical form again.

The ECMAScript std (nee Javascript) has, I think, a good example of a usable facility that refused to get mired down in impossible details; e.g., it flat-out refuses to recognize leap seconds. mxDateTime is similarly sane, but MAL keeps threatening to flirt with insanity <wink>.

BTW, I doubt there'd be any discussion of leap seconds in the C std if some astronomers hadn't been early Unix users. It's never a net win in the end to try to make a scientist happy <0.9 wink>.

I'd be happy to support time computations, provided we keep the leap seconds out. I propose a representation that resembles a compressed struct tm (or tmx), with appropriately-sized bit fields for year, month, day, hour, minute, second, millisecond, and microsecond, and timezone and DST info. Since the most likely situation is extraction in local time, these should be stored as local time with an explicit timezone. (I don't want to store these things in a database without an explicit timezone, even if it costs another 12 bit field.) An app extracting the local time without checking the timezone could be fooled by a time stored with a different timezone. Do we care? Time computations are only slightly complex because they have to be calendar-aware, but at least they don't have to be DST-aware -- they can just thake the timezone offset in minutes and apply it. The DST info should probably be two bits: one telling whether DST is in effect at the given time, one telling whether DST is honored in the given timezone. Maybe it should also allow "missing info" for either. Details, details. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wednesday, February 27, 2002, at 03:21 , M.-A. Lemburg wrote:

The discussion is going astray again: Fredrik proposed an abstract base type, i.e. a type providing only the name and an interface which is defined as convention.

I am all for adding such an abstract base type (and others as well, e.g. for numbers, sequences, money, decimal, etc.) with minimal interfaces, but not for fixing a complex interface on top of these.

Oops, I had missed that bit as well, that adding an *abstract* base type was the intention. I'm all for that as well. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Guido> The timetuple() method provides access to all of these Guido> simultaneously. Isn't that enough? t.year() could be spelled as Guido> t.timetuple()[0]. Since we're discussing an abstract type it probably doesn't apply directly, but perhaps timetuple() could be specified to return a super-tuple like os.stat() does... Skip

*all* early reviewers argued that time zones are a representation thingie, and doesn't belong in the abstract type.

Then maybe the reviewers didn't have a sufficiently wide range of applications in mind? If I were to create a database of email messages, I'd be seriously annoyed if it normalized the timezone info away. A message sent at 10pm EST has a different feel to it than one sent at 4am MET. It should *sort* on UTC, but it should use the original timezone to display the dates. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum writes:

...

10pm EST has a different feel to it than one sent at 4am MET. It should *sort* on UTC, but it should use the original timezone to display the dates.

[Fred]

Sounds like a user preference, not a universal truth.

Fair enough. Even some of my well-traveled friends cannot do timezone arithmetic in their head... :-)

Is it important that the timezone is part of the date/time type, though? Is it important that it be part of the abstract base date/time?

Specific implementations should certainly be able to add support for timezones, and perhaps some hypothetical default date/time type should include it for convenience, but that doesn't tell me it's fundamental.

I guess I want it to be possible to have an implementation that keeps track of the timezone as entered. It's true that time deltas are a nightmare when dealing with different timezones. --Guido van Rossum (home page: http://www.python.org/~guido/)

oops. two lines of code, and one bug. should be:

if isinstance(v, basemoney): # yay! it's money print float(v) # let's hope it's not too much

</F>

Guido van Rossum wrote:

...

I propose adding an "abstract" money base type to the standard library, to be subclassed by real money/decimal implementations.

Why do we need this? I guess that would be Question #1...

For databases ?! The DB API has long had a monetary or at least decimal type on its plate... never got around to implementing one, though :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Guido van Rossum wrote:

...

...

...

I propose adding an "abstract" money base type to the standard library, to be subclassed by real money/decimal implementations.

Why do we need this? I guess that would be Question #1...

For databases ?! The DB API has long had a monetary or at least decimal type on its plate... never got around to implementing one, though :-)

I can only find one reference to money or decimal in the DB API PEP, and that's as a future task. I guess that's what you mean by "on its plate".

Exactly :-)

Since I'm not a database expert, maybe you can explain the use of this in more detail? And why would we need a monetary type rather than a fixed-point decimal type?

A decimal would do as well, I suppose, at least in terms of storing the raw value. The reason for trying to come up with a monetary type is to make operations between monetary values having two different currencies illegal. Coercion between two of those would always have to be made explicit (for obvious reasons). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Guido van Rossum wrote:

...

A decimal would do as well, I suppose, at least in terms of storing the raw value. The reason for trying to come up with a monetary type is to make operations between monetary values having two different currencies illegal. Coercion between two of those would always have to be made explicit (for obvious reasons).

Are you sure you're trying to solve a real problem here? There are lots of operations on monetary values that make no sense (try multiplying two amounts of money).

Indeed, monetary types solve different problems than decimal types. Financial applications do have a need for these kind of implicit error checks. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Guido van Rossum wrote:

...

Indeed, monetary types solve different problems than decimal types. Financial applications do have a need for these kind of implicit error checks.

But this is easily done by creating a custom class -- which has the advantage that the set of constraints can be specialized to the needs of a specific application. When we add a monetary type to the language we'll never get it right for all apps. OTOH, I think we could get a fixed point type right.

True. A real implementation of a good working decimal type with adjustable rounding rules would certainly go a long way and the money type could be built on top of it.

What support for money does SQL have?

SQL-92 doesn't have support for it, but some modern database engines do, e.g. MS SQL Server, PostgreSQL (even though it's deprecated there, now), MS Access. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Fredrik Lundh wrote:

and while I'm at it:

I propose adding an "abstract" money base type to the standard library, to be subclassed by real money/decimal implementations.

if isinstance(v, basemoney): # yay! it's money print float(money) # let's hope it's not too much

The goal is not to standardize any behaviour beyond this; anything else should be provided by subtypes.

More details here:

http://effbot.org/ideas/money-type.htm

I can produce PEP and patch if necessary.

Sounds like a plan. One thing though: the RE "[+|-]?\d+(.\d+)?" should be extended to allow for currency symbols and names in front or after the monetary value. Currency for money is a bit like timezones for datetime, so it's a good idea, not to add it to the base type interface. However, the interface should be extendable to include currency information. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/

Currency for money is a bit like timezones for datetime, so it's a good idea, not to add it to the base type interface. However, the interface should be extendable to include currency information.

Currency is much worse than timezones -- once you are interested in exchange rates, you need to know *when* to calculate the exchange rate (as well as other parameters such as which exchange rate). So please let's keep the currency out of the money type; it's utterly application dependent what to do with that information. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:

...

http://effbot.org/ideas/time-type.htm

I can produce PEP and patch if necessary.

Yes, a PEP, please! Jim Fulton has been asking for this for a long time too. His main requirement is that timestamp objects are small, both in memory and as pickles, because Zope keeps a lot of these around.

I also need time-zone support.

They are currently represented either as long ints (with a little under 64 bits) or as 8-byte strings.

ZODB has a TimeStamp type that uses a 32-bit unsigned integer to store year, month,, day, hour, and minute in a way that makes it dirt simple to extract a component. It uses a 32-bit integer to store seconds in units of 60/2**32 seconds. This type isn't appropriate for general use because it only allows dates later than Dec 31, 1899.

A dedicated timestamp object could be smaller than that.

A type that only needed minute precision could easily be expressed with 32-bits. Of course, the two-word object overhead makes the difference between 32-bits and 64-bits rather unexciting.

Your idea of a base type (which presumably standarizes at least one form of representation) sounds like a breakthrough that can help satisfy different other needs.

I agree. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Tim Peters wrote:

[Jim Fulton]

...

ZODB has a TimeStamp type that uses a 32-bit unsigned integer to store year, month,, day, hour, and minute in a way that makes it dirt simple to extract a component.

You really think so? It's a mixed-radix scheme:

v=((((y-1900)*12+mo-1)*31+d-1)*24+h)*60+m;

so requires lots of expensive integer division and remainder operations to pick apart again (the trend in CPUs is to make these relatively more expensive, not less, and e.g. Itanium doesn't even have an integer division instruction).

Compared to storing date-times as offsets from an epoch, this is much simpler and cheaper.

If we had this to do over again, I'd strongly suggest assigning 12 bits to the year, 4 to the month, 5 each to day and hour, and 6 to the minute. The components would then be truly dirt simple and dirt cheap to extract, and we wouldn't even have to bother switching between 0-based and 1-based for the months and days (let 'em stay 1-based). They would still sort and compare correctly in packed format. The only downside I can see is that not pursuing every last drop of potential compression would shrink the dynamic range from 8000+ years to 4000+ years, but we're likely to have much worse problems in Zope by the year 5900 anyway <wink>.

Sounds good to me. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org