Oh yeah - please hold off on the release23-maint branch for a bit, in case Jack wants to cut a 2.3.1 release for the Mac... Jack? Anthony -- Anthony Baxter It's never too late to have a happy childhood.

On woensdag, sep 24, 2003, at 10:15 Europe/Amsterdam, Anthony Baxter wrote:

Oh yeah - please hold off on the release23-maint branch for a bit, in case Jack wants to cut a 2.3.1 release for the Mac... Jack?

I don't know whether I'll find the time, give me until the end of the week to decide, please. And off on a tangent: I think we need a python-dev-announce mailing list or something of the sort. I've been busy with Real Work the last month, and I only skim python-dev when I have time, so today is the first time I hear about 2.3.1 (and, being on a train without internet-access right now I can't even test it:-). I'm not happy with this. For one thing, I haven't seen any indication that anyone tried a framework build on MacOSX, and how such a build interacts with the pre-installed 2.3 on Panther. Even though I wouldn't have had time to do any fixes for MacOSX that are on my plate I would at least have tested that 2.3.1 doesn't break anything... I wouldn't mind a checklist of things that need to be tested (plus preferred people responsible for them) before a release goes out. -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

Oops. See python.org/sf/812007 Python 2.3.1 (#1, Sep 24 2003, 15:55:24) [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

from os import fsync Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: cannot import name fsync

...

...

Barry Warsaw wrote IMO, this needs to be added to PEP 102 and there needs to be a list of people who will give a go, no-go on a new release. There needs to be timeouts though because we're all way too busy and it may just be that a release needs to go out before a person on the critical path has a chance to sign off on it.

I'm re-doing PEP 101 and 102 to change the various names to roles. I'll add a section to the top saying "You need to get a person signed up and agreed for each of the following roles. The people who have done each role in the past are..." Anthony -- Anthony Baxter It's never too late to have a happy childhood.

Anthony Baxter writes:

I'm re-doing PEP 101 and 102 to change the various names to roles. I'll add a section to the top saying "You need to get a person signed up and agreed for each of the following roles. The people who have done each role in the past are..."

Make sure you only list the people willing to do it again or at least discuss the role with new volunteers. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

Anthony Baxter writes:

Mea maxima culpa. I'll add a new section to the start of PEP101 and PEP102 about communicating with the appropriate people directly.

We had a bunch of testers try out the release candidates on various Mac OS X releases - they reported no breakages. There's also been few/no checkins to the Mac-specific chunks of the tree for the release23-maint branch.

I'm happy to store up checkins for the 23 branch for a couple of weeks to give you time to do a release if you so wish (consider it penance for my forgetting to email you).

I think in future we should probably do the "2.3.2c1", wait a week, do "2.3.2" thing. It's more work, but there have been enough nits that I think it would have been worthwhile (readline vs. no threads, fsync and the docs issue spring to mind). Cheers, mwh -- 81. In computing, turning the obvious into the useful is a living definition of the word "frustration". -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

...

...

Thomas Heller wrote Michael Hudson writes:

...

I think in future we should probably do the "2.3.2c1", wait a week, do "2.3.2" thing. It's more work, but there have been enough nits that I think it would have been worthwhile (readline vs. no threads, fsync and the docs issue spring to mind).

+1

+1 from me, also. I'll make sure to put words to the effect in the PEPs. -- Anthony Baxter It's never too late to have a happy childhood.

On 25-sep-03, at 5:20, Anthony Baxter wrote:

Mea maxima culpa. I'll add a new section to the start of PEP101 and PEP102 about communicating with the appropriate people directly.

Don't worry: I assume there's no harm done, and otherwise I'll just lobby for a quick 2.3.2.

We had a bunch of testers try out the release candidates on various Mac OS X releases - they reported no breakages. There's also been few/no checkins to the Mac-specific chunks of the tree for the release23-maint branch.

There's a couple of version numbers that would have been nice to update, in the various .plist files. In other words, if you build 2.3.1 from source with a framework build then "Show Info" on the applications will show them to be 2.3, not 2.3.1. Same for the Apple Help Book that is built in to the IDE.

I'm happy to store up checkins for the 23 branch for a couple of weeks to give you time to do a release if you so wish (consider it penance for my forgetting to email you).

No, don't worry. I'm going to skip 2.3.1 for MacPython-OS9. I'll catch up again with 2.3.2. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

"Jason R. Mastaler" writes:

Python 2.3.1 (#1, Sep 24 2003, 15:55:24) [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

...

from os import fsync Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: cannot import name fsync

Why is that a bug? Your system just does not have a declared fsync function. Regards, Martin

...

...

Jack Jansen wrote There's a couple of version numbers that would have been nice to update, in the various .plist files. In other words, if you build 2.3.1 from source with a framework build then "Show Info" on the applications will show them to be 2.3, not 2.3.1. Same for the Apple Help Book that is built in to the IDE.

If you could point to the files, I'll add them to the PEPs. -- Anthony Baxter It's never too late to have a happy childhood.

martin@v.loewis.de (Martin v. Löwis) writes:

Jack Jansen writes:

...

There's a couple of version numbers that would have been nice to update, in the various .plist files. In other words, if you build 2.3.1 from source with a framework build then "Show Info" on the applications will show them to be 2.3, not 2.3.1. Same for the Apple Help Book that is built in to the IDE.

As a procedural note, I would suggest that suggest changes are done immediately *after* the previous release, instead of shortly before the next one. The release branch should be releasable at any time, without having to ask anybody for permission. What gets done is done, what doesn't get done has to wait for the next release. The less coordination, the better.

+1 I guess this should be added to PEP 102, too. Cheers, mwh -- BUGS Never use this function. This function modifies its first argument. The identity of the delimiting character is lost. This function cannot be used on constant strings. -- the glibc manpage for strtok(3)

On Friday, September 26, 2003, at 07:50 AM, A.M. Kuchling wrote:

On Thu, 25 Sep 2003 12:37:08 -0600, Jason R. Mastaler wrote:

...

Will there be a 2.3.2 release, or some other sort of bugfix release to address the problems in 2.3.1?

Another minor nit: the 2.3.1 .tgz file contains the CVS/ subdirectories. This doesn't actually hurt anything, though, just cluttering up the tree a bit.

Was it not made with cvs export? -Barry

...

Another minor nit: the 2.3.1 .tgz file contains the CVS/ subdirectories. This doesn't actually hurt anything, though, just cluttering up the tree a bit.

Was it not made with cvs export?

Ouch. Isn't this in the PEP? The CVS directories *do* hurt, IMO (by confusing people). --Guido van Rossum (home page: http://www.python.org/~guido/)

Anthony Baxter writes:

...

Will there be a 2.3.2 release, or some other sort of bugfix release to address the problems in 2.3.1? I think the os.fsync() problem is serious enough to warrant revoking 2.3.1 and making a new release to replace it rather quickly.

There's a workaround on the 2.3.1 bugs page. The only problems I'm aware of with 2.3.1 are the HP/UX build problems (workaround on bugs.html), the documentation mistakenly identifying itself as '2.4.0alpha', and the os.fsync problem.

There's a workaround, but the problem is that I think I'll be the one bearing the brunt of this bug for who knows how long. I've gotten about a half dozen problem reports about it already on the TMDA lists. TMDA uses os.fsync() in its mbox and Maildir delivery code, so practically every user will find their TMDA broken as soon as they install or upgrade to 2.3.1. Since Python point releases are usually so solid, they don't consider it might be a bug in the interpreter, and assume it's a TMDA bug. TMDA is just one example. Repeat for the other packages out there which rely on os.fsync(). The longer 2.3.1 is out there, the more it will be installed, incorporated into OS package distributions, etc, and the harder this breakage will be to shake.

Barry Warsaw writes:

On Fri, 2003-09-26 at 11:47, Jason R.Mastaler wrote:

...

The longer 2.3.1 is out there, the more it will be installed, incorporated into OS package distributions, etc, and the harder this breakage will be to shake.

It certainly seems reasonable to me to release 2.3.2 fairly quickly.

But not earlier than at least one week after 2.3.2rc1 ;-) Thomas

Jason R. Mastaler wrote:

Thomas Heller writes:

...

...

It certainly seems reasonable to me to release 2.3.2 fairly quickly.

But not earlier than at least one week after 2.3.2rc1 ;-)

Well, if you're going to wait that long, I think you might consider adding some sort of warning to the 2.3.1 download area. My fear is that some large OS vendor (e.g, Redhat) will pick up 2.3.1 for a future release.

Too late: 2.3.1 is in at least some versions of Debian already -- I'm getting reports on the os.fsync() issue already from my users (it breaks getmail). Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with only this change would suit me. Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------

...

...

Barry Warsaw wrote Was it not made with cvs export?

Ouch. Isn't this in the PEP? The CVS directories *do* hurt, IMO (by confusing people).

It's in PEPs 101 and 102.

I'm very very confused, since I _did_ use cvs export. Really. Repeatedly. Ugh. Given this, I'm happy to look at a 2.3.2 in a couple of weeks - I'd like very much to finish the PEP rewrite and get it reviewed first. Arrrrrgh. (no, not talk-like-a-pirate day, frustration) Anthony -- Anthony Baxter It's never too late to have a happy childhood.

...

...

"Jason R. Mastaler" wrote There's a workaround, but the problem is that I think I'll be the one bearing the brunt of this bug for who knows how long. I've gotten about a half dozen problem reports about it already on the TMDA lists. TMDA uses os.fsync() in its mbox and Maildir delivery code, so practically every user will find their TMDA broken as soon as they install or upgrade to 2.3.1. Since Python point releases are usually so solid, they don't consider it might be a bug in the interpreter, and assume it's a TMDA bug. TMDA is just one example. Repeat for the other packages out there which rely on os.fsync().

Hm. I ran the Zope2, Zope3, and Twisted test suites against the release build before the release. I'm open to additional suggestions (I just grabbed tmda, but it doesn't have a test suite). In this case, the problem hopefully would've been picked up with a release candidate, but I'm also happy to add additional slabs of test suites to run against code before it's released... Anthony -- Anthony Baxter It's never too late to have a happy childhood.

...

...

"Tim Peters" wrote That sounds dangerous to me: there's *always* an HP/UX build problem, if you count (as I do) getting threads to work. Every release, some HP/UX user submits a patch that breaks the build for other HP/UX users. So unless we've got a bona fide HP/UX expert on tap now (do we? that would be way cool <wink>), don't hold up solving a clear and widespread Linux problem by waiting for one to appear.

On the scale of "HP/UX build problems we've known and loved", this one's a trivial one. https://sourceforge.net/tracker/?func=detail&atid=105470&aid=811160&group_id=5470 has a patch (from the gcc bugtracker - they ran into the exact same problem) Now, the readline+no-threads=boom coredump, well, that involves threads, and the original reporter even mentions HP/UX, but it's more a simple-ish one - missing #ifdef WITH_THREAD calls, as far as I can see. https://sourceforge.net/tracker/?func=detail&atid=105470&aid=811844&group_id=5470 (Someone should probably do a scan of the code in Modules to make sure all the threading API calls are wrapped in WITH_THREAD #ifdefs. Not me, now, though - way past bedtime). -- Anthony Baxter It's never too late to have a happy childhood.

...

...

Jeremy Hylton wrote We should treat 2.3.1 like 2.3.1rc1 and aim for Sep. 30 for a 2.3.2.

Wouldn't it be better to cut a 2.3.2rc before the final 2.3.2? once-bitten-ly, Anthony

[Jeremy Hylton]

...

We should treat 2.3.1 like 2.3.1rc1 and aim for Sep. 30 for a 2.3.2.

[Anthony Baxter]

Wouldn't it be better to cut a 2.3.2rc before the final 2.3.2?

once-bitten-ly,

Well, Jeremy's "Sep. 30" is only 4 days away. If 2.3.2 is restricted to fixing just the obvious big problems already reported against 2.3.1, in reality it's like Jeremy said: 2.3.1 was really 2.3.2c1 <0.5 wink>. Don't get too gun-shy. There were lots of changes in who did what for the 2.3.1 release, and some glitches were inevitable. It *had* gotten so smooth with the same people doing the same things all the time that, e.g., you must have noticed that Guido stopped doing anything for releases. I would have liked to pay a lot more attention but just couldn't make time for it; I'm sure the same is true of the other PythonLabs alumni (including Guido). So some things could have gone better. It's not that big a deal -- next time they will go better. Remember that we pioneered all possible ways to screw up a release already, and discovered that it's a finite set <wink>.

On Fri, Sep 26, 2003 at 02:24:27PM -0400, Tim Peters wrote:

Don't get too gun-shy. There were lots of changes in who did what for the 2.3.1 release, and some glitches were inevitable.

Some humble advice from someone who has done quite a few software releases in the last few years (mostly internal). The release process should be automated as much as possible. I think the Python project could do a bit better in this regard. Some ideas: * try to reduce the number of version numbers in different files need to be adjusted. Also, try to automate the adjustment. * write scripts to generate the final package and also verify its sanity (e.g. check version numbers, dates, detect files that should not be included in the release) * write scripts to upload the package for distribution. Ensure proper naming, permissions, etc. Python already has a release checklist. That's good. Be sure to follow it when doing a release and make sure it's up-to-date. Unfortunately Python doesn't do releases often enough for the process to be really polished. One effective way of improving the process is to have a new person do it with a veteran looking on. The new person should try to complete the release using only the checklist as the guide. The veteran is on hand to make sure nothing goes wrong. The veteran should also take notes; documenting any errors or omissions in the release checklist or ideas on ways to further automate the process. Neil

Neil Schemenauer writes:

* try to reduce the number of version numbers in different files need to be adjusted.

I suggest to update the version numbers in these files to the next version *after* a release is done. So, in the release23-maint branch the version numbers could *now* be adjusted to 2.3.2. And it would help if I could build the HTML docs myself from CVS. I did manage to create the pdf files with TeTex under windows, but I didn't succeed with the html pages so far. This way, a Windows release could be built at any time, and I would not only install and test it, but also use it as my default version. Thomas

Charles Cazabon writes:

Too late: 2.3.1 is in at least some versions of Debian already

The Debian situation is a bit better because they can incorporate patches and issue a point release of the package itself. I contacted the Debian/Python maintainer, and he plans to incorporate the os.fsync() fix as soon as the sourceforge CVS delay catches up to him. More troublesome are distros like Redhat whose packages are not dynamically updateable. They are sealed in stone once they become part of a release.

Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with only this change would suit me.

I'd be in favour of revoking 2.3.1 as well because of the severity of this bug. I don't think we'll be completely rid of it otherwise.

>> So unless we've got a bona fide HP/UX expert on tap now (do we? that >> would be way cool <wink>), don't hold up solving a clear and >> widespread Linux problem by waiting for one to appear. Jason> Just to be clear, the os.fsync() problem is not Linux-only. It's Jason> also on FreeBSD (probably every Unix platform actually). Correct. There are two new C macros in pyconfig.h, HAVE_FSYNC and HAVE_FDATASYNC. When originally added, the former was mistyped as HAVE_SYNC. Skip

Thomas Heller writes:

Neil Schemenauer writes:

...

* try to reduce the number of version numbers in different files need to be adjusted.

I suggest to update the version numbers in these files to the next version *after* a release is done. So, in the release23-maint branch the version numbers could *now* be adjusted to 2.3.2.

And it would help if I could build the HTML docs myself from CVS. I did manage to create the pdf files with TeTex under windows, but I didn't succeed with the html pages so far.

It occurs to me that I don't know *why* Fred is so much the documentation man; I've not had any trouble processing the docs into HTML lately (haven't tried on Windows, admittedly, and I haven't tried to make info ever). What else needs to be done? There must be quite a bit of mucking about on creosote to do, I guess. Finding a substitue Jack might be harder (though given Bob Ippolito's work rate at the moment we could probably make him MacPython maintainer without him noticing <wink>). Cheers, mwh -- In short, just business as usual in the wacky world of floating point <wink>. -- Tim Peters, comp.lang.python

...

...

"Jason R. Mastaler" wrote Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with only this change would suit me.

I'd be in favour of revoking 2.3.1 as well because of the severity of this bug. I don't think we'll be completely rid of it otherwise.

I';m not sure what the term "revoke" would mean in this context. Replacing it soon would work for me, and a big warning on the 2.3.1 page about the fsync problem and the HP/UX build problem (these are the two that _must_ be fixed in 2.3.2, as far as I'm concerned). I can whack a bit of text on the 2.3.1 page saying "there will be a 2.3.2 release shortly - stay tuned" if people think it's appropriate. Anthony -- Anthony Baxter It's never too late to have a happy childhood.

"Anthony Baxter" wrote in message news:200309270437.h8R4bxk0006927@localhost.localdomain...

I'm not sure what the term "revoke" would mean in this context.

Stop distribution by removing from download page . Given that it works fine for some platforms and most uses on the rest, a warning message seems ok to me. TJR

martin@v.loewis.de (Martin v. Löwis) writes:

I don't think the problem is severe; os.fsync is rarely used.

How did you arrive at this notion?

"Jason R. Mastaler" writes:

...

I don't think the problem is severe; os.fsync is rarely used.

How did you arrive at this notion?

From four considerations:

1. It did not show up in any of the tests. 2. To use it, you have to use "bare" file descriptors; this is relatively un-pythonic. 3. To use it, you have to worry about data getting on the disk - this really requires some kind of expert application. 4. It is not universally available, so the expert application I had to assume in 3) would most likely test for presence of fsync, and fall back to just not use it (and perhaps invoke sync(1) instead). Regards, Martin

Anthony Baxter writes:

Hm. The only usage of it I can find in the Zope2, Zope3 and Twisted codebases is a single use inside zodb, and it's wrapped in a check that os.fsync exists (since it's not available on all platforms). I also did a grep over all the python packages I've downloaded or installed (which is a _lot_) and the TMDA code is the only package to use it.

Also, getmail, as Charles Cazabon mentioned earlier in this thread. Also, Barry is considering using fsync() in the next release of Mailman, albeit as an option turned off by default. TMDA has thousands of users, Mailman has many more. So, this potentially affects many individuals regardless of whether a large number of packages use fsync() or not.

I'm still planning on cutting a 2.3.2 to fix the problem, but I see no reason to make any further steps to "revoke" 2.3.1 further than the warning message on the release page - which is mostly aimed at people who might be planning to package the code.

Yeah, I think what you are doing is fine. If 2.3.2 is available soon, I think people will just tend to pass over 2.3.1.

"Martin v. Löwis" writes:

It's not broken: it's absent.

I'm not sure I see the distinction. The system provides fsync(), and the Python documentation says os.fsync() is available, but it isn't. Python therefore doesn't work as advertised. This is broken to me.

Jeremy Hylton wrote:

I suppose ZODB is such an expert application. It has to cope with systems that do not provide fsync(), but it provides degraded service on such platforms. It is very important for the database to call fsync() when it commits a transaction.

I mostly agree: ZODB is indeed advanced, and it is indeed a good idea to check for presence of os.fsync before using it. While this is OT, I'd still like to question the usefulness of fsync(2) in the first place, for applications like ZODB. I assume fsync is used as a write barrier, to make sure old modifications are on disk, before making new changes. There are several cases in which this might be relevant: 1. The application will crash soon after performing fsync, leaving data potentially in an inconsistent state. Here, using fsync is not necessary, as the system will still perform all modifications on disk, even though the process has long terminated. 2. The application will be kill(2)ed soon after fsync completes (or even while fsync completes). Like 1), fsync is not needed. 3. There is an operating system crash (kernel panic or similar). fsync does not help, as, for a buggy kernel, anything might have happened to the data before. 4. There is a disk failure. fsync does not help, as the data on disk might not be recoverable. 5. There is a power outage. This is the case where fsync should help: everything up to the write barrier is on disk. Of course, if the disk drive itself has write caching, fsync may have completed without the data being on the disk (this would be an fsync bug, but I believe Linux suffers from this particular bug). So in short, fsync(2) helps only in case of a power outage; for normal operation, it is not needed. In the case of a power outage, it is doubtful whether it has the desired effect. Slightly more on-topic: os.fsync is even worse, as it cannot be used to implement a write barrier in case of multiple threads. It runs with the GIL released, so while fsync is running, other threads might change the file. The semantics of fsync in this case is pretty unclear, but it is likely not a write barrier. Regards, Martin

[cc to webmaster@starship.python.net] Michael Hudson writes:

Thomas Heller writes:

...

And it would help if I could build the HTML docs myself from CVS. I did manage to create the pdf files with TeTex under windows, but I didn't succeed with the html pages so far.

It occurs to me that I don't know *why* Fred is so much the documentation man; I've not had any trouble processing the docs into HTML lately (haven't tried on Windows, admittedly, and I haven't tried to make info ever).

I could do it myself if the tools needed to build the docs would be available on the starship. Last time I checked, neither Tex nor latex2html were installed. So, I hereby kindly request: Could the tools to build the Python docs from source please be installed on the starship? Thanks, Thomas

[Jeremy Hylton]

...

I suppose ZODB is such an expert application. It has to cope with systems that do not provide fsync(), but it provides degraded service on such platforms. It is very important for the database to call fsync() when it commits a transaction.

[Tim]

I'm not sure that ZODB *intends* to run on any system w/o fsync anymore. It had to before 2.3 because os.fsync() didn't exist on Windows, and ZODB has to run on Windows. I implemented os.fysnc() for Windows in 2.3 precisely so that ZODB would work better on Windows.

More mundanely, as I recall, we were having a specific problem wherein a backup script wasn't getting the correct size for a ZODB Data.fs file on Windows, and file.flush()'ing Data.fs didn't help enough -- backup kept screwing up until FileStorage's

if fsync is not None: fsync(file.fileno())

actually did something on Windows.

Bizarre. On Unix, fsync()'s only point is to do something that is only ever noticeable if there's an OS or hardware failure. But on Windows, it was needed to get the OS to synchronize the directory contents so that another process or thread would see the right thing. Anyway, I think that on Unix, that problem doesn't exist, and there the only problem of not having fsync() is the minuscule risk of hardware or OS failure right after you wrote the data but before it actually was written to the media. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sat, 2003-09-27 at 12:14, Jason R.Mastaler wrote:

Also, Barry is considering using fsync() in the next release of Mailman, albeit as an option turned off by default.

It'll be in Mailman 2.1.3, but turned off by default. I plan on adding something to the release notes about it, warning that it may not be available on every system (although we don't have to worry about Windows <wink>). Its use shouldn't be enabled if it's not available. -Barry

On Sat, 2003-09-27 at 13:48, "Martin v. Löwis" wrote:

5. There is a power outage. This is the case where fsync should help: everything up to the write barrier is on disk. Of course, if the disk drive itself has write caching, fsync may have completed without the data being on the disk (this would be an fsync bug, but I believe Linux suffers from this particular bug).

So in short, fsync(2) helps only in case of a power outage; for normal operation, it is not needed. In the case of a power outage, it is doubtful whether it has the desired effect.

This was the situation that folks reported causing message loss in Mailman 2.1.2. Those folks also reported that adding the fsync helped them (I guess a UPS would have helped too <wink>). I've added it to MM2.1.3, but I haven't enabled it by default. I'm still not sure the specific problem can be fixed in better ways (e.g. running on a more reliable host, running it on a sync'ing filesystem, or maybe opening the file with O_SYNC). Also, I did some benchmarking with fsync's added and on the system I tested I saw a 97% performance hit when always fsync'ing. That made me nervous. Before I enable it unconditionally, I'd want to be more certain that it's necessary, or useful for the majority of sites, and that the performance hit is either worth it, not as bad as my benchmarks showed, or at least not as observably bad under real-world conditions. -Barry

On 27 September 2003, Thomas Heller said:

I could do it myself if the tools needed to build the docs would be available on the starship. Last time I checked, neither Tex nor latex2html were installed.

So, I hereby kindly request: Could the tools to build the Python docs from source please be installed on the starship?

OK, I'm installing tetex-base and latex2html now. No guarantees if the version of latex2html in Debian testing will actually work, of course. ;-) (No, wait, it should work: I have the same version (2000-beta1-8) on my PC, and the last time I did Python doc work, I *think* I was able to produce HTML output.) Greg -- Greg Ward http://www.gerg.ca/ A closed mouth gathers no foot.

Barry Warsaw wrote:

Before I enable it unconditionally, I'd want to be more certain that it's necessary, or useful for the majority of sites, and that the performance hit is either worth it, not as bad as my benchmarks showed, or at least not as observably bad under real-world conditions.

You should first understand why message loss occurs if there is no fsync. For example, if the MTA stores incoming messages on disk, then invokes mailman, message loss may occur if the MTA deletes the message from the spool before mailman has fsync'ed its copy, and the power goes out at this moment. If so, users might be better of using a powerful filesystem. I *think* that, e.g. on Linux, ext3fs with ordered data writes might be sufficient, as deletion of the file should only occur after the data have been written (OTOH, it might be that ext3 in journalled mode is needed, if data and metadata changes are not mutually subject to the ordering). This should be less expensive than fsync, as the journalled file system will still perform lazy writes - just in the right order. Also notice that some file system drivers are incapable of implementing fsync correctly, so they fall back to treat fsync(2) like sync(2), which is quite expensive. Regards, Martin

[Martin v. Löwis]

As Guido explains, fsync is not necessary for that kind of application on a POSIX system. Once write(2) has completed, all other processes immediately see the changed data.

But we're not calling write(2) -- Python's file.write() (which my little driver used, and also what ZODB uses) calls the buffered fwrite. Ignoring that, some systems aren't "pure". I note that, on my Windows box, some Cygwin utilties "see" the hoped-for file sizes while running my driver without the fsync under Cygwin Python, but Windows utilities don't. I was using Cygwin 2.2.2; I don't know whether 2.3.1 under Cygwin thinks fsync has gone missing.

[Guido]

Tim, trust me. The scenario you go on to sketch is unique to Windows.

I would rather trust the test program, but I don't see much point to the argument (if that's what this is): if os.fsync has gone missing on platforms that "used to" support it, of course that's a serious regression.

fsync() was invented on Unix. On that platform, there's *never* a need to do fsync() to allow another process to see the right data. fsync()'s effect is just not visible at the abstraction level presented by the kernel. fsync() is needed because sometimes (e.g. through a power or kernel failure) the kernel's abstraction is broken. Everybody except you who is arguing for fsync()'s importance does so because they have experienced (or fear to experience) crashes.

It's a bug in the Windows implementation of Unix-style I/O that a commit() call (which is what fsync() is aliased to in posixmodule.c on Windows) is needed in order to ensure that other processes see what a process wrote.

The test driver (and ZODB) use buffered I/O, not unbuffered POSIX I/O. Assuring that writes done to buffered streams are visible to other processes is trickier than Martin's reliance on the write(2) man page, since write(2) isn't involved in the chain.

Given that fsync() on Windows doesn't have the configuration problem that started all this,

I don't know whether Cygwin Python suffers the same problem. If it does, then people running ZODB under Cygwin 2.3.1 (which I don't believe has been released yet) are in for nasty surprises if they run repozo.py under the "native" Windows Python. I've tried this under Cygwin 2.2.2 (see other email). Since that did have os.fsync(), but the native Windows Python didn't at that time, my best guess is that Cygwin Python is affected.

I'm not sure how *your* argument for fsync() can help convince Martin that we should do a 2.3.2 release ASAP.

It's enough for me that a function that used to be there has gone away without good reason. It's easier to cut a new release than argue about how much trouble the missing os.fsync should cause, as people have already poured more energy into complaining about it than would be needed to cut a new release.

[Tim]

...

The test driver (and ZODB) use buffered I/O, not unbuffered POSIX I/O. Assuring that writes done to buffered streams are visible to other processes is trickier than Martin's reliance on the write(2) man page, since write(2) isn't involved in the chain.

[Guido]

All you need to do is call f.flush(). This calls write(2) internally (even on Windows).

Sorry, I don't understand the point of this Unixish chest-puffing <0.8 wink>. POSIX doesn't mandate that fflush() call write(), although it requires, as an extension to the C standard, pretty much the same end result. Nevertheless, calling f.flush() isn't enough on Windows, regardless of whether the native Windows Python or Cygwin Python is doing it. Without the os.fsync(f.fileno()) too, native Windows utilities and the Windows Python see a wrong current file size. This gets pretty bizarre too: after letting Cygwin Python write and flush a million bytes (i.e., at the first prompt in the test driver), the Cygwin Python (in a different process) os.path.getsize() returns a million, but the Windows Python returns 0. Put in the fsync too and everyone is happy. This can't work if fsynch goes missing in the Cygwin Python.

Greg Ward writes:

On 27 September 2003, Thomas Heller said:

...

I could do it myself if the tools needed to build the docs would be available on the starship. Last time I checked, neither Tex nor latex2html were installed.

So, I hereby kindly request: Could the tools to build the Python docs from source please be installed on the starship?

OK, I'm installing tetex-base and latex2html now. No guarantees if the version of latex2html in Debian testing will actually work, of course. ;-)

Somehow this seems to have avoided installing some important executables -- like, e.g., "latex". Are there some other packages that need to be installed too? Cheers, mwh -- 58. Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

Tim> [Martin v. Löwis] >> As Guido explains, fsync is not necessary for that kind of >> application on a POSIX system. Once write(2) has completed, all other >> processes immediately see the changed data. Tim> But we're not calling write(2) -- Python's file.write() (which my Tim> little driver used, and also what ZODB uses) calls the buffered Tim> fwrite. Then shouldn't a file.flush() call be sufficient to force a call to write(2) on POSIX systems? Tim> Ignoring that, some systems aren't "pure". I note that, on my Tim> Windows box, some Cygwin utilties "see" the hoped-for file sizes Tim> while running my driver without the fsync under Cygwin Python, but Tim> Windows utilities don't. I was using Cygwin 2.2.2; I don't know Tim> whether 2.3.1 under Cygwin thinks fsync has gone missing. It's been established that os.fsync() is required for Windows in at least some circumstances, so you have to assume it's required under all circumstances, correct? Skip

>> ... It's been established that os.fsync() is required for Windows in >> at least some circumstances, so you have to assume it's required >> under all circumstances, correct? Tim> Sorry, I didn't follow the question. We've already established that os.fsync() is required in at least some cases on Windows. It seemed to me that you were trying to construct some other cases (possibly involving Cygwin) where that wasn't necessary. I probably just didn't follow your intent. Skip

Neil Schemenauer wrote:

* write scripts to generate the final package and also verify its sanity (e.g. check version numbers, dates, detect files that should not be included in the release)

FWIW, one thing I like to do is compare the list of files in the new release with the list of files in the previous release. It's amazing what can creep in. I've attached the script I use--it's ugly but it works. Shane #!/bin/sh # This utility compares the directory listing of two tar files. # It detects .gz or .bz2 extensions automatically and acts accordingly. if [ "$1" == "" ] || [ "$2" == "" ]; then echo usage: $0 tarfile1 tarfile2 exit 1 fi f1=`mktemp /tmp/difftar1.XXXXXX` f2=`mktemp /tmp/difftar2.XXXXXX` if echo $1 | grep '\.gz' > /dev/null; then flags='tfz' elif echo $1 | grep '\.bz2' > /dev/null; then flags='tfj' else flags='tf' fi tar $flags $1 | sort > $f1 if echo $2 | grep '\.gz' > /dev/null; then flags='tfz' elif echo $2 | grep '\.bz2' > /dev/null; then flags='tfj' else flags='tf' fi tar $flags $2 | sort > $f2 diff -u $f1 $f2 rm -f $f1 $f2

On 29 September 2003, Michael Hudson said:

Somehow this seems to have avoided installing some important executables -- like, e.g., "latex". Are there some other packages that need to be installed too?

Argh. Installing tetex-bin (and -doc, -extra, -lib just for fun) now. Greg -- Greg Ward http://www.gerg.ca/ A man without religion is like a fish without a bicycle.

Thomas Heller writes:

I suggest to update the version numbers in these files to the next version *after* a release is done. So, in the release23-maint branch the version numbers could *now* be adjusted to 2.3.2.

The Doc/ tree no longer depends on version numbers stored in the Doc/ tree; the makefile causes the script Doc/tools/getversioninfo to pull the version numbers from Include/patchlevel.h.

And it would help if I could build the HTML docs myself from CVS. I did manage to create the pdf files with TeTex under windows, but I didn't succeed with the html pages so far.

Are you using Cygwin? What problems did you encounter? I'll help if I can; I have a Windows machine (sometimes), but don't know anything about non-Cygwin Windows TeX systems (and I don't have Cygwin installed most of the time). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

Greg Ward writes:

Argh. Installing tetex-bin (and -doc, -extra, -lib just for fun) now.

Thanks, Greg. Works great now. Thomas

Michael Hudson writes:

One thing that puzzled me: Doc/Makefile seems to require that Doc/tools is on $PATH, unless I'm misunderstanding something.

It definately doesn't require that; I've never used Doc/tools/ on $PATH. One thing it was requiring (only recently) was that there was a mkhowto symlink somewhere on the $PATH that pointed to the mkhowto script. I've removed that constraint for the trunk. The intention is that we should be able to use a mkhowto script from a different checkout; you can still modify the MKHOWTO make variable to do that, but it's not so valuable on the trunk as on the maintenance branches (where we want to use mkhowto from the trunk). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

I wrote:

The intention is that we should be able to use a mkhowto script from a different checkout; you can still modify the MKHOWTO make variable to do that, but it's not so valuable on the trunk as on the maintenance branches (where we want to use mkhowto from the trunk).

Thomas Heller writes:

Do we? Why?

Definately. I don't want to maintain several versions of the tools; they're almost external application at this point. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

Thomas Heller writes:

Ok, but it would be nice if pep 101 and 102 would contain instructions how to build the docs (on unix), I will try to do the same for windows.

That's in the version of PEP 101 in CVS; the online version isn't up-to-date due to the anonymous CVS access on SF using their backup repositories that aren't update frequently enough. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

"Fred L. Drake, Jr." writes:

Michael Hudson writes:

...

One thing that puzzled me: Doc/Makefile seems to require that Doc/tools is on $PATH, unless I'm misunderstanding something.

It definately doesn't require that; I've never used Doc/tools/ on ^^^^^^^^^^ $PATH. One thing it was requiring (only recently) was that there was a mkhowto symlink somewhere on the $PATH that pointed to the mkhowto script.

Well, OK. I was getting "mkhowto: command not found" messages.

I've removed that constraint for the trunk.

Thanks! Cheers, mwh -- Very clever implementation techniques are required to implement this insanity correctly and usefully, not to mention that code written with this feature used and abused east and west is exceptionally exciting to debug. -- Erik Naggum on Algol-style "call-by-name"

Anthony Baxter writes:

The current documentation release tools don't build the latex packages. I tried using the Makefile targets, but they seemed to want to check out a HEAD revision of the dist/src/Doc directory. Oops.

Argh. Yes; this is a problem with mksourcepkg on branches. I think I can improve that a bit, but the best way seems to be running it with a second argument giving the specific tag we're interested in. The script also runs into the "we can't keep our anonymous CVS servers up to date" problem, so I'll mod my local copy not try to use the anonymous servers for now.

I've commented out the latex row of the download table for now - Fred, can you look into this and fix?

I'll have them up shortly. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation