
On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.1 (final). Python 2.3.1 is a pure bug fix release of Python 2.3, released in late July. A number of obscure crash-causing bugs have been fixed, various memory leaks have been squished, but no new features have been added to the language or to the library. For more information on Python 2.3.1, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.3.1 Highlights of this new release include: - Bugfixes - The Windows installer now ships with documentation in searchable htmlhelp format, rather than individual HTML files. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at http://www.python.org/2.3/highlights.html This release was the work of a number of dedicated python-dev folks who took the time to apply bugfixes back to the 2.3 code. The community owes them many thanks for this work. Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python 2.3.1 Release Manager (on behalf of the entire python-dev team)

Oh yeah - please hold off on the release23-maint branch for a bit, in case Jack wants to cut a 2.3.1 release for the Mac... Jack? Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

On woensdag, sep 24, 2003, at 10:15 Europe/Amsterdam, Anthony Baxter wrote:
Oh yeah - please hold off on the release23-maint branch for a bit, in case Jack wants to cut a 2.3.1 release for the Mac... Jack?
I don't know whether I'll find the time, give me until the end of the week to decide, please. And off on a tangent: I think we need a python-dev-announce mailing list or something of the sort. I've been busy with Real Work the last month, and I only skim python-dev when I have time, so today is the first time I hear about 2.3.1 (and, being on a train without internet-access right now I can't even test it:-). I'm not happy with this. For one thing, I haven't seen any indication that anyone tried a framework build on MacOSX, and how such a build interacts with the pre-installed 2.3 on Panther. Even though I wouldn't have had time to do any fixes for MacOSX that are on my plate I would at least have tested that 2.3.1 doesn't break anything... I wouldn't mind a checklist of things that need to be tested (plus preferred people responsible for them) before a release goes out. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

On Wed, 2003-09-24 at 08:43, Jack Jansen wrote:
IMO, this needs to be added to PEP 102 and there needs to be a list of people who will give a go, no-go on a new release. There needs to be timeouts though because we're all way too busy and it may just be that a release needs to go out before a person on the critical path has a chance to sign off on it. -Barry

I'm re-doing PEP 101 and 102 to change the various names to roles. I'll add a section to the top saying "You need to get a person signed up and agreed for each of the following roles. The people who have done each role in the past are..." Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Mea maxima culpa. I'll add a new section to the start of PEP101 and PEP102 about communicating with the appropriate people directly. We had a bunch of testers try out the release candidates on various Mac OS X releases - they reported no breakages. There's also been few/no checkins to the Mac-specific chunks of the tree for the release23-maint branch. I'm happy to store up checkins for the 23 branch for a couple of weeks to give you time to do a release if you so wish (consider it penance for my forgetting to email you). -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter <anthony@interlink.com.au> writes:
I think in future we should probably do the "2.3.2c1", wait a week, do "2.3.2" thing. It's more work, but there have been enough nits that I think it would have been worthwhile (readline vs. no threads, fsync and the docs issue spring to mind). Cheers, mwh -- 81. In computing, turning the obvious into the useful is a living definition of the word "frustration". -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

There's a workaround on the 2.3.1 bugs page. The only problems I'm aware of with 2.3.1 are the HP/UX build problems (workaround on bugs.html), the documentation mistakenly identifying itself as '2.4.0alpha', and the os.fsync problem. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter <anthony@interlink.com.au> writes:
There's a workaround, but the problem is that I think I'll be the one bearing the brunt of this bug for who knows how long. I've gotten about a half dozen problem reports about it already on the TMDA lists. TMDA uses os.fsync() in its mbox and Maildir delivery code, so practically every user will find their TMDA broken as soon as they install or upgrade to 2.3.1. Since Python point releases are usually so solid, they don't consider it might be a bug in the interpreter, and assume it's a TMDA bug. TMDA is just one example. Repeat for the other packages out there which rely on os.fsync(). The longer 2.3.1 is out there, the more it will be installed, incorporated into OS package distributions, etc, and the harder this breakage will be to shake.

Thomas Heller <theller@python.net> writes:
It certainly seems reasonable to me to release 2.3.2 fairly quickly.
But not earlier than at least one week after 2.3.2rc1 ;-)
Well, if you're going to wait that long, I think you might consider adding some sort of warning to the 2.3.1 download area. My fear is that some large OS vendor (e.g, Redhat) will pick up 2.3.1 for a future release.

Jason R. Mastaler <jason@mastaler.com> wrote:
Too late: 2.3.1 is in at least some versions of Debian already -- I'm getting reports on the os.fsync() issue already from my users (it breaks getmail). Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with only this change would suit me. Charles -- ----------------------------------------------------------------------- Charles Cazabon <python@discworld.dyndns.org> GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------

That doesn't fit in the numbering scheme, it would have to be 2.3.2. Given the amount of traffic about this I'm tempted to agree that we need to move quickly and issue a fix for this issue (only). --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]
[Anthony Baxter]
We really should fix the HP/UX build problem as well.
That sounds dangerous to me: there's *always* an HP/UX build problem, if you count (as I do) getting threads to work. Every release, some HP/UX user submits a patch that breaks the build for other HP/UX users. So unless we've got a bona fide HP/UX expert on tap now (do we? that would be way cool <wink>), don't hold up solving a clear and widespread Linux problem by waiting for one to appear.

On the scale of "HP/UX build problems we've known and loved", this one's a trivial one. https://sourceforge.net/tracker/?func=detail&atid=105470&aid=811160&group_id=5470 has a patch (from the gcc bugtracker - they ran into the exact same problem) Now, the readline+no-threads=boom coredump, well, that involves threads, and the original reporter even mentions HP/UX, but it's more a simple-ish one - missing #ifdef WITH_THREAD calls, as far as I can see. https://sourceforge.net/tracker/?func=detail&atid=105470&aid=811844&group_id=5470 (Someone should probably do a scan of the code in Modules to make sure all the threading API calls are wrapped in WITH_THREAD #ifdefs. Not me, now, though - way past bedtime). -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

>> So unless we've got a bona fide HP/UX expert on tap now (do we? that >> would be way cool <wink>), don't hold up solving a clear and >> widespread Linux problem by waiting for one to appear. Jason> Just to be clear, the os.fsync() problem is not Linux-only. It's Jason> also on FreeBSD (probably every Unix platform actually). Correct. There are two new C macros in pyconfig.h, HAVE_FSYNC and HAVE_FDATASYNC. When originally added, the former was mistyped as HAVE_SYNC. Skip

On Fri, 2003-09-26 at 13:52, Anthony Baxter wrote:
If we only fix the few critical problems -- CVS directories, fsync problem, Mac version numbers (?) -- then there doesn't seem to be much risk. But it wouldn't be too hard to convince me. (I gave my all clear on 2.3.1, after all.) Jeremy

[Jeremy Hylton]
We should treat 2.3.1 like 2.3.1rc1 and aim for Sep. 30 for a 2.3.2.
[Anthony Baxter]
Wouldn't it be better to cut a 2.3.2rc before the final 2.3.2?
once-bitten-ly,
Well, Jeremy's "Sep. 30" is only 4 days away. If 2.3.2 is restricted to fixing just the obvious big problems already reported against 2.3.1, in reality it's like Jeremy said: 2.3.1 was really 2.3.2c1 <0.5 wink>. Don't get too gun-shy. There were lots of changes in who did what for the 2.3.1 release, and some glitches were inevitable. It *had* gotten so smooth with the same people doing the same things all the time that, e.g., you must have noticed that Guido stopped doing anything for releases. I would have liked to pay a lot more attention but just couldn't make time for it; I'm sure the same is true of the other PythonLabs alumni (including Guido). So some things could have gone better. It's not that big a deal -- next time they will go better. Remember that we pioneered all possible ways to screw up a release already, and discovered that it's a finite set <wink>.

On Fri, Sep 26, 2003 at 02:24:27PM -0400, Tim Peters wrote:
Don't get too gun-shy. There were lots of changes in who did what for the 2.3.1 release, and some glitches were inevitable.
Some humble advice from someone who has done quite a few software releases in the last few years (mostly internal). The release process should be automated as much as possible. I think the Python project could do a bit better in this regard. Some ideas: * try to reduce the number of version numbers in different files need to be adjusted. Also, try to automate the adjustment. * write scripts to generate the final package and also verify its sanity (e.g. check version numbers, dates, detect files that should not be included in the release) * write scripts to upload the package for distribution. Ensure proper naming, permissions, etc. Python already has a release checklist. That's good. Be sure to follow it when doing a release and make sure it's up-to-date. Unfortunately Python doesn't do releases often enough for the process to be really polished. One effective way of improving the process is to have a new person do it with a veteran looking on. The new person should try to complete the release using only the checklist as the guide. The veteran is on hand to make sure nothing goes wrong. The veteran should also take notes; documenting any errors or omissions in the release checklist or ideas on ways to further automate the process. Neil

Neil Schemenauer <nas@arctrix.com> writes:
* try to reduce the number of version numbers in different files need to be adjusted.
I suggest to update the version numbers in these files to the next version *after* a release is done. So, in the release23-maint branch the version numbers could *now* be adjusted to 2.3.2. And it would help if I could build the HTML docs myself from CVS. I did manage to create the pdf files with TeTex under windows, but I didn't succeed with the html pages so far. This way, a Windows release could be built at any time, and I would not only install and test it, but also use it as my default version. Thomas

Thomas Heller <theller@python.net> writes:
It occurs to me that I don't know *why* Fred is so much the documentation man; I've not had any trouble processing the docs into HTML lately (haven't tried on Windows, admittedly, and I haven't tried to make info ever). What else needs to be done? There must be quite a bit of mucking about on creosote to do, I guess. Finding a substitue Jack might be harder (though given Bob Ippolito's work rate at the moment we could probably make him MacPython maintainer without him noticing <wink>). Cheers, mwh -- In short, just business as usual in the wacky world of floating point <wink>. -- Tim Peters, comp.lang.python

[cc to webmaster@starship.python.net] Michael Hudson <mwh@python.net> writes:
I could do it myself if the tools needed to build the docs would be available on the starship. Last time I checked, neither Tex nor latex2html were installed. So, I hereby kindly request: Could the tools to build the Python docs from source please be installed on the starship? Thanks, Thomas

On 27 September 2003, Thomas Heller said:
OK, I'm installing tetex-base and latex2html now. No guarantees if the version of latex2html in Debian testing will actually work, of course. ;-) (No, wait, it should work: I have the same version (2000-beta1-8) on my PC, and the last time I did Python doc work, I *think* I was able to produce HTML output.) Greg -- Greg Ward <gward@python.net> http://www.gerg.ca/ A closed mouth gathers no foot.

Greg Ward <gward@python.net> writes:
Somehow this seems to have avoided installing some important executables -- like, e.g., "latex". Are there some other packages that need to be installed too? Cheers, mwh -- 58. Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

Michael Hudson wrote:
Looks like you need tetex-bin, judging from my debian (unstable) system: bash-2.05b$ dpkg -S `which latex` tetex-bin: /usr/bin/latex It's probably worth grabbing tetex-extra, too; its description says: ... Together with tetex-bin and tetex-base, this will give you a complete teTeX installation. -Andrew.

On 29 September 2003, Michael Hudson said:
Argh. Installing tetex-bin (and -doc, -extra, -lib just for fun) now. Greg -- Greg Ward <gward@python.net> http://www.gerg.ca/ A man without religion is like a fish without a bicycle.

On 26-sep-03, at 23:34, Michael Hudson wrote:
Not that I'm planning to get run over by a tram shortly, but just in case: I think Just or Ronald or yourself or any various other python-devvers would make a better substitute *at this point in time*. Bob does some truly wonderful things at terrific speed, but that has some drawbacks for maintainability, IMHO. Being MacPython maintainer is 10% doing fun stuff and 90% taking care of all the nitty-gritty details, making everything work in as many situations as possible, documenting, etc etc etc. -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

Michael Hudson writes:
It's certainly gotten easier to deal with the documentation on modern Linux distributions. At CNRI, we used mostly Solaris boxes, and I have to build my own teTeX installations from source, and hand-select a version of LaTeX2HTML that worked for me. At this point, all the software that I can't just install from a RedHat CD is part of what gets pulled down from CVS. I've been able to build the docs on Cygwin as well, though I've not tried lately. A lot of what it takes to build the docs is written into Doc/Makefile, but it does require a solid make (it even uses $(shell ...) now, so maybe only GNU make will do; not sure).
What else needs to be done? There must be quite a bit of mucking about on creosote to do, I guess.
There's a bit, but that's getting easier and easier as I've gone through it a few times now. I updated PEP 101 the other evening so anyone can do what's needed to build the packages and get them in the download locations. There's more to be written to explain what else needs to be updated on the site. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

"Fred L. Drake, Jr." <fdrake@acm.org> writes:
Oh, there was a reason I put a "lately" in what I said...
One thing that puzzled me: Doc/Makefile seems to require that Doc/tools is on $PATH, unless I'm misunderstanding something.
Well, progress! I think it's a worthy goal that no single person is required to make a release, and that actually this isn't too far off. Cheers, mwh -- Never meddle in the affairs of NT. It is slow to boot and quick to crash. -- Stephen Harris -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html

Michael Hudson writes:
One thing that puzzled me: Doc/Makefile seems to require that Doc/tools is on $PATH, unless I'm misunderstanding something.
It definately doesn't require that; I've never used Doc/tools/ on $PATH. One thing it was requiring (only recently) was that there was a mkhowto symlink somewhere on the $PATH that pointed to the mkhowto script. I've removed that constraint for the trunk. The intention is that we should be able to use a mkhowto script from a different checkout; you can still modify the MKHOWTO make variable to do that, but it's not so valuable on the trunk as on the maintenance branches (where we want to use mkhowto from the trunk). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

Thomas Heller writes:
Ok, but it would be nice if pep 101 and 102 would contain instructions how to build the docs (on unix), I will try to do the same for windows.
That's in the version of PEP 101 in CVS; the online version isn't up-to-date due to the anonymous CVS access on SF using their backup repositories that aren't update frequently enough. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

"Fred L. Drake, Jr." <fdrake@acm.org> writes:
Well, OK. I was getting "mkhowto: command not found" messages.
I've removed that constraint for the trunk.
Thanks! Cheers, mwh -- Very clever implementation techniques are required to implement this insanity correctly and usefully, not to mention that code written with this feature used and abused east and west is exceptionally exciting to debug. -- Erik Naggum on Algol-style "call-by-name"

Thomas Heller writes:
The Doc/ tree no longer depends on version numbers stored in the Doc/ tree; the makefile causes the script Doc/tools/getversioninfo to pull the version numbers from Include/patchlevel.h.
Are you using Cygwin? What problems did you encounter? I'll help if I can; I have a Windows machine (sometimes), but don't know anything about non-Cygwin Windows TeX systems (and I don't have Cygwin installed most of the time). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

"Fred L. Drake, Jr." <fdrake@acm.org> writes:
No, I'm not using cygwin. I have seen too many broken cvs files with line end problems, I suspect people check in files with MSDOS line ending under cygwin. Now that I can build the docs on starship (thanks, Greg!) it's not needed anymore to do it under Windows, but for the archives here are my experiences: Doing 'nmake pdf' (this is the MSVC6 make utility) in the src/Doc directory worked, it created the pdf docs with MikTeX I had installed. Maybe I had to trivially edit the Makefile (replace 'cp' with 'copy' and such) before. It doesn't work anymore with the recent checkins to the Makefile it didn't work anymore, although installing the Mingw32 gnumake helped. Then I tried to bring 'make html' to work, installed latex2html (I have Perl already), but this always complained about pnmtopng missing (or something like that). And the make failed with an error such as 'image format unsupported'. Well, I tried to find and install native windows pnm2png and png2pnm tools, had to replace incompatible zlib.dll and so on. It didn't work, instead it broke my ssh and maybe other stuff. At this point I gave up, removed the software, and be happy that I managed to get my ssh working again. Thomas

Thomas Heller writes:
I appreciate your taking the time!
Most of the "cp" commands were removed as mkhowto became more capable, are were replaced by calls to shutil.copyfile(). There are still a few "cp" commands in the Makefile, though, and the "clean" target and friends still us "rm".
It doesn't work anymore with the recent checkins to the Makefile it didn't work anymore, although installing the Mingw32 gnumake helped.
That's expected, since it now uses the GNU-ish $(shell ...) syntax to call an external script from Doc/tools/. Removing this would require even more painful gyrations to maintain the same functionality, or would require that Python version numbers once more appear in the documentation source tree.
That's painful. netpbm is a documented requirement for LaTeX2HTML, but is a pain. I had to install that from source under Cygwin.
At this point I gave up, removed the software, and be happy that I managed to get my ssh working again.
That certainly sounds like a pain. I'll think about what I can do to make it easier, but I don't think it can take a high priority. I'm glad you got it working on Starship. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

The current documentation release tools don't build the latex packages. I tried using the Makefile targets, but they seemed to want to check out a HEAD revision of the dist/src/Doc directory. Oops. I've commented out the latex row of the download table for now - Fred, can you look into this and fix? Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter writes:
Argh. Yes; this is a problem with mksourcepkg on branches. I think I can improve that a bit, but the best way seems to be running it with a second argument giving the specific tag we're interested in. The script also runs into the "we can't keep our anonymous CVS servers up to date" problem, so I'll mod my local copy not try to use the anonymous servers for now.
I've commented out the latex row of the download table for now - Fred, can you look into this and fix?
I'll have them up shortly. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

Neil Schemenauer wrote:
* try to reduce the number of version numbers in different files need to be adjusted. Also, try to automate the adjustment.
Check out the package vertooo (on SourceForge) for this. It does just this (automating version numbers in lots of files). It's even written in Python. :-)

Neil Schemenauer wrote:
FWIW, one thing I like to do is compare the list of files in the new release with the list of files in the previous release. It's amazing what can creep in. I've attached the script I use--it's ugly but it works. Shane #!/bin/sh # This utility compares the directory listing of two tar files. # It detects .gz or .bz2 extensions automatically and acts accordingly. if [ "$1" == "" ] || [ "$2" == "" ]; then echo usage: $0 tarfile1 tarfile2 exit 1 fi f1=`mktemp /tmp/difftar1.XXXXXX` f2=`mktemp /tmp/difftar2.XXXXXX` if echo $1 | grep '\.gz' > /dev/null; then flags='tfz' elif echo $1 | grep '\.bz2' > /dev/null; then flags='tfj' else flags='tf' fi tar $flags $1 | sort > $f1 if echo $2 | grep '\.gz' > /dev/null; then flags='tfz' elif echo $2 | grep '\.bz2' > /dev/null; then flags='tfj' else flags='tf' fi tar $flags $2 | sort > $f2 diff -u $f1 $f2 rm -f $f1 $f2

Should there be an announcement that 2.3.2 is forthcoming and hold off on further downloads or distribution of 2.3.1? Raymond -----Original Message----- From: python-dev-bounces@python.org [mailto:python-dev-bounces@python.org] On Behalf Of Guido van Rossum Sent: Friday, September 26, 2003 12:40 PM To: Charles Cazabon Cc: python-dev@python.org Subject: Re: [Python-Dev] Re: RELEASED Python 2.3.1
That doesn't fit in the numbering scheme, it would have to be 2.3.2. Given the amount of traffic about this I'm tempted to agree that we need to move quickly and issue a fix for this issue (only). --Guido van Rossum (home page: http://www.python.org/~guido/) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev

Charles Cazabon <python@discworld.dyndns.org> writes:
Too late: 2.3.1 is in at least some versions of Debian already
The Debian situation is a bit better because they can incorporate patches and issue a point release of the package itself. I contacted the Debian/Python maintainer, and he plans to incorporate the os.fsync() fix as soon as the sourceforge CVS delay catches up to him. More troublesome are distros like Redhat whose packages are not dynamically updateable. They are sealed in stone once they become part of a release.
Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with only this change would suit me.
I'd be in favour of revoking 2.3.1 as well because of the severity of this bug. I don't think we'll be completely rid of it otherwise.

I';m not sure what the term "revoke" would mean in this context. Replacing it soon would work for me, and a big warning on the 2.3.1 page about the fsync problem and the HP/UX build problem (these are the two that _must_ be fixed in 2.3.2, as far as I'm concerned). I can whack a bit of text on the 2.3.1 page saying "there will be a 2.3.2 release shortly - stay tuned" if people think it's appropriate. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

"Anthony Baxter" <anthony@interlink.com.au> wrote in message news:200309270437.h8R4bxk0006927@localhost.localdomain...
I'm not sure what the term "revoke" would mean in this context.
Stop distribution by removing from download page . Given that it works fine for some platforms and most uses on the rest, a warning message seems ok to me. TJR

Hm. The only usage of it I can find in the Zope2, Zope3 and Twisted codebases is a single use inside zodb, and it's wrapped in a check that os.fsync exists (since it's not available on all platforms). I also did a grep over all the python packages I've downloaded or installed (which is a _lot_) and the TMDA code is the only package to use it. I'm still planning on cutting a 2.3.2 to fix the problem, but I see no reason to make any further steps to "revoke" 2.3.1 further than the warning message on the release page - which is mostly aimed at people who might be planning to package the code. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter <anthony@interlink.com.au> writes:
Also, getmail, as Charles Cazabon mentioned earlier in this thread. Also, Barry is considering using fsync() in the next release of Mailman, albeit as an option turned off by default. TMDA has thousands of users, Mailman has many more. So, this potentially affects many individuals regardless of whether a large number of packages use fsync() or not.
Yeah, I think what you are doing is fine. If 2.3.2 is available soon, I think people will just tend to pass over 2.3.1.

Jason> Also, Barry is considering using fsync() in the next release of Jason> Mailman, albeit as an option turned off by default. Something tells me Barry will probably guard its use with a hasattr(). ;-) Skip

On Sat, 2003-09-27 at 12:14, Jason R.Mastaler wrote:
Also, Barry is considering using fsync() in the next release of Mailman, albeit as an option turned off by default.
It'll be in Mailman 2.1.3, but turned off by default. I plan on adding something to the release notes about it, warning that it may not be available on every system (although we don't have to worry about Windows <wink>). Its use shouldn't be enabled if it's not available. -Barry

"Jason R. Mastaler" <jason@mastaler.com> writes:
I don't think the problem is severe; os.fsync is rarely used.
How did you arrive at this notion?
From four considerations:
1. It did not show up in any of the tests. 2. To use it, you have to use "bare" file descriptors; this is relatively un-pythonic. 3. To use it, you have to worry about data getting on the disk - this really requires some kind of expert application. 4. It is not universally available, so the expert application I had to assume in 3) would most likely test for presence of fsync, and fall back to just not use it (and perhaps invoke sync(1) instead). Regards, Martin

Martin v. Löwis <martin@v.loewis.de> wrote: > "Jason R. Mastaler" <jason@mastaler.com> writes: > > > > I don't think the problem is severe; os.fsync is rarely used. > > > > How did you arrive at this notion? > > >From four considerations: [...] > 3. To use it, you have to worry about data getting on the disk - > this really requires some kind of expert application. Yes, like delivering mail, or writing to a database, or doing any of a thousand other tasks that use the filesystem to store and synchronize data. os.fsync() breakage is a /big/ problem, particularly because the resulting traceback makes users think the application is broken, not the interpreter. Charles -- ----------------------------------------------------------------------- Charles Cazabon <python@discworld.dyndns.org> GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------

Charles Cazabon wrote:
os.fsync() breakage is a /big/ problem, particularly because the resulting traceback makes users think the application is broken, not the interpreter.
It's not broken: it's absent. In a sense, the application /is/ broken, as it should have dealt with os.fsync being absent in the first place. Applications should be prepared that POSIX functions are not available in Python even if the C library provides them on the system, as the system headers may fail to declare a prototype for the function (a problem which triggered that erroneous change in 2.3.1 in the first place). Python is taking the address of fsync, so merely having the function in the C library is not enough, nor is it to define fsync as a function-style macro. In either case, os.fsync will be rightfully absent. FWIW, it was a Redhat system that failed to provide a fsync prototype. Regards, Martin

"Martin v. Löwis" <martin@v.loewis.de> writes:
It's not broken: it's absent.
I'm not sure I see the distinction. The system provides fsync(), and the Python documentation says os.fsync() is available, but it isn't. Python therefore doesn't work as advertised. This is broken to me.

On Sat, 2003-09-27 at 06:30, Martin v. Löwis wrote:
I suppose ZODB is such an expert application. It has to cope with systems that do not provide fsync(), but it provides degraded service on such platforms. It is very important for the database to call fsync() when it commits a transaction. Jeremy

Jeremy Hylton wrote:
I mostly agree: ZODB is indeed advanced, and it is indeed a good idea to check for presence of os.fsync before using it. While this is OT, I'd still like to question the usefulness of fsync(2) in the first place, for applications like ZODB. I assume fsync is used as a write barrier, to make sure old modifications are on disk, before making new changes. There are several cases in which this might be relevant: 1. The application will crash soon after performing fsync, leaving data potentially in an inconsistent state. Here, using fsync is not necessary, as the system will still perform all modifications on disk, even though the process has long terminated. 2. The application will be kill(2)ed soon after fsync completes (or even while fsync completes). Like 1), fsync is not needed. 3. There is an operating system crash (kernel panic or similar). fsync does not help, as, for a buggy kernel, anything might have happened to the data before. 4. There is a disk failure. fsync does not help, as the data on disk might not be recoverable. 5. There is a power outage. This is the case where fsync should help: everything up to the write barrier is on disk. Of course, if the disk drive itself has write caching, fsync may have completed without the data being on the disk (this would be an fsync bug, but I believe Linux suffers from this particular bug). So in short, fsync(2) helps only in case of a power outage; for normal operation, it is not needed. In the case of a power outage, it is doubtful whether it has the desired effect. Slightly more on-topic: os.fsync is even worse, as it cannot be used to implement a write barrier in case of multiple threads. It runs with the GIL released, so while fsync is running, other threads might change the file. The semantics of fsync in this case is pretty unclear, but it is likely not a write barrier. Regards, Martin

[Martin v. Löwis]
That's important, but not primarily for the catastrophic error-recovery scenarios you go on to sketch. The most important error-recovery procedure is preventative, running backups against a ZODB database while the database is active (Tools/repozo.py in a recent ZODB distribution is the right tool for this). Since a ZODB process may run for months, it's not practical to say that backups require shutting ZODB down. Without both flushing and fsync'ing, the backup process can't get at all the data that's "really" in the file. Here's a little Python driver: import os fp = file('test.dat', 'wb') guts = 'x' * 1000000 n = 0 while 1: fp.write(guts) fp.flush() os.fsync(fp.fileno()) n += len(guts) proceed = raw_input("wrote %d so far; hit key" % n) At least on Windows, both the flush and the fsync are necessary to see one million bytes (via a different process) at the first prompt, two million at the second, and so on. With neither, another process typically sees 0 bytes before the file gets huge. With just one of them, it seems hard to predict, ranging from 0 to "almost" a million additional bytes per prompt. ZODB does a flush and an fsync after each transaction, so that the backup script (or any other distinct process) sees the most recent data available. Besides missing large gobs of newer data, without the fsync the backup script may see incomplete data for the most recent transaction that managed to wind up on disk, and erroneously conclude that the database is corrupted. In short, fsync is necessary to support ZODB best practice.

[Tim]
That's important, but not primarily for the catastrophic error-recovery scenarios you go on to sketch.
Tim, trust me. The scenario you go on to sketch is unique to Windows. fsync() was invented on Unix. On that platform, there's *never* a need to do fsync() to allow another process to see the right data. fsync()'s effect is just not visible at the abstraction level presented by the kernel. fsync() is needed because sometimes (e.g. through a power or kernel failure) the kernel's abstraction is broken. Everybody except you who is arguing for fsync()'s importance does so because they have experienced (or fear to experience) crashes. It's a bug in the Windows implementation of Unix-style I/O that a commit() call (which is what fsync() is aliased to in posixmodule.c on Windows) is needed in order to ensure that other processes see what a process wrote. Given that fsync() on Windows doesn't have the configuration problem that started all this, I'm not sure how *your* argument for fsync() can help convince Martin that we should do a 2.3.2 release ASAP. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]
Tim, trust me. The scenario you go on to sketch is unique to Windows.
I would rather trust the test program, but I don't see much point to the argument (if that's what this is): if os.fsync has gone missing on platforms that "used to" support it, of course that's a serious regression.
The test driver (and ZODB) use buffered I/O, not unbuffered POSIX I/O. Assuring that writes done to buffered streams are visible to other processes is trickier than Martin's reliance on the write(2) man page, since write(2) isn't involved in the chain.
Given that fsync() on Windows doesn't have the configuration problem that started all this,
I don't know whether Cygwin Python suffers the same problem. If it does, then people running ZODB under Cygwin 2.3.1 (which I don't believe has been released yet) are in for nasty surprises if they run repozo.py under the "native" Windows Python. I've tried this under Cygwin 2.2.2 (see other email). Since that did have os.fsync(), but the native Windows Python didn't at that time, my best guess is that Cygwin Python is affected.
I'm not sure how *your* argument for fsync() can help convince Martin that we should do a 2.3.2 release ASAP.
It's enough for me that a function that used to be there has gone away without good reason. It's easier to cut a new release than argue about how much trouble the missing os.fsync should cause, as people have already poured more energy into complaining about it than would be needed to cut a new release.

All you need to do is call f.flush(). This calls write(2) internally (even on Windows). --Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim]
[Guido]
All you need to do is call f.flush(). This calls write(2) internally (even on Windows).
Sorry, I don't understand the point of this Unixish chest-puffing <0.8 wink>. POSIX doesn't mandate that fflush() call write(), although it requires, as an extension to the C standard, pretty much the same end result. Nevertheless, calling f.flush() isn't enough on Windows, regardless of whether the native Windows Python or Cygwin Python is doing it. Without the os.fsync(f.fileno()) too, native Windows utilities and the Windows Python see a wrong current file size. This gets pretty bizarre too: after letting Cygwin Python write and flush a million bytes (i.e., at the first prompt in the test driver), the Cygwin Python (in a different process) os.path.getsize() returns a million, but the Windows Python returns 0. Put in the fsync too and everyone is happy. This can't work if fsynch goes missing in the Cygwin Python.

[Martin v. Löwis]
But we're not calling write(2) -- Python's file.write() (which my little driver used, and also what ZODB uses) calls the buffered fwrite. Ignoring that, some systems aren't "pure". I note that, on my Windows box, some Cygwin utilties "see" the hoped-for file sizes while running my driver without the fsync under Cygwin Python, but Windows utilities don't. I was using Cygwin 2.2.2; I don't know whether 2.3.1 under Cygwin thinks fsync has gone missing.

Tim> [Martin v. Löwis] >> As Guido explains, fsync is not necessary for that kind of >> application on a POSIX system. Once write(2) has completed, all other >> processes immediately see the changed data. Tim> But we're not calling write(2) -- Python's file.write() (which my Tim> little driver used, and also what ZODB uses) calls the buffered Tim> fwrite. Then shouldn't a file.flush() call be sufficient to force a call to write(2) on POSIX systems? Tim> Ignoring that, some systems aren't "pure". I note that, on my Tim> Windows box, some Cygwin utilties "see" the hoped-for file sizes Tim> while running my driver without the fsync under Cygwin Python, but Tim> Windows utilities don't. I was using Cygwin 2.2.2; I don't know Tim> whether 2.3.1 under Cygwin thinks fsync has gone missing. It's been established that os.fsync() is required for Windows in at least some circumstances, so you have to assume it's required under all circumstances, correct? Skip

[Skip Montanaro]
Then shouldn't a file.flush() call be sufficient to force a call to write(2) on POSIX systems?
The POSIX spec doesn't say that, but it does say enough that it shouldn't matter whether or not an fflush implementation happens to call write under the covers -- POSIX fflush is required to write data "to the file" regardless of how it's implemented. A difficulty with Windows is that a file ain't an inode -- a file doesn't exist except in a directory, and the directory structure records the current length of the files in the directory. Writing data to a file doesn't necessarily update the file length stored in the file's directory entry, and flushing doesn't necessary update it either; _commit() does. Native Windows utilities get their idea of a file's length from reading the file's directory entry.
Sorry, I didn't follow the question.

>> ... It's been established that os.fsync() is required for Windows in >> at least some circumstances, so you have to assume it's required >> under all circumstances, correct? Tim> Sorry, I didn't follow the question. We've already established that os.fsync() is required in at least some cases on Windows. It seemed to me that you were trying to construct some other cases (possibly involving Cygwin) where that wasn't necessary. I probably just didn't follow your intent. Skip

[Skip Montanaro]
Cygwin doesn't appear to need fsync() so long as you stick purely to Cygwin tools in all processes -- Cygwin appears POSIX-like in that respect. But Cygwin runs on Windows, and fsync() is still needed in Cygwin Python programs if native Windows utilities (including the native Windows Python) are to see correct current file sizes too. If os.fsync() is missing in Cygwin 2.3.1 (I don't know whether it is, but suspect that it is), then problems follow from that. A specific example is ZODB running under Cygwin, while the ZODB live-backup script repozo.py runs under native Windows Python. If the former can't do an fsync(), the latter can't know the correct current file size.

On Sat, 2003-09-27 at 13:48, "Martin v. Löwis" wrote:
This was the situation that folks reported causing message loss in Mailman 2.1.2. Those folks also reported that adding the fsync helped them (I guess a UPS would have helped too <wink>). I've added it to MM2.1.3, but I haven't enabled it by default. I'm still not sure the specific problem can be fixed in better ways (e.g. running on a more reliable host, running it on a sync'ing filesystem, or maybe opening the file with O_SYNC). Also, I did some benchmarking with fsync's added and on the system I tested I saw a 97% performance hit when always fsync'ing. That made me nervous. Before I enable it unconditionally, I'd want to be more certain that it's necessary, or useful for the majority of sites, and that the performance hit is either worth it, not as bad as my benchmarks showed, or at least not as observably bad under real-world conditions. -Barry

Barry Warsaw wrote:
You should first understand why message loss occurs if there is no fsync. For example, if the MTA stores incoming messages on disk, then invokes mailman, message loss may occur if the MTA deletes the message from the spool before mailman has fsync'ed its copy, and the power goes out at this moment. If so, users might be better of using a powerful filesystem. I *think* that, e.g. on Linux, ext3fs with ordered data writes might be sufficient, as deletion of the file should only occur after the data have been written (OTOH, it might be that ext3 in journalled mode is needed, if data and metadata changes are not mutually subject to the ordering). This should be less expensive than fsync, as the journalled file system will still perform lazy writes - just in the right order. Also notice that some file system drivers are incapable of implementing fsync correctly, so they fall back to treat fsync(2) like sync(2), which is quite expensive. Regards, Martin

On Sat, 2003-09-27 at 13:48, "Martin v. Löwis" wrote:
This is a vacuously true statement :-). Anything might have happened; among other things, the kernel buffers could have been flushed to disk before the kernel panic. I imagine that there are many bugs that could cause a kernel panic or deadlock that would not also corrupt data on its way to disk. That is, we can't avoid bugs that are going to cause problems writing to disk, but we can do our best to minimize the effects of other bugs.
(There are OSes other than Linux.)
I see two cases where it improves the changes that data is on persistent storage before the database reports to the client that the write succeeded.
Sure. ZODB only allows a single thread to write at a time. Jeremy

[Jeremy Hylton]
I'm not sure that ZODB *intends* to run on any system w/o fsync anymore. It had to before 2.3 because os.fsync() didn't exist on Windows, and ZODB has to run on Windows. I implemented os.fysnc() for Windows in 2.3 precisely so that ZODB would work better on Windows. More mundanely, as I recall, we were having a specific problem wherein a backup script wasn't getting the correct size for a ZODB Data.fs file on Windows, and file.flush()'ing Data.fs didn't help enough -- backup kept screwing up until FileStorage's if fsync is not None: fsync(file.fileno()) actually did something on Windows.

[Tim]
Bizarre. On Unix, fsync()'s only point is to do something that is only ever noticeable if there's an OS or hardware failure. But on Windows, it was needed to get the OS to synchronize the directory contents so that another process or thread would see the right thing. Anyway, I think that on Unix, that problem doesn't exist, and there the only problem of not having fsync() is the minuscule risk of hardware or OS failure right after you wrote the data but before it actually was written to the media. --Guido van Rossum (home page: http://www.python.org/~guido/)

Hm. I ran the Zope2, Zope3, and Twisted test suites against the release build before the release. I'm open to additional suggestions (I just grabbed tmda, but it doesn't have a test suite). In this case, the problem hopefully would've been picked up with a release candidate, but I'm also happy to add additional slabs of test suites to run against code before it's released... Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

On Thu, 25 Sep 2003 12:37:08 -0600, Jason R. Mastaler <jason@mastaler.com> wrote:
Will there be a 2.3.2 release, or some other sort of bugfix release to address the problems in 2.3.1?
Another minor nit: the 2.3.1 .tgz file contains the CVS/ subdirectories. This doesn't actually hurt anything, though, just cluttering up the tree a bit. --amk

Ouch. Isn't this in the PEP? The CVS directories *do* hurt, IMO (by confusing people). --Guido van Rossum (home page: http://www.python.org/~guido/)

I'm very very confused, since I _did_ use cvs export. Really. Repeatedly. Ugh. Given this, I'm happy to look at a 2.3.2 in a couple of weeks - I'd like very much to finish the PEP rewrite and get it reviewed first. Arrrrrgh. (no, not talk-like-a-pirate day, frustration) Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

On 25-sep-03, at 5:20, Anthony Baxter wrote:
Don't worry: I assume there's no harm done, and otherwise I'll just lobby for a quick 2.3.2.
There's a couple of version numbers that would have been nice to update, in the various .plist files. In other words, if you build 2.3.1 from source with a framework build then "Show Info" on the applications will show them to be 2.3, not 2.3.1. Same for the Apple Help Book that is built in to the IDE.
No, don't worry. I'm going to skip 2.3.1 for MacPython-OS9. I'll catch up again with 2.3.2. -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

Jack Jansen <Jack.Jansen@cwi.nl> writes:
As a procedural note, I would suggest that suggest changes are done immediately *after* the previous release, instead of shortly before the next one. The release branch should be releasable at any time, without having to ask anybody for permission. What gets done is done, what doesn't get done has to wait for the next release. The less coordination, the better. Regards, Martin

martin@v.loewis.de (Martin v. Löwis) writes:
+1 I guess this should be added to PEP 102, too. Cheers, mwh -- BUGS Never use this function. This function modifies its first argument. The identity of the delimiting character is lost. This function cannot be used on constant strings. -- the glibc manpage for strtok(3)

martin@v.loewis.de (Martin v. Löwis) writes:
Err, no. I assume you've read the rest of your mail by now... Cheers, mwh -- The Internet is full. Go away. -- http://www.disobey.com/devilshat/ds011101.htm

Martin> "Jason R. Mastaler" <jason@mastaler.com> writes: >> Python 2.3.1 (#1, Sep 24 2003, 15:55:24) >> [GCC 3.2 20020903 (Red Hat Linux 8.0 3.2-7)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> from os import fsync >> Traceback (most recent call last): >> File "<stdin>", line 1, in ? >> ImportError: cannot import name fsync >> >>> Martin> Why is that a bug? Your system just does not have a declared Martin> fsync function. No, it was a typo in the configure scripts. I checked in a fix for both the head and release23-maint branches yesterday. Red Hat 8.0 does have fsync: FSYNC(2) Linux Programmer's Manual FSYNC(2) NAME fsync, fdatasync - synchronize a file's complete in-core state with that on disk ... That is from an 8.0 system. Skip

Oh yeah - please hold off on the release23-maint branch for a bit, in case Jack wants to cut a 2.3.1 release for the Mac... Jack? Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

On woensdag, sep 24, 2003, at 10:15 Europe/Amsterdam, Anthony Baxter wrote:
Oh yeah - please hold off on the release23-maint branch for a bit, in case Jack wants to cut a 2.3.1 release for the Mac... Jack?
I don't know whether I'll find the time, give me until the end of the week to decide, please. And off on a tangent: I think we need a python-dev-announce mailing list or something of the sort. I've been busy with Real Work the last month, and I only skim python-dev when I have time, so today is the first time I hear about 2.3.1 (and, being on a train without internet-access right now I can't even test it:-). I'm not happy with this. For one thing, I haven't seen any indication that anyone tried a framework build on MacOSX, and how such a build interacts with the pre-installed 2.3 on Panther. Even though I wouldn't have had time to do any fixes for MacOSX that are on my plate I would at least have tested that 2.3.1 doesn't break anything... I wouldn't mind a checklist of things that need to be tested (plus preferred people responsible for them) before a release goes out. -- - Jack Jansen <Jack.Jansen@oratrix.com> http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman -

On Wed, 2003-09-24 at 08:43, Jack Jansen wrote:
IMO, this needs to be added to PEP 102 and there needs to be a list of people who will give a go, no-go on a new release. There needs to be timeouts though because we're all way too busy and it may just be that a release needs to go out before a person on the critical path has a chance to sign off on it. -Barry

I'm re-doing PEP 101 and 102 to change the various names to roles. I'll add a section to the top saying "You need to get a person signed up and agreed for each of the following roles. The people who have done each role in the past are..." Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Mea maxima culpa. I'll add a new section to the start of PEP101 and PEP102 about communicating with the appropriate people directly. We had a bunch of testers try out the release candidates on various Mac OS X releases - they reported no breakages. There's also been few/no checkins to the Mac-specific chunks of the tree for the release23-maint branch. I'm happy to store up checkins for the 23 branch for a couple of weeks to give you time to do a release if you so wish (consider it penance for my forgetting to email you). -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter <anthony@interlink.com.au> writes:
I think in future we should probably do the "2.3.2c1", wait a week, do "2.3.2" thing. It's more work, but there have been enough nits that I think it would have been worthwhile (readline vs. no threads, fsync and the docs issue spring to mind). Cheers, mwh -- 81. In computing, turning the obvious into the useful is a living definition of the word "frustration". -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

There's a workaround on the 2.3.1 bugs page. The only problems I'm aware of with 2.3.1 are the HP/UX build problems (workaround on bugs.html), the documentation mistakenly identifying itself as '2.4.0alpha', and the os.fsync problem. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter <anthony@interlink.com.au> writes:
There's a workaround, but the problem is that I think I'll be the one bearing the brunt of this bug for who knows how long. I've gotten about a half dozen problem reports about it already on the TMDA lists. TMDA uses os.fsync() in its mbox and Maildir delivery code, so practically every user will find their TMDA broken as soon as they install or upgrade to 2.3.1. Since Python point releases are usually so solid, they don't consider it might be a bug in the interpreter, and assume it's a TMDA bug. TMDA is just one example. Repeat for the other packages out there which rely on os.fsync(). The longer 2.3.1 is out there, the more it will be installed, incorporated into OS package distributions, etc, and the harder this breakage will be to shake.

Thomas Heller <theller@python.net> writes:
It certainly seems reasonable to me to release 2.3.2 fairly quickly.
But not earlier than at least one week after 2.3.2rc1 ;-)
Well, if you're going to wait that long, I think you might consider adding some sort of warning to the 2.3.1 download area. My fear is that some large OS vendor (e.g, Redhat) will pick up 2.3.1 for a future release.

Jason R. Mastaler <jason@mastaler.com> wrote:
Too late: 2.3.1 is in at least some versions of Debian already -- I'm getting reports on the os.fsync() issue already from my users (it breaks getmail). Could 2.3.1 be revoked or replaced ASAP? "2.3.1b" or similar with only this change would suit me. Charles -- ----------------------------------------------------------------------- Charles Cazabon <python@discworld.dyndns.org> GPL'ed software available at: http://www.qcc.ca/~charlesc/software/ -----------------------------------------------------------------------

That doesn't fit in the numbering scheme, it would have to be 2.3.2. Given the amount of traffic about this I'm tempted to agree that we need to move quickly and issue a fix for this issue (only). --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]
[Anthony Baxter]
We really should fix the HP/UX build problem as well.
That sounds dangerous to me: there's *always* an HP/UX build problem, if you count (as I do) getting threads to work. Every release, some HP/UX user submits a patch that breaks the build for other HP/UX users. So unless we've got a bona fide HP/UX expert on tap now (do we? that would be way cool <wink>), don't hold up solving a clear and widespread Linux problem by waiting for one to appear.

On the scale of "HP/UX build problems we've known and loved", this one's a trivial one. https://sourceforge.net/tracker/?func=detail&atid=105470&aid=811160&group_id=5470 has a patch (from the gcc bugtracker - they ran into the exact same problem) Now, the readline+no-threads=boom coredump, well, that involves threads, and the original reporter even mentions HP/UX, but it's more a simple-ish one - missing #ifdef WITH_THREAD calls, as far as I can see. https://sourceforge.net/tracker/?func=detail&atid=105470&aid=811844&group_id=5470 (Someone should probably do a scan of the code in Modules to make sure all the threading API calls are wrapped in WITH_THREAD #ifdefs. Not me, now, though - way past bedtime). -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

>> So unless we've got a bona fide HP/UX expert on tap now (do we? that >> would be way cool <wink>), don't hold up solving a clear and >> widespread Linux problem by waiting for one to appear. Jason> Just to be clear, the os.fsync() problem is not Linux-only. It's Jason> also on FreeBSD (probably every Unix platform actually). Correct. There are two new C macros in pyconfig.h, HAVE_FSYNC and HAVE_FDATASYNC. When originally added, the former was mistyped as HAVE_SYNC. Skip

On Fri, 2003-09-26 at 13:52, Anthony Baxter wrote:
If we only fix the few critical problems -- CVS directories, fsync problem, Mac version numbers (?) -- then there doesn't seem to be much risk. But it wouldn't be too hard to convince me. (I gave my all clear on 2.3.1, after all.) Jeremy

[Jeremy Hylton]
We should treat 2.3.1 like 2.3.1rc1 and aim for Sep. 30 for a 2.3.2.
[Anthony Baxter]
Wouldn't it be better to cut a 2.3.2rc before the final 2.3.2?
once-bitten-ly,
Well, Jeremy's "Sep. 30" is only 4 days away. If 2.3.2 is restricted to fixing just the obvious big problems already reported against 2.3.1, in reality it's like Jeremy said: 2.3.1 was really 2.3.2c1 <0.5 wink>. Don't get too gun-shy. There were lots of changes in who did what for the 2.3.1 release, and some glitches were inevitable. It *had* gotten so smooth with the same people doing the same things all the time that, e.g., you must have noticed that Guido stopped doing anything for releases. I would have liked to pay a lot more attention but just couldn't make time for it; I'm sure the same is true of the other PythonLabs alumni (including Guido). So some things could have gone better. It's not that big a deal -- next time they will go better. Remember that we pioneered all possible ways to screw up a release already, and discovered that it's a finite set <wink>.

On Fri, Sep 26, 2003 at 02:24:27PM -0400, Tim Peters wrote:
Don't get too gun-shy. There were lots of changes in who did what for the 2.3.1 release, and some glitches were inevitable.
Some humble advice from someone who has done quite a few software releases in the last few years (mostly internal). The release process should be automated as much as possible. I think the Python project could do a bit better in this regard. Some ideas: * try to reduce the number of version numbers in different files need to be adjusted. Also, try to automate the adjustment. * write scripts to generate the final package and also verify its sanity (e.g. check version numbers, dates, detect files that should not be included in the release) * write scripts to upload the package for distribution. Ensure proper naming, permissions, etc. Python already has a release checklist. That's good. Be sure to follow it when doing a release and make sure it's up-to-date. Unfortunately Python doesn't do releases often enough for the process to be really polished. One effective way of improving the process is to have a new person do it with a veteran looking on. The new person should try to complete the release using only the checklist as the guide. The veteran is on hand to make sure nothing goes wrong. The veteran should also take notes; documenting any errors or omissions in the release checklist or ideas on ways to further automate the process. Neil

Neil Schemenauer <nas@arctrix.com> writes:
* try to reduce the number of version numbers in different files need to be adjusted.
I suggest to update the version numbers in these files to the next version *after* a release is done. So, in the release23-maint branch the version numbers could *now* be adjusted to 2.3.2. And it would help if I could build the HTML docs myself from CVS. I did manage to create the pdf files with TeTex under windows, but I didn't succeed with the html pages so far. This way, a Windows release could be built at any time, and I would not only install and test it, but also use it as my default version. Thomas

Thomas Heller <theller@python.net> writes:
It occurs to me that I don't know *why* Fred is so much the documentation man; I've not had any trouble processing the docs into HTML lately (haven't tried on Windows, admittedly, and I haven't tried to make info ever). What else needs to be done? There must be quite a bit of mucking about on creosote to do, I guess. Finding a substitue Jack might be harder (though given Bob Ippolito's work rate at the moment we could probably make him MacPython maintainer without him noticing <wink>). Cheers, mwh -- In short, just business as usual in the wacky world of floating point <wink>. -- Tim Peters, comp.lang.python

[cc to webmaster@starship.python.net] Michael Hudson <mwh@python.net> writes:
I could do it myself if the tools needed to build the docs would be available on the starship. Last time I checked, neither Tex nor latex2html were installed. So, I hereby kindly request: Could the tools to build the Python docs from source please be installed on the starship? Thanks, Thomas

On 27 September 2003, Thomas Heller said:
OK, I'm installing tetex-base and latex2html now. No guarantees if the version of latex2html in Debian testing will actually work, of course. ;-) (No, wait, it should work: I have the same version (2000-beta1-8) on my PC, and the last time I did Python doc work, I *think* I was able to produce HTML output.) Greg -- Greg Ward <gward@python.net> http://www.gerg.ca/ A closed mouth gathers no foot.

Greg Ward <gward@python.net> writes:
Somehow this seems to have avoided installing some important executables -- like, e.g., "latex". Are there some other packages that need to be installed too? Cheers, mwh -- 58. Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html

Michael Hudson wrote:
Looks like you need tetex-bin, judging from my debian (unstable) system: bash-2.05b$ dpkg -S `which latex` tetex-bin: /usr/bin/latex It's probably worth grabbing tetex-extra, too; its description says: ... Together with tetex-bin and tetex-base, this will give you a complete teTeX installation. -Andrew.

On 29 September 2003, Michael Hudson said:
Argh. Installing tetex-bin (and -doc, -extra, -lib just for fun) now. Greg -- Greg Ward <gward@python.net> http://www.gerg.ca/ A man without religion is like a fish without a bicycle.

On 26-sep-03, at 23:34, Michael Hudson wrote:
Not that I'm planning to get run over by a tram shortly, but just in case: I think Just or Ronald or yourself or any various other python-devvers would make a better substitute *at this point in time*. Bob does some truly wonderful things at terrific speed, but that has some drawbacks for maintainability, IMHO. Being MacPython maintainer is 10% doing fun stuff and 90% taking care of all the nitty-gritty details, making everything work in as many situations as possible, documenting, etc etc etc. -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman

Michael Hudson writes:
It's certainly gotten easier to deal with the documentation on modern Linux distributions. At CNRI, we used mostly Solaris boxes, and I have to build my own teTeX installations from source, and hand-select a version of LaTeX2HTML that worked for me. At this point, all the software that I can't just install from a RedHat CD is part of what gets pulled down from CVS. I've been able to build the docs on Cygwin as well, though I've not tried lately. A lot of what it takes to build the docs is written into Doc/Makefile, but it does require a solid make (it even uses $(shell ...) now, so maybe only GNU make will do; not sure).
What else needs to be done? There must be quite a bit of mucking about on creosote to do, I guess.
There's a bit, but that's getting easier and easier as I've gone through it a few times now. I updated PEP 101 the other evening so anyone can do what's needed to build the packages and get them in the download locations. There's more to be written to explain what else needs to be updated on the site. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

"Fred L. Drake, Jr." <fdrake@acm.org> writes:
Oh, there was a reason I put a "lately" in what I said...
One thing that puzzled me: Doc/Makefile seems to require that Doc/tools is on $PATH, unless I'm misunderstanding something.
Well, progress! I think it's a worthy goal that no single person is required to make a release, and that actually this isn't too far off. Cheers, mwh -- Never meddle in the affairs of NT. It is slow to boot and quick to crash. -- Stephen Harris -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html

Michael Hudson writes:
One thing that puzzled me: Doc/Makefile seems to require that Doc/tools is on $PATH, unless I'm misunderstanding something.
It definately doesn't require that; I've never used Doc/tools/ on $PATH. One thing it was requiring (only recently) was that there was a mkhowto symlink somewhere on the $PATH that pointed to the mkhowto script. I've removed that constraint for the trunk. The intention is that we should be able to use a mkhowto script from a different checkout; you can still modify the MKHOWTO make variable to do that, but it's not so valuable on the trunk as on the maintenance branches (where we want to use mkhowto from the trunk). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
participants (23)
-
"Martin v. Löwis"
-
A.M. Kuchling
-
Andrew Bennetts
-
Anthony Baxter
-
Barry Warsaw
-
Charles Cazabon
-
Fred L. Drake, Jr.
-
Greg Ward
-
Guido van Rossum
-
Jack Jansen
-
Jack Jansen
-
Jason R. Mastaler
-
Jeremy Hylton
-
martin@v.loewis.de
-
Michael Hudson
-
Neil Schemenauer
-
Raymond Hettinger
-
Shane Hathaway
-
Sjoerd Mullender
-
Skip Montanaro
-
Terry Reedy
-
Thomas Heller
-
Tim Peters