[Python-checkins] CVS: python/nondist/peps pep-0101.txt,NONE,1.1
Barry Warsaw
bwarsaw@users.sourceforge.net
Wed, 22 Aug 2001 15:41:42 -0700
Update of /cvsroot/python/python/nondist/peps
In directory usw-pr-cvs1:/tmp/cvs-serv3078
Added Files:
pep-0101.txt
Log Message:
PEP 101, Doing Python Releases, Warsaw & van Rossum
--- NEW FILE: pep-0101.txt ---
PEP: 101
Title: Doing Python Releases 101
Version: $Revision: 1.1 $
Last-Modified: $Date: 2001/08/22 22:41:40 $
Author: barry@zope.com (Barry A. Warsaw), guido@python.org (Guido van Rossum)
Status: Active
Type: Informational
Created: 22-Aug-2001
Post-History:
Abstract
Making a Python release is an arduous processes that takes a
minimum of half a day's work even for an experienced releaser.
Until recently, most -- if not all -- of that burden was borne by
Guido himself. But several recent releases have been performed by
other folks, so this PEP attempts to collect, in one place, all
the steps needed to make a Python release.
(Note: These steps were recorded during the Python 2.2a2 release
done by Barry and Guido. It will be checked for accuracy with the
subsequent release, to be done by Barry.)
How to Make A Release
Here are the steps taken to make a Python release. Some steps are
more fuzzy than others because there's little that can be
automated (e.g. writing the NEWS entries). Where a step is
usually performed by An Expert, the name of that expert is given.
Otherwise, assume the step is done by the Release Manager (RM),
the designated person performing the release. Almost every place
the RM is mentioned below, this step can also be done by Guido,
naturally!
XXX: We should include a dependency graph to illustrate the steps
that can be taken in parallel, or those that depend on other
steps.
We use the following conventions in the examples below. Where a
release number is given, it is of the form X.YaZ, e.g. 2.1a3 for
Python 2.2 alpha 2, where "a" == alpha, "b" == beta, "rc" ==
release candidate, and "f" == final. If a micro release number is
used, then we'll say X.Y.MaZ.
- 3 or 4 days before the release, create a branch for X.YaZ.
All Python development happens on the trunk. A few days before
the release, the RM will create a branch for this release. Only
the RM is authorized to make commits to this branch, but
checkins can proceed as normal on the trunk. It is the
responsibility of the RM to decide on a case-by-case basis which
trunk modifications should be merged into the branch.
To create a branch the following steps are taken:
* CVS tag the trunk with the symbolic name "rXYaZ-fork", e.g.
% cvs tag r2.2a3-fork
* Make the branch with the symbolic name "rXYaZ-branch", e.g.
% cvs tag -b r2.2a3-branch
* Check out a clean version of the branch into a new directory.
You'll be doing a lot of work in this directory and you want
to keep it straight from your trunk working directory. E.g.
% cvs -d <cvsroot> -q co -d python-22a3 python/dist/src
- For the next 3 or 4 days, selectively merge stuff from trunk
into branch. For each change you see on the trunk (i.e. via the
python-checkins mailing list), you need to decide whether the
change should also be applied to the branch.
If so, it's fairly easy to apply the change by diffing the file
and patching it manually. You can also sometimes get away with
just copying the file from the trunk directory to the branch
directory, but be careful so you don't lose changes that only
exist in the branch!
- The night before the release, the RM updates the Misc/NEWS file,
adding high level items new to this release. E.g. if we're
releasing 2.2a3, there must be a section at the top of the file
explaining "What's new in Python 2.2a3". It will be followed by
a section entitled "What's new in Python 2.2a2".
Note that you /hope/ that as developers add new features to the
trunk, they've updated the NEWS file accordingly. You can't be
positive, so double check. If you're a Unix weenie, it helps to
verify with Tim Peters about changes on Windows, and Jack Jansen
about changes on the Mac.
- Also the night before the release, a freeze is put on checkins
into the branch. At this point, nobody except Fred Drake or Tim
Peters should make any commits to the branch. If the RM screwed
up and some desperate last minute change to the branch is
necessary, it can mean extra work for Fred and Tim. So try to
avoid this!
- Once the branch is frozen, Fred Drake needs to create the HTML
from the documentation. He does this and uploads the file to
www.python.org. Then he tells Tim Peters where this file is.
This may generate some last minute changes on the branch.
- Tim Peters grabs the HTML and uses this to build the Windows
installer. Tim then creates a new "release" named X.YaZ on the
SourceForge file release manager.
(Diversion: SF's file manager has "packages" and "releases". We
use packages to name major upcoming releases, e.g. python-2.2 or
python-2.1.1. Inside each package are a number of "releases"
for each new actual release -- i.e. the thing you're building.
An example of a release name is 2.2a3. Once created, packages
and releases are never deleted, but old ones are hidden to
reduce confusion. More on this below.)
If this is the first release for this major Python version, Tim
will create a new package containing the major Python version
number.
- Tim does performs his Windows magic, generating an installer
executable. He uploads this file to SourceForge under the
release he just created. He then sends the RM a notice which
includes the MD5 checksum of the Windows executable.
Note that Tim's creation of the Windows executable may generate
a few more commits on the branch.
- Now, you're ready to build the source tarball. First cd to your
working directory for the branch. E.g.
% cd .../python-22a3
- Do a "cvs update" in this directory. You should not see any "M"
files, but you may see several "P" files. I.e. you better not
have any uncommitted changes in your working directory, but you
may pick up some of Fred's or Tim's last minute changes.
- Now tag the branch using a symbolic name like "rXYaZ",
e.g. r22a3
% cvs tag r22a3
- Change to a neutral directory, i.e. one in which you can do a
fresh, virgin, export of the branch. You will be creating a new
directory at this location, to be named "Python-X.YaZ". Do a
CVS export of the tagged branch.
% cd ~
% cvs -d <cvsroot> export -rr22a3 -d Python-2.2a3 python/dist/src
- Generate the tarball. Note that we're not using the `z' option
on the tar command because 1) that's only supported by GNU tar
as far as we know; 2) we're going to max out the compression
level, which isn't a supported option.
% tar cf - Python-2.2a2 | gzip -9 > Python-2.2a2.tgz
- Calculate the MD5 checksum of the tgz file you just created
% md5sum Python-2.2a2.tgz
Note that if you don't have the md5sum program, there is a
Python replacement in the Tools/scripts/md5sum.py file.
- Now you want to perform the very important step of checking the
tarball you just created, to make sure a completely clean,
virgin build passes the regression test. Here are the best
steps to take:
% cd /tmp
% tar zxvf ~/Python-2.2a3.tgz
% cd Python-2.2a3
% ls
(Do things look reasonable?)
% ./configure
(Loads of configure output)
% make test
(Do all the expected tests pass?)
If the tests pass, then you can feel good that the tarball is
fine. If some of the tests pass, or anything else about the
freshly unpacked directory looks weird, you better stop now and
figure out what the problem is.
- Start your upload to SF. You need to get Python-2.2a3.tgz into
SourceForge. This can take a while both because of the time it
takes to upload 650+KB of data, /and/ because SF has a 30 minute
delay built into the file release process. The next few steps
can be taken in parallel, so it's best to start the upload now
and keep an eye on its progress.
I've found that the `ncftpput' program is a great tool to use if
you have it available. You can execute the following command to
do the upload:
% ncftpput -F upload.sf.net incoming Python-2.2a3.tgz
If you don't have ncftpput around, you can use whatever ftp
client you're comfortable with. Just be sure that you're
uploading this to the "incoming" directory on upload.sf.net.
- You also need to upload the tgz file to creosote.python.org.
Usually Tim will have already uploaded the exe file to creosote,
but if not, you'll need to do that too. These steps can take a
long time depending on your network bandwidth. You have two
choices:
1) Upload them to SF first, then wget them from creosote. Pros:
easy to do; much friendlier to your own personal bandwidth.
Cons: can take even longer because you're subject to the 30
minute SF file upload delay, and the upload rate from
SF->creosote never seems to get above 20 KB/sec.
2) scp both files from your own machine to creosote. Pros: you
avoid the 30 minute SF delay. Cons: you don't get much else
done if you're on a small pipe.
- While you're waiting, you can start twiddling the web pages to
include the announcement.
XXX: This section will be rewritten once we CVS-ify the web
pages! It's only a rough sketch for now because only the
Pythonlabs guys can do this part, and there's no need to expose
our cruft. ;)
% (on leroy) cd ~guido/w/X.Y
% cp index.ht new-index.ht
(Edit the file for content: usually you can globally replace
X.Ya(Z-1) with X.YaZ. However, you'll need to think about the
"What's New?" section. You also need to watch out about two
specific links in the file, one which references "the full
scoop" on changes -- a shownotes link, and one which references
the downloads on SF -- a showfiles link. You won't be able to
update either link until you've completed the steps below. For
now just note what needs to change. Also, update the MD5
checksums.)
- Preview the web page by doing a "make" -- NOT a "make install".
View the page via a file: url.
- Similarly, edit the ../index.ht file, i.e. the python.org home
page. In the Big Blue Announcement Block, move the paragraph
for the new version up to the top and boldify the phrase "Python
X.YaZ is out". Edit for content, and preview as above. Do NOT
do a "make install" yet!
- Now we're waiting for the ncftpput command, and the scp to
creosote to finish. Da de da, da de dum, hmm, hmm, dum de dum.
- Do the SourceForge file release dance.
* Go to the Python project and click on "Admin"
* Click on "Edit/Release Files"
* Since Tim has usually by now created the package and release
we're going to use, scroll down and click on "Edit Releases"
for the package we're releasing into.
* Find the release named X.YaZ and click on "Edit This Release"
You should now perform Step 1 of the file release dance...
* The "Status" field should be "Active" not "Hidden"
* In the text box that says "Paste The Notes In", paste in all
the "What's New" entries from the Misc/NEWS file that describe
this major version of Python, /not/ just the ones for this
particular release. E.g. If we're releasing Python 2.2a3,
we'd include the "What's New" sections for Python 2.2a3,
2.2a2, and 2.2a1.
* Leave the "Paste The Change Log In" section blank, but click
on "Preserve my pre-formatted text".
* Hit the Submit/Refresh button for Step 1.
This will bring you back to the file release page. DO NOT do
this step until your ftp upload is complete! Once it is, you
can perform Step 2 of the file release dance...
* Click on the checkbox next to the file Python-X.YaZ.tgz. Be
sure no other box is checked! Then click on the "Add Files
and/or Refresh View" button for Step 2.
And now, Step 3...
* There should be exactly two files listed here, one is the tgz
file you just added, and the other is the exe file that Tim
added earlier.
* For the tgz file, be sure that the "Processor" field says
"Any" and the "File Type" field says "Source .gz".
* Click on "Update/Refresh" for the .tgz file.
* For the exe file, make sure that the "Processor" field says
"i386" and the "File Type" field says "Other". Tim usually
gets this right <wink>, but if not, make any necessary changes
and click on "Update/Refresh" for the exe file.
Step 4...
DO NOT DO STEP 4 NOW. Wait until after you send out the email
announcement to send the SF email notice.
- Still on SF, click on the "Files" button at the top of the
page. Find the release you've just made and click on it -- not
on the tgz or exe file, but on the release link under the
package name. E.g. package named python-2.2, click on the
"2.2a3" link.
This should be a page that says "Release Name: X.YaZ" and it
should contain the "What's New" sections you pasted in earlier.
Note the url of this page. Copy and paste it into the
~guido/w/X.Y/new-index.ht file you created above. This is the
"shownotes" link mentioned earlier.
- Now click on the "Summary" link at the top of the page and
scroll down to the "Latest File Releases" section. Find the
package you just made a release for (the Version should be
X.YaZ, and the Date should be today's date). Click on the
"Download" link.
Your new release should be highlighted in pink. Note the url
for this page. Copy and paste it into the
~guido/w/X.Y/new-index.ht file from above. This is the
"showfiles" link mentioned earlier.
- Now you need to go to creosote.python.org and move all the files
in place over there. Our policy is that every Python version
gets its own directory, but each directory may contain several
releases. We keep all old releases, moving them into a "prev"
subdirectory when we have a new release.
So, there's a directory called "2.2" which contains
Python-2.2a2.exe and Python-2.2a2.tgz, along with a "prev"
subdirectory containing Python-2.2a1.exe and Python-2.2a1.tgz.
So...
* On creosote, cd to ~ftp/pub/python/X.Y creating it if
necessary.
* Move the previous release files to a directory called "prev"
creating the directory if necessary (make sure the directory
has g+ws bits on). If this is the first alpha release of a
new Python version, skip this step.
* Move the .tgz file and the .exe file to this directory. Make
sure they are world readable. They should also be group
writable.
- Now preview the new-index.ht file once more. IMPORTANT: follow
every link on the page to make sure it goes where you expect it
to go, and that what you expect to be there is there.
- If everything looks good, move new-index.ht to index.ht and do a
"make install" in this directory. Go up to the parent directory
(i.e. the root of the web page hierarchy) and do a "make
install" there too. You're release is now live!
- Now it's time to write the announcement for the mailing lists.
This is the fuzzy bit because not much can be automated. You
can use one of Guido's earlier announcements as a template, but
please edit it for content!
Once the announcement is ready, send it to the following
addresses:
python-list@python.org
python-announce@python.org
python-dev@python.org
- Go back to the file releases page on SF and complete Step 4,
sending out the email notification.
Now it's time to do some cleanup. These steps are very important!
- Go back to SF, Admin->Edit/Release Files. Click on "Edit
Releases" for the package you just added to. For each old
release, click on "Edit This Release" and under Step 1, change
the "Status" to "Hidden". Click on the Step 1 Submit/Refresh
button.
- Merge the branch back into the trunk! Now that we've released
this branch, we don't need it any more. We've already tagged it
so we can always reproduce it. Note that merging branches is a
bit of a black art, but here's what's worked for us.
* Check out a completely clean, virgin working directory of the
trunk, by doing this in the directory that is the parent of
your branch working directory python-XYaZ:
% cvs -d <cvsroot> co -d python-clean python/dist/src
* Run a diff against your branch by doing this in the common
parent directory containing both python-clean and python-XYaZ:
% diff -r python-clean python-22a2 | grep ^diff | grep -v CVS
* Take the output of this and stick it in a file,
e.g. /tmp/diffcmd.sh
* Edit diffcmd.sh to get rid of files that you know don't have
important changes. You're looking for files that have updates
in the branch that haven't made it to the trunk. If you've
been diligent about merging changes from the trunk into the
branch, there shouldn't be many of these files.
* Edit /tmp/diffcmd.sh, changing all the -r's into -u's. Run
the /tmp/diffcmd.sh command like so:
% sh /tmp/diffcmd.sh > /tmp/pydiff.txt
* Attempt to patch your python-clean working directory. Do this
first, noting that --dry-run does not actually apply any
patches, it just makes sure that the patch command runs
successfully to completion:
% patch -p1 --dry-run < /tmp/pydiff.txt
* If this goes well, run it again, taking out the --dry-run
option. If this fails, or if it prompts you for a file to
patch, try using -p0 instead of -p1. Otherwise, your diff
command was messed up, so try again.
* cd to python-clean and do a "cvs commit". Use as your log
message something like "Merging the rXYaZ-branch tag back into
the trunk".
* Edit the file Include/patchlevel.h so that the PY_VERSION
string says something like "X.YaZ+". Note the trailing `+'
indicating that the trunk is going to be moving forward with
development. E.g. the line should look like:
#define PY_VERSION "2.2a2+"
Commit this change.
* Now test your clean, merged trunk by doing
% make distclean
% ./configure
% make test
To ensure that the regression test suite passes. If not, you
screwed up somewhere!
What Next?
Rejoice. Drink. Be Merry. Write a PEP like this one. Or be
like unto Guido and take A Vacation.
You've just made a Python release!
Copyright
This document has been placed in the public domain.
Local Variables:
mode: indented-text
indent-tabs-mode: nil
End: