From: M.-A. Lemburg [mailto:firstname.lastname@example.org]
> I don't know why distutils installs in Python\Lib\ instead of
> e.g. Lib\site-packages\, but I do know that you override this
> option using setup.cfg, so that your particular installer
> installs in a different destination directory.
But that only works if you're using distutils to install. As was pointed
out, the installer doesn't (currently?) respect this setting. For modules
with complex dependencies (OpenGL, Numeric, wxWindows are examples) building
from scratch is often not a realistic option.
> The problem of course is that distutils default setting will
> become a standard and changing the location would only cause more
Exactly my point. Is the current distutils default the best standard? If
not, we should change it NOW, before the user base is too large to even
contemplate a change. If the distributed site.py doesn't support the
"obvious" place (site-packages), surely now is the time to lobby for the
> There is no such thing as bdist_zip; bdist_dumb is probably what
> you meant and yes, it works the way you describe.
Thanks - sorry for the confusion. Thomas said that it currently uses
absolute paths rather than relative, which means it needs fixing. But
assuming it is fixed, would it get used? Would you distribute the mx utils
as a bdist_dumb file as well as the bdist_wininst one you currently supply?
The package repository proposal may make this happen, by distributing the
job of building binary distributions. I don't know...
From: M.-A. Lemburg [mailto:email@example.com]
> > PS Do the installers butlt by distutils (bdist_wininst and
> > the like) respect setup.cfg? If not, I believe they should.
> > Or at least, the installer should include a way for the end
> > user to specify an alternative install directory...
> Hmm, the build commands do respect the settings in the setup.cfg file,
> but the installer itself doesn't.
> It would be nice, if the wininst installer would allow even more
> flexibility, e.g. allow setting the install directory (this would
> then have to generate a .pth file to properly setup the Python path),
Not necessarily. All I care about is that I absolutely will not accept the
default of putting modules in the Python executable directory. I feel that
this is completely wrong, and needs change. There is a perfectly respectable
Lib/site-packages directory, but Python ignores it (on Windows). The fix is
a 1-line change to site.py, which I understand is not done because of some
form of policy issue, rather than for any technical reason (ie, Guido
doesn't like it...)
I have hacked site.py to override this. I want to override the default
location where modules are installed, as well. I simply will not use an
installer which dumps arbitrary modules in the Python executable directory.
The wininst installer is hackable, as it is basically an exe prepended onto
a zip file, so I can unzip the distribution wherever I like. But I wish I
didn't need to do this. OTOH, I don't want my Windows "add/remove programs"
applet cluttered with loads of Python modules, either (which I believe is
also something the wininst installer does). So personally, I would prefer to
just have a zip which I could install myself...
Two questions - is the bdist_zip option functional (no time to try it just
now)? And how can we persuade package authors to provide a bdist_zip version
of their packages as well as an installer version?
(Sorry - I HATE installers, and particularly so when I have no way of
controlling or limiting what they do...)
Playing with the new meta info, I noticed that the
license specified in this way:
setup(..., license="MIT/X11", ...)
isn't recognized, while it works if one uses:
setup(..., licence="MIT/X11", ...)
I've updated my prototype catalog server
It now keeps track of who uploaded what. Plus, the registration process
sends email, so you need to provide a valid email address. Also, you can
now edit your user account information. It still does not implement pep
To contribute packages, get the lastest distutils (1.0.2pre) which
implements PEP 241. Create a package. Create a user account on the
catalog server, and upload your your package.
If you just want to play around, you can manually create a PKG-INFO file
for your archive and upload that.
I'd love feedback. Does this seem like I'm going in the right direction?
What could be done to make it easier for uploaders and/or downloaders.
Did you encounter any errors? Does the server not parse your package
P.S. The source of the catalog server is available on the catalog
server. (Though, I cheated, and added a PKG-INFO file manually.)
Amos Latteier mailto:firstname.lastname@example.org
Digital Creations http://www.digicool.com
Various events related to the Catalog- and Distutils-SIG happened at
IPC9 this past week. To summarize them:
* Sean Reifschneider demonstrated swalow during the Lightning Talks
session and got a favorable reaction from the attendees.
* Packaging was discussed at Paul Prescod's BoF session, and some
decisions were made there. (More about that below...)
* On Developer's Day, Moshe turned over half of his "Batteries Included"
session to me for catalog discussion, and a few more items were
added to the task list.
So, here's the plan of action:
1) For 2.1final, change the Distutils sdist command to construct a
text file containing the metadata for a package. Exactly *what*
metadata to collect was left open for subsequent discussion on the
Catalog-SIG. (Amos has a patch to implement this that I'll look at
when I can.)
This is the only thing to be done in time for 2.1, so it's the only
really pressing task.
2) Once that metadata support is there, someone can start running an
experimental swalow server and begin adding to its database.
If Python 2.2 comes along in 6-12 months, that should be enough
time to have an idea of what should go into 2.2 to support it.
3) Other Distutils changes would be to design and create a package
database, and implement an uninstall command. The 'sdist' command
would also grow an --upload or --register option that would
automatically submit the package to the catalog (but first we'd need
to finalize how that should be done).
I'll make a separate post to start the metadata discussion.
I am trying to distutilsify my Python packages, but I am stuck at an
almost undocumented point: installation of header files. In fact, the
Distutils manual only says that it can be done, but not how, and the
only real-life example that does it seems to be NumPy.
Here's my situation: I have a distribution called "ScientificPython"
which contains a Python package "Scientific", which in turn contains
an extension module for which some header files exist. These files
are supposed to end up in $prefix/pythonx.y/Scientific, such that
they can be included as "Scientific/xxx". However, if I do exactly what
NumPy does, they end up in $prefix/pythonx.y/ScientificPython. The
only way I found to change the header installation directory is by
specifying an option during installation, but I want this defined
in setup.py. Any suggestions?
Konrad Hinsen | E-Mail: hinsen(a)cnrs-orleans.fr
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-126.96.36.199.24
Rue Charles Sadron | Fax: +33-188.8.131.52.17
45071 Orleans Cedex 2 | Deutsch/Esperanto/English/
France | Nederlands/Francais
From: Thomas Heller [mailto:email@example.com]
> > The wininst installer is hackable, as it is basically
> > an exe prepended onto a zip file, so I can unzip the
> > distribution wherever I like. But I wish I didn't need
> > to do this. OTOH, I don't want my Windows "add/remove
> > programs" applet cluttered with loads of Python modules,
> > either (which I believe is also something the wininst
> > installer does).
> Only in the newest version (1.0.2pre, as in python2.1b2)
> the module is added to the add/remove programs list. Of
> course I could include a flag into the installer to let
> the user decide whether to record deinstall information,
> but normally this is not decided by the poor user. IMO
> most Windows installers are already asking questions where
> no one can provide the correct answer (You have already
> installed xxx.dll in anther language and in another
> version. Would you like it to be replaced?)
Agreed, I know of no other installer which offers the option to not set up
uninstall info. On the other hand, I know of no other case of "parts" of a
system having separate installers, so there's not much precedent. I have 17
or so modules - which would be half as many again entries in my add/remove
In the absence of a Python-specific "installed modules" repository (which I
actually think would be a better approach), I don't see much reason to
change from the Windows uninstall facility. But maybe having the description
*always* start with "Python -" (for example "Python - Numeric", "Python -
wxPython") would at least keep them together.
> > So personally, I would prefer to just have a zip which
> > I could install myself...
> Since the installer is an exe prepended onto a zip file
> (as you noted), Winzip is happy to open and extract it as
> a zip file. Maybe if this fact would be documented better,
> it could come to rescue for the 'expert' user who has his
> own opinions about the install location, uninstall, and so
If that is guaranteed to remain the case, then yes it should be documented.
> > Two questions - is the bdist_zip option functional
> > (no time to try it just now)? And how can we persuade
> > package authors to provide a bdist_zip version of their
> > packages as well as an installer version?
> Last time I checked, 'bdist --formats=zip' creates
> a zip-file with pathnames relative to the root
> directory 'python20/package/module.py' or even
> worse 'Program Files/python/package/module.py' or
> 'Programme/python/package/module.py' depending on where
> your python is installed.
That's roughly what I remembered - and in my opinion that counts as "broken"
If format=zip remains broken, while format=bdist_wininst is working, zip
format will fall into disuse, to the extent that developers will not even
think of it - and so the only binary distributions available will be
installer-based :-( (Note: I don't have a problem with installers being
available, just with them being the only thing available).
> > (Sorry - I HATE installers, and particularly so when I
> > have no way of controlling or limiting what they do...)
> This is (mostly) true for any installers I know. (OK, the
> install location is normally changeable...)
True. Actually, what I hate with a vengeance is installers which don't tell
you what they *do*. Do they just install files in the application directory?
Do they add entries to the registry? Install files in the Windows directory?
Can I ask that somewhere, the full details of exactly what the wininst
installer does and does not do, be documented. What it doesn't do is at
least as important as what it does. No hidden magic! (And yes, I know
there's always the source. But sources can change - documentation is a
contract, which doesn't get violated.)
From: M.-A. Lemburg [mailto:firstname.lastname@example.org]
> This again defaults to:
> 'unix_prefix': # without PYTHONHOME set
> 'headers': '$base/include/python$py_version_short/$dist_name',
> 'unix_home': # with PYTHONHOME set
> 'headers': '$base/include/python/$dist_name',
> 'headers': '$base/Include/$dist_name',
> 'headers': '$base/Include/$dist_name',
I think the original poster was saying that he wanted to use something other
than $dist_name in these. Whether allowing that is a reasonable idea, I
can't say. But it's very different from wanting to alter the base install
PS Do the installers butlt by distutils (bdist_wininst and the like) respect
setup.cfg? If not, I believe they should. Or at least, the installer should
include a way for the end user to specify an alternative install
Included below is the version of PEP243 after it's initial round of review.
I welcome any feedback.
Title: Module Repository Upload Mechanism
Author: jafo-pep(a)tummy.com (Sean Reifschneider)
Type: Standards Track
For a module repository system (such as Perl's CPAN) to be
successful, it must be as easy as possible for module authors to
submit their work. An obvious place for this submit to happen is
in the Distutils tools after the distribution archive has been
successfully created. For example, after a module author has
tested their software (verifying the results of "setup.py sdist"),
they might type "setup.py sdist --submit". This would flag
Distutils to submit the source distribution to the archive server
for inclusion and distribution to the mirrors.
This PEP only deals with the mechanism for submitting the software
distributions to the archive, and does not deal with the actual
The upload will include the Distutils "PKG-INFO" meta-data
information (as specified in PEP-241 ), the actual software
distribution, and other optional information. This information
will be uploaded as a multi-part form encoded the same as a
regular HTML file upload request. This form is posted using
ENCTYPE="multipart/form-data" encoding [RFC1867].
The upload will be made to the host "modules.python.org" on port
80/tcp (POST http://modules.python.org:80/swalowpost.cgi). The form
will consist of the following fields:
distribution -- The file containing the module software (for
example, a .tar.gz or .zip file).
distmd5sum -- The MD5 hash of the uploaded distribution,
encoded in ASCII representing the hexadecimal representation
of the digest ("for byte in digest: s = s + ('%02x' %
pkginfo (optional) -- The file containing the distribution
meta-data (as specified in PEP-241 ). Note that if this is not
included, the distribution file is expected to be in .tar format
(gzipped and bzipped compreesed are allowed) or .zip format, with a
"PKG-INFO" file in the top-level directory it extracts
infomd5sum (required if pkginfo field is present) -- The MD5 hash
of the uploaded meta-data, encoded in ASCII representing the
hexadecimal representation of the digest ("for byte in digest:
s = s + ('%02x' % ord(byte))").
platform (optional) -- A string representing the target
platform for this distribution. This is only for binary
distributions. It is encoded as
signature (optional) -- A OpenPGP-compatible signature [RFC2440]
of the uploaded distribution as signed by the author. This may be
used by the cataloging system to automate acceptance of uploads.
protocol_version -- A string indicating the protocol version that
the client supports. This document describes protocol version "1".
The status of the upload will be reported using HTTP non-standard
("X-*)" headers. The "X-Swalow-Status" header may have the following
SUCCESS -- Indicates that the upload has succeeded.
FAILURE -- The upload is, for some reason, unable to be
TRYAGAIN -- The server is unable to accept the upload at this
time, but the client should try again at a later time.
Potential causes of this are resource shortages on the server,
administrative down-time, etc...
Optionally, there may be a "X-Swalow-Reason" header which includes a
human-readable string which provides more detailed information about
If there is no "X-Swalow-Status" header, or it does not contain one of
the three strings above, it should be treated as a temporary failure.
>>> f = urllib.urlopen('http://modules.python.org:80/swalowpost.cgi')
>>> s = f.headers['x-swalow-status']
>>> s = s + ': ' + f.headers.get('x-swalow-reason', '<None>')
>>> print s
FAILURE: Required field "distribution" missing.
The upload client must submit the page in the same form as
Netscape Navigator version 4.76 for Linux produces when presented
with the following form:
<FORM NAME="fileupload" METHOD="POST" ACTION="swalowpost.cgi"
<INPUT TYPE="file" NAME="distribution"><BR>
<INPUT TYPE="text" NAME="distmd5sum"><BR>
<INPUT TYPE="file" NAME="pkginfo"><BR>
<INPUT TYPE="text" NAME="infomd5sum"><BR>
<INPUT TYPE="text" NAME="platform"><BR>
<INPUT TYPE="text" NAME="signature"><BR>
<INPUT TYPE="hidden" NAME="protocol_version" VALUE="1"><BR>
<INPUT TYPE="SUBMIT" VALUE="Upload">
The following are valid os names:
aix beos debian dos freebsd hpux mac macos mandrake netbsd
openbsd qnx redhat solaris suse windows yellowdog
The above include a number of different types of distributions of
Linux. Because of versioning issues these must be split out, and
it is expected that when it makes sense for one system to use
distributions made on other similar systems, the download client
will make the distinction.
Version is the official version string specified by the vendor for
the particular release. For example, "2000" and "nt" (Windows),
"9.04" (HP-UX), "7.0" (RedHat, Mandrake).
The following are valid architectures:
alpha hppa ix86 powerpc sparc ultrasparc
I currently have a proof-of-concept client and server implemented.
I plan to have the Distutils patches ready for the 2.1 release.
Combined with Andrew's PEP-241  for specifying distribution
meta-data, I hope to have a platform which will allow us to gather
real-world data for finalizing the catalog system for the 2.2
 Metadata for Python Software Package, Kuchling,
[RFC1867] Form-based File Upload in HTML
[RFC2440] OpenPGP Message Format
This document has been placed in the public domain.
A smart terminal is not a smart*ass* terminal, but rather a terminal
you can educate. -- Rob Pike
Sean Reifschneider, Inimitably Superfluous <jafo(a)tummy.com>
tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python
I just read thru PEP 243 and got to the section "Return Data". That
information is bogus. HTTP already provides the upload client with various
status codes. Specifically, let's look at the list in the PEP:
PEP value HTTP Status
SUCCESS 200 (Ok)
FAILURE 4xx or 5xx
TRYAGAIN 4xx or 5xx
The PEP lists "resource shortage" as a possible TRYAGAIN error. That would
map to 507 (Insufficent Storage). I'm sure there are other values that would
make sense within this area.
The PEP goes on to say that a human-readable string should provide further
information. That is exactly what the body of the HTTP response is for.
So... the answer here is to use the HTTP status codes like they're intended,
and to use the HTTP response body as they're intended. This beats added
another protocol layer on top of the response.
p.s. I'm not on the Distutils-SIG; I'd appreciate a CC for replies.
Greg Stein, http://www.lyra.org/