Here is PEP 243, discussing how to make Distutils submit .tar.gz
packages (and the like) to the mythical catalog server. I've got code for
this prototyped, but give me a couple of days to make it completely
PEP-compliant.
Moshe has already commented:
a. you don't say which URL to post to (just spell it out:
POST http://modules.python.org:80/ (and let upload override that)
b. have the codes be HTTP codes and the headers HTTP non-standard
(X-) headers. how about using the HTTP headers and error-codes, and
have *all* the page human readable.
Good comments, I'll look at building them in...
I look forward to any comments -- I'd like to start working on distutils
patches within the week.
Thanks,
Sean
==========================================================================
PEP: 243
Title: Module Repository Upload Mechanism
Version: $Revision$
Author: jafo-pep(a)tummy.com (Sean Reifschneider)
Status: Draft
Type: Standards Track
Created: 18-Mar-2001
Python-Version: 2.1
Post-History:
Discussions-To: distutils-sig(a)python.org
Abstract
For a module repository system (such as Perl's CPAN) to be
successful, it must be as easy as possible for module authors to
submit their work. An obvious place for this submit to happen is
in the Distutils tools after the distribution archive has been
successfully created. For example, after a module author has
tested their software (verifying the results of "setup.py sdist"),
they might type "setup.py sdist --submit". This would flag
Distutils to submit the source distribution to the archive server
for inclusion and distribution to the mirrors.
This PEP only deals with the mechanism for submitting the software
distributions to the archive, and does not deal with the actual
archive/catalog server.
Upload Process
The upload will include the Distutils "PKG-INFO" meta-data
information (as specified in PEP-241 [1]), the actual software
distribution, and other optional information. This information
will be uploaded as a multi-part form encoded the same as a
regular HTML file upload request. This form is posted using
ENCTYPE="multipart/form-data" encoding.
The upload will be made to the host "modules.python.org" on port
80/tcp. The form will consist of the following fields:
distribution -- The file containing the module software (for
example, a .tar.gz or .zip file).
distmd5sum -- The MD5 hash of the uploaded distribution,
encoded in ASCII representing the hexadecimal representation
of the digest ("for byte in digest: s = s + ('%02x' %
ord(byte))").
pkginfo -- The file containing the distribution meta-data (as
specified in PEP-241 [1]).
infomd5sum -- The MD5 hash of the uploaded meta-data, encoded
in ASCII representing the hexadecimal representation of the
digest ("for byte in digest: s = s + ('%02x' % ord(byte))").
platform (optional) -- A string representing the target
platform for this distribution. This is only for binary
distributions. It is encoded as
"<os_name>-<os_version>-<platform architecture>".
signature (optional) -- A GPG signature of the uploaded
distribution as signed by the author. This may be used by the
cataloging system to automate acceptance of uploads.
Return Data
The upload will report the status of the upload by sending the
string "Upload status:" followed by one of the following:
SUCCESS -- Indicates that the upload has succeeded.
FAILURE -- The upload is, for some reason, unable to be
processed.
TRYAGAIN -- The server is unable to accept the upload at this
time, but the client should try again at a later time.
Potential causes of this are resource shortages on the server,
administrative down-time, etc...
Following the above string may be a human-readable string
providing further information. This message continues to the end
of the returned data-stream.
Returned data which does not fit the above format should be
treated as a temporary failure.
Sample Form
The upload client must submit the page in the same form as
Netscape Navigator version 4.76 for Linux produces when presented
with the following form:
<H1>Upload file</H1>
<FORM NAME="fileupload" METHOD="POST" ACTION="swalowpost.cgi"
ENCTYPE="multipart/form-data">
<INPUT TYPE="file" NAME="distribution"><BR>
<INPUT TYPE="text" NAME="distmd5sum"><BR>
<INPUT TYPE="file" NAME="pkginfo"><BR>
<INPUT TYPE="text" NAME="infomd5sum"><BR>
<INPUT TYPE="text" NAME="platform"><BR>
<INPUT TYPE="text" NAME="signature"><BR>
<INPUT TYPE="SUBMIT" VALUE="Upload">
</FORM>
Platforms
The following are valid os names:
debian
hpux
mandrake
redhat
solaris
suse
windows
yellowdog
The above include a number of different types of distributions of
Linux. Because of versioning issues these must be split out, and
it is expected that when it makes sense for one system to use
distributions made on other similar systems, the download client
will make the distinction.
Version is the official version string specified by the vendor for
the particular release. For example, "nt" (Windows), "9.04"
(HP-UX), "7.0" (RedHat, Mandrake).
The following are valid architectures:
alpha
hppa
ix86
powerpc
sparc
ultrasparc
Status
I currently have a proof-of-concept client and server implemented.
I plan to have the Distutils patches ready for the 2.1 release.
Combined with Andrew's PEP-241 [1] for specifying distribution
meta-data, I hope to have a platform which will allow us to gather
real-world data for finalizing the catalog system for the 2.2
release.
References
[1] Metadata for Python Software Package, Kuchling,
http://python.sourceforge.net/peps/pep-0241.html
Copyright
This document has been placed in the public domain.
Local Variables:
mode: indented-text
indent-tabs-mode: nil
End:
--
I used to think that the brain was the most wonderful organ in
my body. Then I realized who was telling me this. -- Emo Phillips
Sean Reifschneider, Inimitably Superfluous <jafo(a)tummy.com>
tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python
I want to get the last XXX removed from PEP 241 so it can be posted
to c.l.py.a and python-dev *today*. The missing bit is:
Platform (multiple use)
A (XXX whitespace? comma?)-separated list of platform
specifications. Platform specifications are limited to the
following list:
XXX copy list from PPD? SourceForge?
Example: 'XXX'
So, what to do? Some options are:
* Specify a short and incomplete list of platform specs:
MacOS/Windows/POSIX/BeOS/... State in the PEP that this list is
incomplete. Binary distributions will use a different metadata
header (Compiled-Platform, Supported-Platform, or something like
that) to specify the OS and CPU they were compiled for.
* Leave 'Platform' to be filled in for binary distributions. Developers
should put platform limitation notes in the 'Description'.
I lean toward #1, and if I don't hear any strong reasons to do otherwise,
that's what I'll check it and post, sometime after 5PM EST.
--amk
Dmitry Balabanov on python-help had requested information about why the
following Manifest.in line didn't work; he used the following line:
###
recursive-include 'Max Fray' *.txt
###
Because there were spaces in the directory, it didn't quite parse
correctly. I've isolated filelist._parse_template_line() to be
responsible for parsing out these lines.
_parse_template_line() uses a simple string.split() to break a line into
tokens. I'd like to know if it's a good idea to allow spaces in directory
names like that. I'm attaching a patch that tries to split the line and
takes quoted elements into account. I'm in digest mode, and a newbie at
distutils, so please be gentle. *grin*
---
Original message follows below:
---
>From dimonb(a)beep.ru Sat Mar 10 11:36:01 2001
From: dimonb(a)beep.ru (Balabanov Dmitry)
Date: Sat, 10 Mar 2001 11:36:01 +0000
Subject: [Python-Help] help with distutils
Message-ID: <01031011360100.00515@doors>
Hi!!
Problem is very simple but i can't solve it...
I have such string in MANIFEST.in:
recursive-include 'Max Fray' *.txt
But `python setup.py sdist --manifest-only` spawns:
running sdist
warning: sdist: missing required meta-data: url
reading manifest template 'MANIFEST.in'
warning: no files found matching 'Fray'' under directory ''Max'
warning: no files found matching '*.txt' under directory ''Max'
writing manifest file 'MANIFEST'
problem is rather simple but i don't know how to solve it with
simple methods:)
do you?
Best regards, DimonB
It has been reported by Riaan Booysen that executing
'python setup.py bdist_wininst' with a setup-script
not defining a version number dies with a traceback.
The reason is that distutils uses '???' as version number,
and xxx-???.win32.zip is an invalid filename on windows:
c:\util\zip.exe -rq C:\temp\test\build\bdist.win32\wininst\x-???.win32.zip .
zip error: Invalid argument
zip error: Could not create output file (C:/temp/test/build/bdist.win32/wininst/x-???.win32.zip)
creating 'C:\temp\test\build\bdist.win32\wininst\x-???.win32.zip' and adding '.' to it
Exception exceptions.AttributeError: "'ZipFile' instance has no attribute 'fp'" in <method ZipFile.__del__ of ZipFile instance at
00885B24> ignored
error: C:\temp\test\build\bdist.win32\wininst\x-???.win32.zip: Invalid argument
IMO this should be fixed so that either a check is done
for a version number - or - a version number of 0.0.0
should be used.
Thomas
[Followups set to distutils-sig(a)python.org, since that's where most of
the posters are.]
I've checked in a first draft of PEP 241. Here it is; please offer
comments and changes.
PEP: 241
Title: Metadata for Python Software Packages
Version: $Revision: 1.1 $
Author: A.M. Kuchling <amk1(a)bigfoot.com>
Type: Standards Track
Created: 12-Mar-2001
Status: Draft
Post-History:
Introduction
This PEP specifies the names and semantics of the fields used to store
metadata about Python software packages.
Including Metadata in Packages
The Distutils 'sdist' command will be modified to extract the
metadata fields from the arguments and write them to a file in the
generated zipfile or tarball. This file will be named METADATA
and will be placed in the top directory of the source
distribution (where the README, INSTALL, and other files usually
go).
Authors might expect to include a METADATA file of their own that
would be used instead of the generated file, but this will not be
permitted. It would be confusing if person B makes a custom
release of person A's software, modifies setup.py accordingly, but
uses identical metadata because person B didn't delete the
METADATA file. When running the 'sdist' command, a user-supplied
METADATA file will be ignored and a warning about this action will
be printed.
XXX are we sure RFC-822 is enough?
The METADATA file format is a single set of RFC-822 headers
parseable by the rfc822.py module. The field names listed in the
following section are used as the header names.
Fields
This section specifies the names and semantics of each of the
supported metadata fields.
Name
The name of the package. XXX what's the set of legal characters?
Example: 'BeagleVote'
Version
A string containing the package's version number. This
field should be parseable by one of the Version classes
(StrictVersion or LooseVersion) in the distutils.version
module.
Example: '1.0a2'
Platforms
A (XXX whitespace? comma?)-separated list of platform
specifications. Platform specifications are limited to the
following list:
XXX copy list from PPD? SourceForge?
Example: 'XXX'
Description
A one-line summary of what the package does.
Example: "A module for collecting votes from beagles."
Long-Description (optional)
A longer description of the package that can run to several
paragraphs. (Software that deals with metadata should not
assume any maximum size for this field, though one hopes that
people won't include their instruction manual as the
long-description.)
Example: 'This module collects votes from beagles in order to
determine their electoral wishes. Do NOT try to use this module
with basset hounds; it makes them grumpy.'
Keywords
A list of additional keywords to be used to assist searching
for this package in a larger catalog.
XXX Keywords should probably be constrained to be from a fixed
list, but I don't think this can be enforced by the 'sdist'
command. (The list might be large, and it could only be updated
with new Distutils releases, which seems too inflexible.)
Example: 'dog puppy voting election'
Home-page (optional)
A string containing the URL for the package's home page.
Example: 'http://www.example.com/~cschultz/bvote/'
Author (optional)
A string containing at a minimum the author's name. Contact
information can also be added, separating each line with
newlines.
Example: 'C. Schultz\nUniversal Features Syndicate\nLos Angeles, CA'
Author-email
A string containing the author's e-mail address. It can contain
a name and e-mail address in the legal forms for a RFC-822
'From:' header ("name <email>" or "email (name)"). It's not
optional because cataloging systems can use the e-mail portion
of this field as a unique key representing the author. A
catalog might provide authors the ability to store their GPG
key, personal home page, and other additional metadata *about
the author*. Author-related metadata fields are not covered by
this PEP. (XXX should they be? I think not, since nothing in
this PEP will deal with author data at all.)
Example: 'C. Schultz <cschultz(a)example.com>'
License
A string selected from a short list of choices, specifying the
license covering the package. Some licenses result in the
software being freely redistributable, so packagers and
resellers can automatically know that they're free to
redistribute the software. Other licenses will require
a careful reading by a human to determine the software can be
repackaged and resold.
The choices are:
XXX copy list from SourceForge, + 'Other'
Copyright
This document has been placed in the public domain.
Local Variables:
mode: indented-text
indent-tabs-mode: nil
End:
Recently, "M.-A. Lemburg" <mal(a)lemburg.com> said:
> Jack Jansen wrote:
> >
> > Could we add a field that would allow for the functionality that is
> > now in the (very little known) checkversion.py script?
>
> Doesn't the Version field have this information ?
The version field has half the info. The other half needed is an easy
way to find the most recent version. With a whole file of
metainformation it's probably the easiest to just make the current
metainformationfile readily accessible on the net (i.e. not only in
an installer archive) and make the URL "permanent". Actually the URL
doesn't even have to be permanent: if the version 1.0 metainfofile has
a URL www.veryoldsite.com/bla/package/metainfo.xml, and that
file (for version 2.0) has www.newersite.com/bla/package/metainfo.xml
you could keep a chain of the things.
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm
Could we add a field that would allow for the functionality that is
now in the (very little known) checkversion.py script?
I think that it would be good enough if the metadata file contained
the URL where the current metadata file is stored. What checkversion
would then do is run through your Python installation looking for
metadata files, and for each one found it would retrieve the current
version of that file over the net. If the version number in the
net-based file was higher than the one in the file on your disk it
would tell you there's a new version of the package available (and
probably offer to download it, etc).
--
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm
Andrew Kuchling <akuchlin(a)mems-exchange.org> wrote :
> On Wed, Mar 14, 2001 at 12:16:03PM -0800, Paul Prescod wrote:
> >I would suggest an algorithmic transformation from metadata to DTD
> >(perhaps performed by hand). For instance turn every metadata item into
> >an element of the same name with a value attribute or content. Then
> >there are no decisions to be made.
>
> Can you (or some volunteer) please write up a description of this
> transformation and post it? I can't really figure out what this would
> buy us, but am not quite sure what you're suggesting.
You may want to check vcalsax, available at
http://www.logilab.org/vcalsax/. This module reads a vcalendar file (used
by korganizer and gnomecal, which looks a lot like rfc822), and emits SAX2
events so that it is possible to process the file as if it were XML, and
thus, for example build a DOM tree out of it.
Alexandre Fayolle
--
http://www.logilab.com
Narval is the first software agent available as free software (GPL).
LOGILAB, Paris (France).
