This is my first email in distutils-sig mailing list. So I may introduce
myself. I am Carlos Tejo, a member of the Semantic Web activity line at
CTIC Foundation, a non-profit organization, where I work with a team of
research fellows. We are developing an small library  in python
language, and now we want to release it properly :-)
I wonder how I should "write" setup.py in order that setuptools use the
parameters of setup() to create a proper PKG-INFO file.
Right now, I have a couple of troubles:
- How can be configured the version of the metadata (AFAIK, rigth it is
always "Metadata-Version: 1.0")? I would like to use version 1.1 (PEP
314. status: final)  or version 1.2 (PEP 345. status: draft) 
- How should be written the "platform" parameter?
Carlos Tejo Alonso
Research & Development Department
CTIC Foundation [Asturias, Spain]
Remainder about Distutils:
- we did a summit at Pycon about distutils
- some people started to work on various tasks (PEPs, etc)
Can everyone here who says "distutils sucks" "setuptools sucks",
"let's rewrite them from scratch" hear that:
Just join the work in progress, help on the PEP, uses cases, and stop
whining about the situation !!!!
Tarek Ziadé | http://ziade.org
At 02:26 PM 4/21/2009 +0200, Lennart Regebro wrote:
>So why don't I use that for setuptools? Well, because:
>c) The setup of setuptools requires setuptools. So to be able to do
>the 2to3 conversion in the setup, I need to first convert the source
>with 2to3. Yes, catch 22.
What I still don't get is why it can't work in a 2-stage process,
running one setup.py with distutils to do the build_2to3, and then
running a different setup.py to do the tests.
I imagine that, if I were trying to support Python 3, what I would do
first is make a Python 2 setuptools command that ran 2to3 on a
setuptools-based Python 2 project, and generated a new source tree --
with all sdist-targeted content copied over, and all .py files
converted (including setup.py itself)... and then ran whatever extra
commands you gave, running Python 3 on the resulting setup.py, such that:
python2 setup.py 2to3 test
would automatically do the equivalent of
cd build/2to3; python3 setup.py test
after creating a converted distribution in build/2to3. That way, you
could also do things like:
python2 setup.py test 2to3 test
to run the tests in Python 2 before converting and running them in Python 3.
If somebody wants to create this command, perhaps that would be a
good idea. It can of course be implemented as a plugin, so a change
to setuptools itself is not required.
In the simplest case, the command could just derive from sdist and
build an sdist tree in build/2to3 each time, and then run 2to3 in
place. Or it could reuse sdist and unpack the sdist into the build
tree. This would be slower, but easier to code. A more advanced
version could check for changes in SOURCES.txt between the original
and the build/2to3 directory in order to find files to add/remove,
and only run 2to3 on changed .py files.
if not os.path.exists(target_sources_txt):
# wipe build tree, build sdist and unpack
target_manifest = load_manifest(target_sources_txt)
for filename in target_manifest:
if filename not in original_manifest:
# delete file
# copy or run 2to3
for filename in original_manifest:
if filename not in target_manifest:
# copy or run 2to3
...but with a lot more os.path.join operations and a lot less
handwaving. ;-) And the initial version of this could just always
do the wipe-and-unpack step, although it'd still need to loop over
the files to run 2to3 anyway, I suppose.
Anyway, I know this is a fair amount of work; It just seems to me
that it has more uses than converting setuptools; i.e., it'd be a
useful rig for anybody doing 2-to-3 porting work.
I have catch 22 problems in finishing the setuptools port to Python 3,
and unless somebody can come up with alternatives, I may rip out
setuptools dependency on itself completely, and only depend on
I have as you know ported setuptools to Python 3. I'm in the process
of trying to fix and clean up that porting so that it can be merged
into trunk, but there is one obstacle left, and that's the question of
Currently there are several ways to install setuptools:
1) Run ez_install.py
2) Download / check-out source and run python setup.py install.
3) Download the correct egg and install that (which always confuses
me, I usually fail, and end up doing 2) instead).
Setuptools also helps with running tests by
4) python setup.py test
Currently for the Python 3 version, I've made a script that copies the
source tree and runs 2to3 on it. Then you can use that new tree to run
tests, install, make eggs and releases etc. The problems with this is:
a) The script is made for my computer only. It wouldn't work on
Windows, and it requires you to have lib2to3 checked out in a
particular place (although I think that's only if you use 3.0.0, and
not 3.0.1, but I haven't tested).
b) It's slow, as it copies all files, even those who hasn't changed
when you fixed a bug. Making a script that only copies the changed
files and runs 2to3 on them is possible, but I'd like not too, it
takes time and is likely to break. It seems like a waste.
Obviously, this is exactly the things distutils are meant to solve.
And indeed, distutils in Python 3 has a command called build_2to3
which solves exactly these problems. And indeed, this is the technique
I've used for the Python 3 branch of zope.interfaces. There I just run
setup.py install, and if I'm doing that under Python 3, it will first
run 2to3 on the code before installing. Same thing with running
setup.py test. It will sync changes to build/, run 2to3 on changed
files and then run the tests in build. It makes porting to Python 3
much easier, and it makes installing from source painless.
So why don't I use that for setuptools? Well, because:
c) The setup of setuptools requires setuptools. So to be able to do
the 2to3 conversion in the setup, I need to first convert the source
with 2to3. Yes, catch 22.
I've tried to get around this problem, but failed. Solutions I tried was:
I) Fall back to distutils if setuptools can't be imported. This means
that I can with some effort make a setup that will work under Python 3
and install setuptools under Python 3. Problem solved? No. Because
running setup.py again will just mean that it tries to import
setuptools from the local Python2 location, and it will fail. So in
this case I can't run tests of build eggs for setuptool.
II) Moving the source into a src directory. This means that when
setup.py runs, it will use the version of setuptools installed under
I), and building eggs etc is possible (I think, I haven't tested that
the eggs are correct). Problem solved? No, because you still can't
test it. Which leads us to problem d:
d) When running the tests, the setuptools module will already be
imported. But it will have imported the one in site-packages, *not*
the one in src, and it will therefore test the one in site-packages.
And that one doesn't have the api_tests.txt copied in, so that will
e) even if api_tests.txt was copied, this would mean you had to
install setuptools before you test it, which is daft.
I don't have a good solution to this, unless we can drop setuptools
dependency on setuptools completely, and just use plain distutils for
installing and testing setuptools. This will mean no more setuptools
source eggs, but then I don't see the purpose of those, really.
Arguments for not doing this and alternative solutions are highly
Lennart Regebro: Python, Zope, Plone, Grok
+33 661 58 14 64
At 04:06 PM 4/21/2009 +0200, Lennart Regebro wrote:
>On Tue, Apr 21, 2009 at 15:03, P.J. Eby <pje(a)telecommunity.com> wrote:
> > python2 setup.py 2to3 test
>Well, yes, but it should be
> python 3 setup.py 2to3 test
>Otherwise it can't reasonably have any idea of which python to use.
Why not? The 2to3 command could simply take an option for the
python3 executable, and be set from the standard config files (e.g. setup.cfg).
>And when you run it with python3, the 2to3 command isn't needed, as
>the 2to3 step can be automatically inserted through version detection.
Um... I'm still not following you. You sound like you're describing
something very different from what I outlined.
>In fact, this is exactly what I'm doing for zope.interface, as
>explained. For zope.interface, you run
> python2.5 setup.py test
>and it will test it with python2.5 You run
> python3.0 setup.py test
>and it will do exactly what you say: copy the files to build, run
>2to3, and then run the test on that in-place.
> python3.0 setup.py install
>will copy the files to build, run 2to3 and then install it.
Right -- but this is totally *not* what I'm saying. Because the
strategy that I outlined doesn't require setuptools to be ported to
python 3 first.
>Yes. That's is exactly what I'm trying to achieve. And I aim to extend
>setuptools with build_2to3 and test_2to3 commands to do exactly that.
>But to do that I need a good setuptools distribution that supports
No, you don't. You only need one command, implemented as a plugin
for the Python2 version of setuptools, and you drive everything from python2.
Once you get to the point where setuptools is ported enough to be
able to have the Python3 version actually run its tests without
crashing, you can go self-hosted. But before that point, you need
Python2 as the bootstrap/host environment for the build process.
>It explains why the test cases
>could run even though setuptools still couldn't install itself. I
>thought it was badly tested, now I know it's by design.
No, it's also badly tested; that's kind of inherited from the
distutils. And by the time I added the 'sandbox' virtualization
tool, there was already tons of stuff that wasn't tested, such that
it didn't seem worth it to do something else.
At 09:11 PM 4/20/2009 +0200, Lennart Regebro wrote:
>"Isolated"? What do you mean?
Making a separate setup script for Python 3, at least for setuptools
itself, if not having a general convention for that, since other
packages may want to ship 2+3 stuff in the same package.
Or, in the alternative, using version testing in setup.py to run an
alternate script for Python 3.
>If you have suggestions on how to solve
>the problems I'm describing, can you please come forward with them,
>I've asked several times already. All you do is say "no, no, no, no,
>no", and "becasue". It isn't helpful.
I'm sorry you feel that way, as I've been *trying* to help. I just
still don't get what the problem is. If I were porting setuptools to
Python 3, I would *want* it to be circular, even if I had to hack on
it a little at first. So I have a hard time understanding why you don't.
But I'm not you -- I understand the code base and I'm not trying to
port it. Maybe if I were trying to port it, I would get what problem
you're having, or maybe I would just keep right on going and not
notice. I don't know. I've been trying to find out what exactly is
stopping you and just can't seem to wrap my brain around it, any more
than you've been able to about the reverse.
> > For end-user convenience. A large number of people installing setuptools
> > are not installing it because they are personally interested in it, but
> > because something else they want uses it.
>Yes? And again you don't explain how this leads to you conclusion.
Mostly because your questions aren't pinpointing what you want to
know. You ask general questions, so I give general answers. We seem
to both be suffering from a surplus of assumptions of what should be
"obvious" to the other person. For example, this:
>Eggs are not easier to install, on the contrary, I have tried and
>failed a couple of times, and ended up using the source install
...seems to indicate that your question was actually about eggs, not
other-than-source distributions in general. I was mostly talking
about the wininst installers and source RPM. The eggs are there, on
the other hand, for ez_setup.py to download. (Not to mention
buildout's bootstrap script, and other tools that depend on
setuptools and want to have an automated overall install process.)
I use a quite bad internet connection:
- Small bandwidth (128KBps)
- Many micro stalls
- 1s latency to the world...
I would like to first download the Plone (+deps) locally, with a dedicated
FTP client, that can handle more or less correctly the stalls.
Have you got some related documentation somewhere in your bookmarks?
Chef de projet chez Vectoris
Phone: +261 33 11 207 36
System: xUbuntu 8.10 with almost all from package install
At 06:31 PM 4/20/2009 +0200, Lennart Regebro wrote:
>Let me reformulate that:
> > Because that's the one that generates the metadata setuptools needs to run,
> > test itself, etc.
>Why do I need setuptools to do that? Why is not distutils enough?
Because distutils doesn't have an egg_info command, or SVN-based
manifest building (for making sdists).
We have a setup where a central /usr/local is copied to all our
machines. The packages installed in said central /usr/local are managed
via stow. (Basically, each package is installed in a separate directory
under /usr/local/stow, and invoking the stow command creates symlinks to
make the package appear under /usr/local.)
This works very well for a wide range of packages: autoconf packages,
CRAN packages for R, etc. This includes plain distutils python packages
(install via "python setup.py install --prefix
/usr/local/stow/some-package-name", then run stow). The only exception
is setuptools-based python packages.
Is there a way to ask setuptools to do an install that looks more like a
standard distutils install? I don't care if I lose some the advanced
setuptools features. Basically, I need an install that's done via just
copying new files. Problems like setuptoosl checking that the
destination directory is in the PYTHONPATH I can work around (although
if there's a switch to disable that check, I'd be happy to learn about
it). The main problem is the file that's edited on each install to add a
new line for each package install via setuptools. Is there a way with
setuptools of getting just a directory tree (or a tarball, etc.) that
either setuptools or myself can just copy somewhere to have an installed
Thanks in advance for any help,