At 07:13 PM 4/20/2009 +0200, Lennart Regebro wrote:
>Which still doesn't really answer the question: Why setuptools need to
>rely on setuptools.
Because there's less duplication and chances of error that
way. (Earlier versions of setuptools relied on manually-created text
files, instead of using egg_info to generate them, and
creating/maintaining the files was a pain.)
> > Because distutils doesn't have an egg_info command
>I've already mentioned that getting rid of the setuptools dependency
>on itself means you can't build setuptools eggs. But I've also pointed
>out that I don't understand why they would be needed.
Because the egg-info files (irrespective of whether you build an
sdist, an .egg, an .exe, or even install a "flat" package) contain
information like entry points and requirements.
In the case of setuptools itself, the entry points are critical
because setuptools finds its own commands, options, etc., almost
exclusively by using entry points. This is so that it's on a level
playing field with other packages wanting to extend those features,
and serves as an example of how to extend itself.
Look, if you want to fork setuptools to not depend on itself, knock
yourself out. I sure can't stop you.
It's simply that, from my POV, I'd rather not complicate the main
branches with duplication to support Python 3, when ISTM that the
Python-3-support bits could be isolated instead. Also, I think that
setuptools self-bootstrapping is both a non-trivial integration test
that shouldn't be dropped, AND a non-trivial example of how to extend
setuptools (since so much of it is implemented using independent
entry points rather than hardwired behavior).
All that having been said, I'm not the one doing the port, so do what
you think best.
> For all other
>packages it's recommended that only source distributions are done
>unless you have c-extensions.
Recommended by whom, for what purpose?
>Why is setuptools an exception?
For end-user convenience. A large number of people installing
setuptools are not installing it because they are personally
interested in it, but because something else they want uses it.
At 06:30 PM 4/20/2009 +0200, Lennart Regebro wrote:
> > Because that's the one that generates the metadata setuptools needs to run,
> > test itself, etc.
>No, setup3.py does that.
I thought you couldn't import setuptools in setup3.py, for all the
reasons you just described?
>Why do I need setup.py?
To define the core entry points, basically. These are really
bootstrapped from the existing entry_points.txt (which is why it's in
SVN), but they need to be regenerated at egg_info/bdist_egg/install
time to match the current Python version.
Apart from that, I believe setuptools only uses itself for finding
its own packages (including data) and manifest building (i.e. finding
files that are under SVN control).
So I suppose that if you dropped the use of find_packages(), you
could make a setup.py that would let you run distutils commands on
the code base, I just don't see how you'd get any setuptools-level
commands to work properly without setuptools being imported first.
i.e., you could probably manage having one setup.py, or two
setup.py's whose only difference is whether they import setuptools
and use find_packages; like a setup3.py that does distutils, and a
regular setup.py that imports setuptools and then execfiles setup3.py.
None of this makes setuptools any less reliant on itself for testing
and building, though, since the core entry points have to exist in
order for commands to be found, options to be verified, etc.
Lennart Regebro wrote:
> Currently, yes. The testsuite is run with the testrunner of
> setuptools. You can run them separately too, I'm sure, but that's a
> pain. There is no test.py to run. Maybe you can use nose or something
> to run the tests, but then again you would need to port *nose* to
> Python 3... :-/
There is a branch of nose for python 3:
svn checkout http://python-nose.googlecode.com/svn/branches/py3k
No stable release yet, so user beware. But I've gotten good reports
from the few brave first adopters.
At 06:02 PM 4/20/2009 +0200, Lennart Regebro wrote:
>On Mon, Apr 20, 2009 at 17:07, P.J. Eby <pje(a)telecommunity.com> wrote:
> > I still don't understand why you can't have a setup3.py that uses only
> > distutils, in order to run the build2to3 on. Am I missing something?
>Well, the question then becomes why not just depends on distutils all
>the way? With a setup3.py that does this, most of the work for
>removing this self-dependency is done.
> > python3 setup3.py build_2to3
> > cd wherevertheoutputgoes
> > python3 setup.py test (or whatever)
>Well, first of all the distutils build command doesn't include the
>files needed, such as setup.py, and also it doesn't copy the txt files
>that is the doctests. So it would need an extended version of
>build_py_2to3 that does copy these files. And instead of doing "cd
>wherever, python2 setup.py test", we can just as well just do a custom
>command that runs the custom build_py_2to3 and runs the tests from
>So in fact, what you are suggesting here is almost exactly what I'm
>suggesting, except that I don't see the point in confusing and
>complicating things by keeping the old setup.py.
Because that's the one that generates the metadata setuptools needs
to run, test itself, etc.
This is a side discussion but quiet important ihmo.
== Problem ==
Some people complained about the fact that is was hard to extend
You end up rewriting the whole command most of the time.
So what's a command ? It's a class that is used by the distribution
instance when you run Distutils.
cmd = Command(distribution)
cmd.finalize_options() <--- allows to check the options if
subcommands where run
cmd.run() <--- runs the code
each command can define sub commands, but most of the time it's a
harcoded list, so you need to inherit the command
if you want to add a new behavior.
== work in progress, ==
What we want to do here is being able to define subsets in run(),
sharing the same options environment.
so basically, a rough, generic run() method could be:
for func in some_funcs:
If "some_funcs" could be defined by a registery with simple names,
anyone could provide new functions
and configure the registery to run a sequence of function.
Given a command name, Distutils can get this list of function, through
Each function could register itself into Distutils, like in what I
have started to work here for the manifest file:
The ordering would be configurable through the setup.cfg file.
Any opinion, idea for this part ?
At 10:19 AM 4/20/2009 +0200, Lennart Regebro wrote:
>I don't have a good solution to this, unless we can drop setuptools
>dependency on setuptools completely, and just use plain distutils for
>installing and testing setuptools.
I still don't understand why you can't have a setup3.py that uses
only distutils, in order to run the build2to3 on. Am I missing something?
i.e., ISTM that you should be able to do something like:
python3 setup3.py build_2to3
python3 setup.py test (or whatever)
I am working on PEP 376
<http://svn.python.org/projects/peps/trunk/pep-0376.txt>, and there's
a part about adding an install and uninstall script into Distutils.
The question is, do we want to propose a reference implementation of
these scripts into Distutils ?
I'd be in favor of just describing in this PEP what a install script
and an uninstall script should do, and not include any reference
implementation in Distutils.
(and leave it to pip, setuptools, etc..)
It forces people to pick on third party tool, but as long as it
complies with the PEP, I guess it's better to leave it outside.
Any opinion ?
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
I realize this may not look like a distutils question, but it's
closely tied to distutils and setuptools.
I did some exploring today and it appears that there isn't a
simple way to profile different versioned packages and bootstrap one
version as the master version without going through the trouble of
implementing a lot of what virtualenv does.
What I thought might be a good idea (and I'm just tossing it out
to see what others might think), is to apply logic to __import__ that
is similar to setuptools.pkg_resources.require() when importing
packages. Here's the new API I was thinking of:
__import__(name[, globals[, locals[, fromlist[,
expr can be anything of the form described under
Here are some explicit examples using the new API:
1. PackageA v2.0 requires PackageB v1.5:
__import__('PackageA', version="2.0", requires="PackageB==1.5")
2. PackageC v2.1 requires any PackageD version between v1.6 and
v1.8, minus v1.7 (buggy version):
__import__('PackageC', version="2.1", requires="PackageD >=1.6,
3. PackageE v0.1.0 requires any PackageF version less than v0.1.10:
__import__('PackageE', version="0.1.0", requires="PackageF<0.1.10")
4. PackageG requires any version of PackageH:
I am back on that problem with the code that builds the file list. The
current Distutils trunk isn't working anymore with setuptools because
of a recursive loop:
distutils.sdist.run() -> setuptools.build_py.data_files ->
setuptools.egg_info.run() -> distutils.sdist.add_defaults() ->
setuptools.build_py.data_files -> etc
The mess is introduced by the fact that build_py is patched by
setuptools in order to inject the files that are provided by the
But it also uses some APIs provided by sdist to complete the file list.
In order to solve this, we need to define clearly what is the role of
each command, and make sure it's a distinct role.
which is not the case right now :
1/ distutils.sdist = this command is used to build a source distribution
2/ setuptools.egg_info = this command is used to build an .egg-info
directory but *also* injects the files founded with (D)VCS in the
3/ distutils.build_py = this command is used to build pure python
module but *also* to find all the .py files to include in a
distribution (used by sdist). In fact, it plays a central role in
sdist to get the files to include.
Here's a first thaught to discuss:
what about introducing a new simple command called "manifest" that
could be used by sdist or any other command, and that would be
responsible of collecting the files that are part of the distribution
(and nothing else). This command would generate the MANIFEST file and
also provide the APIs to get the files included into MANIFEST.
This command would introduce and use a simple plugin system similar to
the "setuptools.file_finders" entry points setuptools has for (D)VCS
files collecting. But with the files list being built as an argument
to the plugin. (partly described here
Distutils, setuptools or any third party tool can register some code
that add or remove file in this file list.
The manifest command would provide default plugins, and setuptools
could refactor part of its code to use the manifest command rather
than calling sdist APIs. The goal would be to have the same result at
the end, but make it simpler to extend, and avoid command
interdependencies like what we have today (and that makes it hard to
maitain and make evolve).
For instance the MANIFEST.in templating system would be one of the
default plugin provided by Distutils.
The initial work would consist of refactoring the current code that
gets the files, using different strategies, into plugins for the new
manifest command, then make sdist uses this command as a subcommand.
The second phase would consist of working at setuptools level to use
the same technique. Setuptools would be able to make existing
"setuptools.file_finders" entry points work with Distutils manifest
command registery by providing a bridge.
Now for the plugin system, I see two options so far:
1/ create a simple ad-hoc plugin system in Distutils, and declare the
plugins in a new section in distutils.cfg / pydistutils.cfg for
loading them (setuptools existing entry points would be collected
through a unique plugin declared that would act like a bridge)
2/ use entry points (so add them into Distutils) and define a entry
point name based on the command name, maybe
"distutils:metadata.file_finders" so the plugin system could be used
elsewhere in distutils.
Tarek Ziadé | http://ziade.org