Mailman 3 April 2009 - Distutils-SIG

Re: [Distutils] The problem with Setuptools on Python 3.
by P.J. Eby April 20, 2009

April 20, 2009

At 07:13 PM 4/20/2009 +0200, Lennart Regebro wrote: >Which still doesn't really answer the question: Why setuptools need to >rely on setuptools. Because there's less duplication and chances of error that way. (Earlier versions of setuptools relied on manually-created text files, instead of using egg_info to generate them, and creating/maintaining the files was a pain.) > > Because distutils doesn't have an egg_info command > >I've already mentioned that getting rid of the setuptools dependency >on itself means you can't build setuptools eggs. But I've also pointed >out that I don't understand why they would be needed. Because the egg-info files (irrespective of whether you build an sdist, an .egg, an .exe, or even install a "flat" package) contain information like entry points and requirements. In the case of setuptools itself, the entry points are critical because setuptools finds its own commands, options, etc., almost exclusively by using entry points. This is so that it's on a level playing field with other packages wanting to extend those features, and serves as an example of how to extend itself. Look, if you want to fork setuptools to not depend on itself, knock yourself out. I sure can't stop you. It's simply that, from my POV, I'd rather not complicate the main branches with duplication to support Python 3, when ISTM that the Python-3-support bits could be isolated instead. Also, I think that setuptools self-bootstrapping is both a non-trivial integration test that shouldn't be dropped, AND a non-trivial example of how to extend setuptools (since so much of it is implemented using independent entry points rather than hardwired behavior). All that having been said, I'm not the one doing the port, so do what you think best. > For all other >packages it's recommended that only source distributions are done >unless you have c-extensions. Recommended by whom, for what purpose? >Why is setuptools an exception? For end-user convenience. A large number of people installing setuptools are not installing it because they are personally interested in it, but because something else they want uses it.

2 1

Re: [Distutils] The problem with Setuptools on Python 3.
by P.J. Eby April 20, 2009

April 20, 2009

At 06:30 PM 4/20/2009 +0200, Lennart Regebro wrote: > > Because that's the one that generates the metadata setuptools needs to run, > > test itself, etc. > >No, setup3.py does that. I thought you couldn't import setuptools in setup3.py, for all the reasons you just described? >Why do I need setup.py? To define the core entry points, basically. These are really bootstrapped from the existing entry_points.txt (which is why it's in SVN), but they need to be regenerated at egg_info/bdist_egg/install time to match the current Python version. Apart from that, I believe setuptools only uses itself for finding its own packages (including data) and manifest building (i.e. finding files that are under SVN control). So I suppose that if you dropped the use of find_packages(), you could make a setup.py that would let you run distutils commands on the code base, I just don't see how you'd get any setuptools-level commands to work properly without setuptools being imported first. i.e., you could probably manage having one setup.py, or two setup.py's whose only difference is whether they import setuptools and use find_packages; like a setup3.py that does distutils, and a regular setup.py that imports setuptools and then execfiles setup3.py. None of this makes setuptools any less reliant on itself for testing and building, though, since the core entry points have to exist in order for commands to be found, options to be verified, etc.

2 1

Re: [Distutils] nose on python 3 [was setuptools on python 3]
by jason pellerin April 20, 2009

April 20, 2009

Lennart Regebro wrote: > Currently, yes. The testsuite is run with the testrunner of > setuptools. You can run them separately too, I'm sure, but that's a > pain. There is no test.py to run. Maybe you can use nose or something > to run the tests, but then again you would need to port *nose* to > Python 3... :-/ There is a branch of nose for python 3: svn checkout http://python-nose.googlecode.com/svn/branches/py3k No stable release yet, so user beware. But I've gotten good reports from the few brave first adopters. JP

1 0

Re: [Distutils] The problem with Setuptools on Python 3.
by P.J. Eby April 20, 2009

April 20, 2009

At 06:02 PM 4/20/2009 +0200, Lennart Regebro wrote: >On Mon, Apr 20, 2009 at 17:07, P.J. Eby <pje(a)telecommunity.com> wrote: > > I still don't understand why you can't have a setup3.py that uses only > > distutils, in order to run the build2to3 on. Am I missing something? > >Well, the question then becomes why not just depends on distutils all >the way? With a setup3.py that does this, most of the work for >removing this self-dependency is done. > > > python3 setup3.py build_2to3 > > cd wherevertheoutputgoes > > python3 setup.py test (or whatever) > >Well, first of all the distutils build command doesn't include the >files needed, such as setup.py, and also it doesn't copy the txt files >that is the doctests. So it would need an extended version of >build_py_2to3 that does copy these files. And instead of doing "cd >wherever, python2 setup.py test", we can just as well just do a custom >command that runs the custom build_py_2to3 and runs the tests from >there too. > >So in fact, what you are suggesting here is almost exactly what I'm >suggesting, except that I don't see the point in confusing and >complicating things by keeping the old setup.py. Because that's the one that generates the metadata setuptools needs to run, test itself, etc.

2 2

Making commands extensible by default
by Tarek Ziadé April 20, 2009

April 20, 2009

Hello, This is a side discussion but quiet important ihmo. == Problem == Some people complained about the fact that is was hard to extend Distutils commands. You end up rewriting the whole command most of the time. So what's a command ? It's a class that is used by the distribution instance when you run Distutils. roughly: cmd = Command(distribution) cmd.initialize_options() cmd.finalize_options() <--- allows to check the options if subcommands where run cmd.run() <--- runs the code each command can define sub commands, but most of the time it's a harcoded list, so you need to inherit the command if you want to add a new behavior. == work in progress, == What we want to do here is being able to define subsets in run(), sharing the same options environment. so basically, a rough, generic run() method could be: def run(): for func in some_funcs: func(self, options) If "some_funcs" could be defined by a registery with simple names, anyone could provide new functions and configure the registery to run a sequence of function. Given a command name, Distutils can get this list of function, through a registery. Each function could register itself into Distutils, like in what I have started to work here for the manifest file: see http://wiki.python.org/moin/Distutils/ManifestPluginSystem The ordering would be configurable through the setup.cfg file. Any opinion, idea for this part ? Cheers Tarek

4 8

Re: [Distutils] The problem with Setuptools on Python 3.
by P.J. Eby April 20, 2009

April 20, 2009

At 10:19 AM 4/20/2009 +0200, Lennart Regebro wrote: >I don't have a good solution to this, unless we can drop setuptools >dependency on setuptools completely, and just use plain distutils for >installing and testing setuptools. I still don't understand why you can't have a setup3.py that uses only distutils, in order to run the build2to3 on. Am I missing something? i.e., ISTM that you should be able to do something like: python3 setup3.py build_2to3 cd wherevertheoutputgoes python3 setup.py test (or whatever)

2 1

Re: [Distutils] [Catalog-sig] Metadata-Version in PKG-INFO
by "Martin v. Löwis" April 19, 2009

April 19, 2009

> For licensing, I would say we could update the PEP to say that the > 'licence' argument could be used to clarify the classifier (e.g., to > spell the version). That works for me. Regards, Martin

3 5

PEP 376 - install/uninstall script in Distutils ?
by Tarek Ziadé April 19, 2009

April 19, 2009

Hello I am working on PEP 376 <http://svn.python.org/projects/peps/trunk/pep-0376.txt>, and there's a part about adding an install and uninstall script into Distutils. The question is, do we want to propose a reference implementation of these scripts into Distutils ? I'd be in favor of just describing in this PEP what a install script and an uninstall script should do, and not include any reference implementation in Distutils. (and leave it to pip, setuptools, etc..) It forces people to pick on third party tool, but as long as it complies with the PEP, I guess it's better to leave it outside. Any opinion ? Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

6 6

multiversioning __import__ and modules?
by Garrett Cooper April 17, 2009

April 17, 2009

Hi guys, I realize this may not look like a distutils question, but it's closely tied to distutils and setuptools. I did some exploring today and it appears that there isn't a simple way to profile different versioned packages and bootstrap one version as the master version without going through the trouble of implementing a lot of what virtualenv does. What I thought might be a good idea (and I'm just tossing it out to see what others might think), is to apply logic to __import__ that is similar to setuptools.pkg_resources.require() when importing packages. Here's the new API I was thinking of: __import__(name[, globals[, locals[, fromlist[, level][,version=None][,requires=expr]]]]])¶ expr can be anything of the form described under <http://docs.python.org/distutils/setupscript.html#relationships-between-dis…>: Here are some explicit examples using the new API: 1. PackageA v2.0 requires PackageB v1.5: __import__('PackageA', version="2.0", requires="PackageB==1.5") 2. PackageC v2.1 requires any PackageD version between v1.6 and v1.8, minus v1.7 (buggy version): __import__('PackageC', version="2.1", requires="PackageD >=1.6, !=1.7, <=1.8") 3. PackageE v0.1.0 requires any PackageF version less than v0.1.10: __import__('PackageE', version="0.1.0", requires="PackageF<0.1.10") 4. PackageG requires any version of PackageH: __import__('PackageG', requires="PackageH") Thoughts? Thanks, -Garrett

1 1

Fixing the mess in sdist/egg_info
by Tarek Ziadé April 16, 2009

April 16, 2009

Hi, I am back on that problem with the code that builds the file list. The current Distutils trunk isn't working anymore with setuptools because of a recursive loop: distutils.sdist.run() -> setuptools.build_py.data_files -> setuptools.egg_info.run() -> distutils.sdist.add_defaults() -> setuptools.build_py.data_files -> etc The mess is introduced by the fact that build_py is patched by setuptools in order to inject the files that are provided by the (D)VCS. But it also uses some APIs provided by sdist to complete the file list. In order to solve this, we need to define clearly what is the role of each command, and make sure it's a distinct role. which is not the case right now : 1/ distutils.sdist = this command is used to build a source distribution 2/ setuptools.egg_info = this command is used to build an .egg-info directory but *also* injects the files founded with (D)VCS in the MANIFEST 3/ distutils.build_py = this command is used to build pure python module but *also* to find all the .py files to include in a distribution (used by sdist). In fact, it plays a central role in sdist to get the files to include. Here's a first thaught to discuss: what about introducing a new simple command called "manifest" that could be used by sdist or any other command, and that would be responsible of collecting the files that are part of the distribution (and nothing else). This command would generate the MANIFEST file and also provide the APIs to get the files included into MANIFEST. This command would introduce and use a simple plugin system similar to the "setuptools.file_finders" entry points setuptools has for (D)VCS files collecting. But with the files list being built as an argument to the plugin. (partly described here http://wiki.python.org/moin/Distutils/ManifestPluginSystem) so Distutils, setuptools or any third party tool can register some code that add or remove file in this file list. The manifest command would provide default plugins, and setuptools could refactor part of its code to use the manifest command rather than calling sdist APIs. The goal would be to have the same result at the end, but make it simpler to extend, and avoid command interdependencies like what we have today (and that makes it hard to maitain and make evolve). For instance the MANIFEST.in templating system would be one of the default plugin provided by Distutils. The initial work would consist of refactoring the current code that gets the files, using different strategies, into plugins for the new manifest command, then make sdist uses this command as a subcommand. The second phase would consist of working at setuptools level to use the same technique. Setuptools would be able to make existing "setuptools.file_finders" entry points work with Distutils manifest command registery by providing a bridge. Now for the plugin system, I see two options so far: 1/ create a simple ad-hoc plugin system in Distutils, and declare the plugins in a new section in distutils.cfg / pydistutils.cfg for loading them (setuptools existing entry points would be collected through a unique plugin declared that would act like a bridge) 2/ use entry points (so add them into Distutils) and define a entry point name based on the command name, maybe "distutils:metadata.file_finders" so the plugin system could be used elsewhere in distutils. Opinions ? Tarek -- Tarek Ziadé | http://ziade.org

2 1