On Tue, 04 Aug 2009 13:59:12 -0700, Tarek Ziadé wrote:

2009/8/4 Sridhar Ratnakumar :

...

What if I already have setuptools installed in the Python installation? besides the warnings, it should be OK You can also install distribute over your setuptools by running bootstraping.py $ wget http://nightly.ziade.org/bootstraping.py $ python bootstraping.py

Methinks for better transition (low switching cost) it would be better do this (which is currently done only by bootstrapping.py) in bootstrap.py itself. At least that's what the users expect zc.buildout's bootstrap.py to do (to wit: install setuptools if not already installed). -srid PS: The traceback I earlier reported about to_reload is solved by running bootstrapping.py prior to running bootstrap.py

2009/8/4 Sridhar Ratnakumar :

Upon using a virtualenv:

$ virtualenv --no-site-packages env1 $ rm -rf env1/lib/python2.6/site-packages/* # remove previously installed setuptools $ env1/bin/python bootstrap.py Downloading http://nightly.ziade.org/distribute-0.6-py2.6.egg Traceback (most recent call last):  File "bootstrap.py", line 41, in <module>    if to_reload: NameError: name 'to_reload' is not defined

This is because you haven't initialized `to_reload` *before* trying to run the code in the first try: block.

oups, I didn't upload the right version, it's now updated, please get it back and try agin

-srid

On Tue, 04 Aug 2009 13:50:26 -0700, Tarek Ziadé wrote:

...

Hi,

I am working on the last bits to be able to release 0.6

1/ Distribute and zc.buildout

I started to work on the last bootstraping problem we have : make sure zc.buildout users can use it.

I've uploaded at http://nightly.ziade.org/ a "bootstrap.py" file that is a patched version of zc.buildout's bootstrap.py. This version makes sure Distribute is installed and activated in a buildout environment. It's also in the repo, in the buildout directory.

Again, this requires a lot of real-world testing before we can ship 0.6. I've started to test various combinations on my side, but as I did for the previous bootstraping script, I wouldn't mind having some feedback of various people on this.

If you use zc.buildout, you can try it by replacing the bootstrap.py file that comes with your buildout with the one I work on, then

a/ add "distribute" to the required eggs in your buildout cfg file

b/ run or re-run the bootstrap:

$ wget http://nightly.ziade.org/bootstrap.py $ python bootstrap.py

If it worked, your buildout should build and work just fine, and the scripts located in bin/ should have their sys.path augmented with the distribute egg. Last, the setuptools egg is faked and you wil notice that it's empty. This has to be done so zc.buildout and any software out there that has a hardcoded dependency on setuptools thinks it's present.

This is reversible, by running the original bootstrap and removing the egg dependency added in the cfg file.

Please let me know if you try and get in any trouble. I was able to run a plone buildout on my side with no problems.

2/ bootstraping file names

To avoid any confusion, I've changed distribute's bootstrap file name. So we will have:

- ez_setup.py = setuptools bootstrap file - bootstraping.py = distribute bootstrap file - bootstrap.py = zc.buildout bootstrap file

If you think "bootstraping" is a silly name, or have a better proposition, please let me know.

3/ release/branches roadmap

I'll probably release 0.6 before the end of this week and I expect pushing a 0.6.1 one if we find big issues with the bootstraping we didn't see before, when the release goes live.

As soon as 0.6 is tagged, I'll create a 0.6 branch for its maintenance and we will be able to work on 0.7 on the default branch, including the Python 3 work. Other Py3k remaning branches will be removed/stripped.  I think this is a simpler approach.

The 0.7 release will get ridd of the bootstraps script and create several distributions, so a massive refactoring is expected. I don't know if this will make it harder for py3k work, Lennart, Ronald ?

Cheers Tarek

-- Tarek Ziadé | http://ziade.org

On Tue, Aug 4, 2009 at 11:11 PM, Sridhar Ratnakumar wrote:

On Tue, 04 Aug 2009 14:04:21 -0700, Tarek Ziadé wrote:

...

...

$ virtualenv --no-site-packages env1 $ rm -rf env1/lib/python2.6/site-packages/* # remove previously installed setuptools $ env1/bin/python bootstrap.py Downloading http://nightly.ziade.org/distribute-0.6-py2.6.egg Traceback (most recent call last):  File "bootstrap.py", line 41, in <module>   if to_reload: NameError: name 'to_reload' is not defined

This is because you haven't initialized `to_reload` *before* trying to run the code in the first try: block.

oups, I didn't upload the right version, it's now updated, please get it back and try agin

Ok, with the new bootstrap.py I have no need for bootstrapping.py.

bootstrap.py and the distribute egg it downloaded works fine here - all tests in pypm pass with s/setuptools/distribute.

So is Distribute going to be a 100% compatible replacement for setuptools?

yes, only for the 0.6 release, to allow people to "gently" switch

Suppose if some software ships with distribute and if the user later tries to install easy_install, what would happen? Cases like that.

if you try afterwards to install setuptools, it will tell you it's installed if you force an upgrade, or use the original zc.buildout bootstrap, you will get back to setuptools in your python *or* buildout environment notice that in any case third party apps should work fine, the only difference is that DIstribute 0.6 fixes some bugs setuptools 0.6c9 has

-srid

-- Tarek Ziadé | http://ziade.org

On Tue, Aug 4, 2009 at 11:01 PM, Jim Fulton wrote:

I'll be surprised if all that's needed to make buildout work with Distribute is a new bootstrap script.

well, after the bootstrap is over you end up with : - a setuptools distribution still installed *but* empty - a distribute distribution installed that provides the setuptools package, the pkg_resources.py, site.py and easy_install.py files which means that any code running afterwards will think setuptools is installed and will import distrbute's files at the moment I can see one case where it won't work: an __import__ done on the setuptools location but so far my Plone works (this is the most complex use case I have so far)

Note that my long term plan is to remove setuptools (or distribute) as a dependency of buildout, although I don't know when I'll have time to get to that.

until then I don't have a better plan to allow users to switch to the fork, unless zc.buildout is changed so setuptools installation it's not harcoded anymore in there

Jim

On Tue, Aug 4, 2009 at 4:50 PM, Tarek Ziadé wrote:

...

Hi,

I am working on the last bits to be able to release 0.6

1/ Distribute and zc.buildout

I started to work on the last bootstraping problem we have : make sure zc.buildout users can use it.

I've uploaded at http://nightly.ziade.org/ a "bootstrap.py" file that is a patched version of zc.buildout's bootstrap.py. This version makes sure Distribute is installed and activated in a buildout environment. It's also in the repo, in the buildout directory.

Again, this requires a lot of real-world testing before we can ship 0.6. I've started to test various combinations on my side, but as I did for the previous bootstraping script, I wouldn't mind having some feedback of various people on this.

If you use zc.buildout, you can try it by replacing the bootstrap.py file that comes with your buildout with the one I work on, then

a/ add "distribute" to the required eggs in your buildout cfg file

b/ run or re-run the bootstrap:

$ wget http://nightly.ziade.org/bootstrap.py $ python bootstrap.py

If it worked, your buildout should build and work just fine, and the scripts located in bin/ should have their sys.path augmented with the distribute egg. Last, the setuptools egg is faked and you wil notice that it's empty. This has to be done so zc.buildout and any software out there that has a hardcoded dependency on setuptools thinks it's present.

This is reversible, by running the original bootstrap and removing the egg dependency added in the cfg file.

Please let me know if you try and get in any trouble. I was able to run a plone buildout on my side with no problems.

2/ bootstraping file names

To avoid any confusion, I've changed distribute's bootstrap file name. So we will have:

- ez_setup.py = setuptools bootstrap file - bootstraping.py = distribute bootstrap file - bootstrap.py = zc.buildout bootstrap file

If you think "bootstraping" is a silly name, or have a better proposition, please let me know.

3/ release/branches roadmap

I'll probably release 0.6 before the end of this week and I expect pushing a 0.6.1 one if we find big issues with the bootstraping we didn't see before, when the release goes live.

As soon as 0.6 is tagged, I'll create a 0.6 branch for its maintenance and we will be able to work on 0.7 on the default branch, including the Python 3 work. Other Py3k remaning branches will be removed/stripped.  I think this is a simpler approach.

The 0.7 release will get ridd of the bootstraps script and create several distributions, so a massive refactoring is expected. I don't know if this will make it harder for py3k work, Lennart, Ronald ?

Cheers Tarek

-- Tarek Ziadé | http://ziade.org _______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Jim Fulton

-- Tarek Ziadé | http://ziade.org

On Tue, Aug 4, 2009 at 11:43 PM, Jim Fulton wrote:

Buildout will often reinstall setuptools, depending which version it thinks it needs. I'd expect this to defeat whatever you've done.

Distribute "installs" setuptools 0.6c9, so unless buildout tries to downgrade setuptools, it should not defeat it, Do you have a scenario in mind I can try ?

If distribute is truly a setuptools clone,

it's the very same code, besides the bug fixes we applied, like the svn 1.6 compatibility fix and stuff like that.

I suppose that the project name to use could be configurable in buildout.  This would take some work, but probably not that much.

zc.buildout.Buildout class could be changed indeed, so methods like boostrap() don't harcode the setuptools distribution name anymore, but how would you handle the install_requires field in zc.buildout ? I mean, beside all the bootstrap, zc.buildout stills depends on the 'setuptools' dist, requiring Distribute to fake it's installed, to avoid an "install battle" where the last distribution installed wins. Or, the setup.py script could try to import pkg_resources and decide which value to put in install_requires depending on what is installed, (choosing setuptools by default if nothing is installed)

 That seems cleaner and less brittle than hacking installed setuptools distributions.

Unfortunalety, even outside zc.buildout, I don't see any other solution than setting up a "fake" setuptools, to avoid changes in third party softwares, like their "install_requires" field. I see the 0.6 version of Distribute as a hack in any case, until Distribute 0.7 change the package/modules names, If anyone think about a better plan for the switch, speak up :) Tarek -- Tarek Ziadé | http://ziade.org

On Wed, Aug 5, 2009 at 2:14 PM, Jim Fulton wrote:

On Tue, Aug 4, 2009 at 5:57 PM, Tarek Ziadé wrote:

...

On Tue, Aug 4, 2009 at 11:43 PM, Jim Fulton wrote:

...

Buildout will often reinstall setuptools, depending which version it thinks it needs. I'd expect this to defeat whatever you've done.

Distribute "installs" setuptools 0.6c9, so unless buildout tries to downgrade setuptools, it should not defeat it,

buildout will downgrade if an older version of setuptools is specified in a versions section or in the setuptools-version option.

...

Do you have a scenario in mind I can try ?

Read the tests for upgrading, which include downgrading.

Then it means that the user explicitely downgraded setuptools in its buildout cfg file, which is a case that I am not handling : if people want to use distribute they'll have to fix their cfg.

...

Unfortunalety, even outside zc.buildout, I don't see any other solution than setting up a "fake" setuptools, to avoid changes in third party softwares, like their "install_requires" field.

I don't agree with this goal. Why not let 3rd-party developers add support for distribute explicitly.

Can you explain why buildout *should* use distribute at this point? What's in it for buildout users?

to benefit from all the bug fixes we did of course, http://bitbucket.org/tarek/distribute/wiki/bug_listing And I am not expecting +250 distributions and more to switch like that, being able to switch without changing anything except the bootstrap and the cfg file(s) and work on already existing distributions (plone.*, zope.*) is a major win. If we don't do this, it'll take months or more for all dependencies to get released again. We want to the fixes *now* then work on a better implementation.

...

I see the 0.6 version of Distribute as a hack in any case, until Distribute 0.7 change the package/modules names,

So Distribute 0.7 won't be backward compatible with 0.6?

...

If anyone think about a better plan for the switch, speak up :)

If you think you can do better than setuptools, I think you should come up with a different tool that has a different name.  You shouldn't try to trick applications into running your tool.

Distribute 0.6 is the will-never-be-released setuptols 0.6 I don't want people to have these two only choices: - stick with setuptools that has not changed for +9 months and wait for an hypotetical release - wait for a distribute completely rethaught 0.7 version wait for zope, plone and al to switch on it With distribute 0.6 it allows people to have an upgraded setuptools-like distribution that allows them to use the existing, release softwares. So unless someone else strongly opposes to this plan, this version will be released nevertheless. I believe the Plone community waits for it.

My own view is that we shouldn't try to fork setuptools, but should do something different and much smaller.  I think setuptools is far more ambitious than it needs to be.

That's the plan for Distrtibute 0.7 too (having several ditinct distributions)

Here's how I'd solve these problems if I had enough time: [...] ... I'd work with Phillip ...

As you might have noticed, Phillip doesn't have the time anymore to work on setuptools code base. This was why we started this fork: to unlock the community from this bottleneck situation. So I don't see how we could solve anything if we work with a project that doesn't make anymore change, and that is owned by someone who doesn't have the time and who is not willing to let us work in it. With all respect to PJE, my opinion is that this locked situation demonstrates that the community needs to move away as soon as possible from the setuptools project because it's not community-driven neither in the spirit of open source. What I mean by "open source spirit" is : It's ok (and probably a good idea) to have a dictator in a open source project, but it's not OK when the dictator is AWOL, which is the case right here. When it happens, a fork is needed, unless the dictator gives the key of the locker to other people (that have some time in turn of course) For instance, you are the zc.buildout dictator, but you have unlocked its maintenance to the community because (that's what I have understood) you don't have the time anymore to work on it, and you think it's stable and advanced enough, and because you *trust* the zope community will eventually do a good job. I am disappointed this scenario didn't happen with setuptools, because that's what I'd have expected.

- I'd (will) update buildout to not use setuptools.  It doesn;t use much of setuptools now.  It would be great if we could come up with a much smaller install library that multiple packages could share.

Jim

-- Jim Fulton

-- Tarek Ziadé | http://ziade.org

2009/8/6 Lennart Regebro :

But faking it may not be needed.

Yes it's needed. Even if some people switch to Distribute in their projects, as soon as you install a project that is still on Setuptools, it will overwrite Distribute files with no warning. You can't have both installed unless we rename all packages and "top" modules

On Thu, Aug 6, 2009 at 9:24 AM, Lennart Regebro wrote:

2009/8/6 Tarek Ziadé :

...

Yes it's needed. Even if some people switch to Distribute in their projects, as soon as you install a project that is still on Setuptools, it will overwrite Distribute files with no warning.

But wait... if you install distribute, and you then install a project that depends on setuptools, you say it will be installed.

no, it will be faked. meaning that just the PKG-INFO will be presented, thus detected by pkg_resources. But at the end the setuptools *package*, the pkg_resources.py and site.py files will be the one installed by setuptools.

But... isn't it distribute doing that installation? In that case, we could just not install setuptools? :-)

Not sure what you mean here.

-- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

-- Tarek Ziadé | http://ziade.org

2009/8/6 Tarek Ziadé :

On Thu, Aug 6, 2009 at 9:24 AM, Lennart Regebro wrote:

...

2009/8/6 Tarek Ziadé :

...

Yes it's needed. Even if some people switch to Distribute in their projects, as soon as you install a project that is still on Setuptools, it will overwrite Distribute files with no warning.

But wait... if you install distribute, and you then install a project that depends on setuptools, you say it will be installed.

no, it will be faked. meaning that just the PKG-INFO will be presented, thus detected by pkg_resources. But at the end the setuptools *package*, the pkg_resources.py and site.py files will be the one installed by <distribute>.

I'm confused. You said that if a project is still on setuptools it will overwrite distribute. Now you said it wouldn't. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

2009/8/6 Tarek Ziadé :

So how does a script like pip or easy_install knows if Setuptools is installed ?

Ah, right, pip. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

On Wed, Aug 5, 2009 at 9:46 PM, Tarek Ziadé wrote:

I'll wait for any other feedback and for Hanno feedback (they are sprinting in Bristol/UK) and I'll push a release Thirsday or Friday unless some bad feedback occurs.

I had some problems, which were all due to the fact that I had various "unofficial" setuptools installations all over the place claiming to be newer setuptools versions, like http://dist.jarn.com/public/setuptools-0.6c9-1.zip. I just removed all traces of setuptools from my various site-packages and buildout egg caches and bootstrapped everything again. After that it worked fine. I also changed a couple of references of ez_setup with pointers to bootstrapping in the code. bootstrapping was suggesting to me to upgrade to a newer version by running ez_setup, which would obviously be a bad idea ;) Hanno

Tarek Ziadé wrote:

...

...

a/ add "distribute" to the required eggs in your buildout cfg file Why is this needed? No one has needed to specify setuptools as a dependency previously, so why should they have to specify distribute now?

for the scripts created in zc.recipe.egg sections, so distribute is included in the sys.path since this recipe re-creates it.

Am I right in thinking this wouldn't be needed if zc.buildout and zc.recipe.egg both depended on distribute rather than setuptools?

...

...

the distribute egg. Last, the setuptools egg is faked and you wil notice that it's empty. This has to be done so zc.buildout and any software out there that has a hardcoded dependency on setuptools thinks it's present. I think this is a bad idea. It can't be that hard to patch buildout to use distribute instead of setuptools. Running s/setuptools/distribute should do it...

but as I explained earlier, even if buildout uses distribute, you can't prevent an egg out there to have an install_requires with setuptools, leading to the mentionned problem (the install battle)

Would it be possible for distribute to raise an error if a package relied on setuptools? I don't see why a third party egg would have install_requires contain setuptools if it's already running from buildout, but it feels better to me to be explicit and raise an error than lie by creating a fake egg. I know fake eggs are popular in the plohn community, but I'd prefer not to drag that abomination in the wider python world. To recap: I think it makes sense for distribute to complain if something explicitly asks for setuptools since, by definition, both the distribute and setuptools distributions can't be made available at the same time.

...

...

2/ bootstraping file names

To avoid any confusion, I've changed distribute's bootstrap file name. So we will have:

- ez_setup.py = setuptools bootstrap file - bootstraping.py = distribute bootstrap file - bootstrap.py = zc.buildout bootstrap file Now you've lost me... What is bootstrap[p]ing.py versus the bootstrap.py that comes with distribute?

bootstraping.py replaces ez_setup.py

Oh Tarek, you really do know how to pick names... Any chance you could switch this to distribute_setup.py or maybe d_setup.py to avoid dumb people like me constantly confusing the two? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Tarek Ziadé wrote:

On Thu, Aug 6, 2009 at 4:29 PM, Chris Withers wrote:

...

...

for the scripts created in zc.recipe.egg sections, so distribute is included in the sys.path since this recipe re-creates it. Am I right in thinking this wouldn't be needed if zc.buildout and zc.recipe.egg both depended on distribute rather than setuptools?

Yes, but I think this takes more energy and more time to convince Jim, then work in zc.buildout

It really doesn't look like a lot of work, in fact I knocked up a branch that shows just that: http://svn.zope.org/zc.buildout/branches/use_distribute/ The changeset required was actually pretty small: http://zope3.pov.lt/trac/changeset/102544 When a distribute egg gets released, I'll try running these tests and see what happens...

If we take the plone example, every single egg has it (because it's in the zopeskel skeleton)

*sigh* Plone suck strikes again :-(

I'd rather be able to run already released eggs in my buildout today rather than waiting for the crowd to adopt it.

Plus, people wont adopt it if they can't taste it,

I guess that's the way it needs to be, and all for the stubbornness of Phil Eby...

...

...

bootstraping.py replaces ez_setup.py Oh Tarek, you really do know how to pick names...

Any chance you could switch this to distribute_setup.py or maybe d_setup.py to avoid dumb people like me constantly confusing the two?

Sure. distribute_setup.py sounds good, I can change. That will be the name, no further change :)

Thanks. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Tarek Ziadé wrote:

...

http://zope3.pov.lt/trac/changeset/102544

You also need to patch zc.buildout.Buildout class, that has some harcoded calls to setuptools installation (like in the bootstrap method)

I've jsut made a bunch of changes to easy_install.py, buildout.py and egg.pyin r102547, can you see anything I missed?

...

When a distribute egg gets released, I'll try running these tests and see what happens...

You should be able to use nightly.ziade.org/ as an egg source in the test fixture

How do I do that?

In any case, like Jim proposed, being able to pick setuptools *or* distribute eg removing the hardcoded installation sounds like a good idea

...but it's a lot more work. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Tarek Ziadé wrote:

I am pretty sure Jim will not want a simple s/setuptools/distribute/ switch but if you can convince him, I am all +1

Why would he not? "distribute" is simply a relabelling of "setuptools" because PJE is too stubborn to let anyone else move the project forward. I see you can release buildout eggs, so if the community backs us, you could just release the buildout egg that did the trick ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

Tarek Ziadé wrote:

On Thu, Aug 6, 2009 at 5:31 PM, Chris Withers wrote:

...

Tarek Ziadé wrote:

...

I am pretty sure Jim will not want a simple s/setuptools/distribute/ switch but if you can convince him, I am all +1 Why would he not?

Because he said in that very same thread he'd rather see a change where buildout can pick, rather than a replacement.

It seems unfair to set an unnecessarily high bar to improvement like that. Hopefully he can be persuaded otherwise when he sees the branch... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

On Thu, Aug 6, 2009 at 5:19 PM, Tres Seaver wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Tarek Ziadé wrote:

...

...

...

the distribute egg. Last, the setuptools egg is faked and you wil notice that it's empty. This has to be done so zc.buildout and any software out there that has a hardcoded dependency on setuptools thinks it's present. I think this is a bad idea. It can't be that hard to patch buildout to use distribute instead of setuptools. Running s/setuptools/distribute should do it...

but as I explained earlier, even if buildout uses distribute, you can't prevent an egg out there to have an install_requires with setuptools, leading to the mentionned problem (the install battle)

Why would an egg depending on setuptools cause setuptools to be installed:  distribute's "PKG-INFO" will convince the system that setuptools is already present?

distribute's "PKG-INFO" doesn't mention setuptools, since its fields are: ... name: Distribute version: 0.6 ... So when pip or easy_install are asked to install "setuptools" they will both look for a PKG-INFO containing: ... name: setuptools ...

Or are you defending the use of the "faked" PKG-INFO (which seems like an absolute requirement to me)?

Yes I am defending the faked PKG-INFO which contains "setuptools" in the name field, and 0.6c9 in the version one. For practical reasons its' not a PKG-INFO I am installing when Distribute is installed. But in a bootstrap process. So distribute setup.py file looks like ths: before_install() setup(...) after_install() and the same code is used by the boostrapping module that replaces ez_setup.py (which is going to be named in something else after chris feedback)

Tres. - -- =================================================================== Tres Seaver          +1 540-429-0999          tseaver@palladion.com Palladion Software   "Excellence by Design"    http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKevR0+gerLs4ltQ4RAtqjAKC9gxXUkymnGI/apLHKHI4S8BbBVQCfdnar nYV9px/IExgIZbjcex60GuU= =ckuR -----END PGP SIGNATURE-----

-- Tarek Ziadé | http://ziade.org