[Distutils] Buildout status

[Jim Fulton]

On Thu, May 12, 2011 at 6:56 AM, Tarek Ziadé <ziade.tarek at gmail.com> wrote:
> On Wed, May 11, 2011 at 6:44 PM, Jim Fulton <jim at zope.com> wrote:
>> On Wed, May 11, 2011 at 10:21 AM, Tarek Ziadé <ziade.tarek at gmail.com> wrote:
>>> On Wed, May 11, 2011 at 3:21 PM, Jim Fulton <jim at zope.com> wrote:
>>> ...
>>> Not sure to folllow.  You want to port Setuptools into py3k... again ?
>>>  If so, this was done already in Distribute, and you can join the
>>> project.
>>
>> I don't really want to join a project. I already have a number of
>> projects.
>
> But the work is not going to happen only into the zc.buildout tree,
> right ?

No. If I do this, I'd work with Phillip Eby.

If you tell me that distribute is being actively maintained, I'll use
distribute. What I've heard in the past is that distribute wasn't
being worked on. I'd interpreted this to mean that it wasn't being
maintained. Perhaps I misinterpreted. Again, if you assure me that it
is being maintained, I'll use it instead.

...

>>> Although, my suggestion would be to start digging into the
>>> Distutils2/packaging project instead, since that will be in the
>>> standard library, and backported in 2.x.
>>
>> When it's mature, I'll use it.
>
> Some of the code you need is usable today.

"Some" isn't good enough. There's another issue that I'll mention below.

>
> examples:
>
> - easy_install's package index can be replaced by the index tool we have.
> - browsing installed packages (the new PEP 376 and the old
> setuptools/distutils standards)
> - ...
>
> There are a few corner cases we don't deal with. For instance, we get
> things installed as non-zipped egg. I think that would be a good
> default for buildout2.
>
> But yeah, the code is not "mature" I guess, but it's driven by known
> use cases for you, not new features we'll ask people to try out. And
> it's going to be published in Python 3.3 in +1 year. Not sure what
> your roadmap looks like but if buildout2 is a rewrite, by the time its
> mature itself, 3.3 will be out imho

Buildout needs setuptools/distribute in 2 ways:

1. Buildout uses it to download and install packages

2. Buildout makes it available in the path when installing source
   distributions that import setuptools.  It's going to be a *long*
   time before this need goes away.

For a while, I was thinking of forking distribute to address #1, but
that wouldn't address #2, which is just as importasnt.

...

>>Regardless, I fully expect to use
>> Packaging when it's ready, but I'm stuck with setuptools/distribute
>> now. I so wish that fork hadn't been done.
>
> I so wished that we could have all worked together in the first
> place...

What?  I don't understand.  You make it sound as if I refused to work
with you.  If you are suggesting that I failed to contribute to
distribute or distutils2, you'll have to excuse me for choosing to
work on other projects. Buildout is just a consumer of setuptools (and
packaging in the future).

...

> But the reality is that some projects can run under Python 3 thanks to
> Distribute.

True, including buildout.

> It's now used in most major Linux distributions, and you
> saying "I will not support anything else that setuptools in buildout2"
> does not sound right to me. By taking the road you've mentioned, it
> seems to me that it's going to be more work and trouble that you
> expect because:
>
> - you are going to re-do all the py3 porting we already did, and that
> needs backward compatibility w/ distribute,  because some people use
> it for the extra python3 features we added
> - you are going to maintain several branches if you don't use 2to3
> - what's going to happen when distutils2-based packages will be
> published ? are you going to add backward compat in setuptools, so
> redo what we did in d2 in reverse ? (this is planned in Distribute, so
> doubled-work again)

What I said was that I wasn't going to support both. Again, if you
assure me that distribute is being maintained, then I'll avoid the
work of porting setuptools to Python 3 and just use distribute.  OTOH,
if you tell me I have to wait for "packaging" to be widely adopted,
well, that's just not going to cut it.

> A simple direction in my opinion would be:
>
> - make sure all stuff done on setuptools lately that were neglected on
> distribute side get merged back - so we really have 100% identical
> behavior that works under py3

I soo f%$#ing hate the fork.

>  that's *way* less work.

<shrug>

> - use when possible, pieces of distutils2 in buildout2, since we
> target backward compat for all our APIs.
>
> Because as far as I know, the Setuptools project is just doing
> bugfixing. I don't think there are yet new features in it. I consider
> that all setuptools-related technologies are now superseded by the new
> stuff we do in packaging (that greatly inherit from them)

>From talking to Phillip, I think he intends to update setuptools to
reflect new peps.  Personally, I'd be happy to see it (and distribute)
just fix bugs and remain stable.

I'm totally on board with the move to packaging, but until all of the
packages that buildout needs to install switch away from setuptools, I
have to be able to provide setuptools support.

>
> An even most simple direction, which would be for the best for all --
> but that's not going to happen I guess :
>
> - merge Distribute and Setuptools projects in the same codebase, and
> have more people maintaining it. It's a mutual benefit. Features added
> in Distribute are not controversial. (per-user easy install support,
> py3 support)

IOW, unfork. I would *love* *LOVE* to see that happen.

> - start to use in it, when possible, pieces of distutils2, so
> Distribute/Setuptools become PEP 345/PEP 376 aware

<shrug> sure. That's reasonable. First, buildout has to get under
control. That's my problem (and the problem of people graciously
helping me).


> - stop adding features so distutils2/packaging slowly takes over

Absolutely.  I couldn't agree more.  Just understand that we're going
to have to support old packages that import setuptools in their setup
scripts indefinitely.  That's one of the biggest challenges I see.
Maybe packaging can (does?) provide a shim for this.

>>> I think it boils down to: we should *all* work together in the
>>> Distutils2/packaging project for all the basic packaging features we
>>> need in third-party tools.
>>
>> I have lots of interesting projects I am working on and want to work
>> on. I have no interest in making packaging a career.
>
> We're all on the same page here. And it seems to me that you're
> planning quite a packaging career in buildout2 ;)

I'm committed to buildout because I use it and the community depends
on it.  I don't want to be responsible for the infrastructure
underneath it -- if I can avoid it.

>> I'll use
>> Packaging when it's ready. In the mean time, lots of people need
>> buildout to work with Python 3 today.
>
> So imho you should take the shortest path to this goal, but in the
> meantime without ignoring what's coming next -- because you'll want to
> use d2-based projects in buildout2 quite soon

That's a good point, but I also have to support setuptools-based
projects indefinitely.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton