[Distutils] "setup.py needs to go away" (was [PEP 376] - Open questions on python-dev)

[Jim Fulton]

On Jul 10, 2009, at 7:39 AM, David Cournapeau wrote:

> Jim Fulton wrote:
>>
>> On Jul 10, 2009, at 1:21 AM, David Cournapeau wrote:
>>> I think it is important that people who come from different  
>>> communities
>>> participate to this: people who work on web development have  
>>> problems
>>> which are mostly orthogonal to mine, but I get the impression that  
>>> most
>>> people on distutils-sig fall in this category.
>>
>>
>> I don't think this perspective is reflective so much of web
>> development as of application development.  Those of us who work on
>> web applications need to provide working applications, often with  
>> hard
>> scalability and reliability requirements.  We need to have control
>> over how our applications are assembled so we can test known
>> configurations and deploy them in a repeatable way. I think this  
>> would
>> apply equally well to other sorts of applications.
>
>> From what I read on this ML and elsewhere, it seems that there is a  
>> key
> difference between 'webapp' and conventional applications deployment.
> For webapp, the developer and the one who install is often the same
> person, or at least a person with the same 'culture'. Normal
> applications are targeted at end users and that makes for a lot of
> difference.

I don't think that makes much difference wrt this discussion. See below.

> On the end user side, .pth, eggs, sys.path hacks are a never-ending
> source of troubles. For this reasons I am very skeptical about things
> which encourage to install multiple, 'system-wide', packages,  
> because it
> does not go into the right direction for the end-user IMO. Not in the
> current state of python and its import system, at least.

I couldn't agree more.  As an application developer, I want to deliver  
complete self-enclosed applications.  I think that's why many  
application developers gravitate towards systems like buildout and  
virtualenv/pip. In fact, the tendency of system Python's to be  
customized by system packagers or system administrators often make the  
unsuitable for supporting applications.  When delivering applications  
to be installed by end users, I try to include everything needed,  
generally including Python. (I haven't been in this position for quite  
some time.)

> Controlling how the application is assembled, in which configuration  
> is
> important - but it should not have consequences on other packages.  
> If an
> application needs a very precise version of a given library, when
> deployed, this library should not be visible to other packages IMHO.

Agreed.

> It
> would make a lot of issues easier: uninstall support, 'queryable'
> system, etc... would stay tractable. Because with the current  
> situation
> with eggs, sys.path hacks and co, I don't see how it is possible to  
> have
> a reliable system which support uninstall and rollback. That's  
> already a
> difficult enough problem as it is. Flexibility at this level is an
> anti-goal.

Ironically, easy_install really doesn't meet the needs of application  
developers, at least not without virtualenv for many of these same  
reasons.

Really, many (I'd guess most) web developers who gravitate to  
setuptools are drawn by the ability to express and manage  
dependencies.  They use tools like buildout or virtualenv that avoid  
the pitfalls you mention above. I suspect you have more in common with  
us than you think. :)

>> Other folks are focussed on making libraries available to a large
>> audience of developers and sometimes even end users.  For example, I
>> guess that numpy and a lot of related tools are often or usually used
>> in ad hoc one-off scripts for analysis of data.
>
> Yes, but it is also used as a basis for applications as well. One of  
> the
> thing I would really like to see is something ala cran (related to the
> statistical software R), where people could, inside a numpy/scipy
> environment, ask for a list of packages according to a list of  
> keywords,
> install them and uninstall them reliably. Without the need to be an
> admin on their machine. From this perspective, I think there are a lot
> of common goals with people who use easy_install and co. But at  
> least I
> am not satisfied with the current implementations and their lack of
> robustness.


Buildout gives you this.  I think pip with virtualenv might as well.   
(Buildout tends to fall down when packages make assumptions about the  
environment they're being installed into. Matplotlib has this  
problem.  I can't use it with a clean python. I have to use a Python  
that has numpy installed first because matplotlib sniffs for numpy.)

Jim

--
Jim Fulton
Zope Corporation