[Distutils] Builders vs Installers

[Donald Stufft]

On Mar 27, 2013, at 9:03 PM, PJ Eby <pje at telecommunity.com> wrote:

> On Wed, Mar 27, 2013 at 6:55 PM, Donald Stufft <donald at stufft.io> wrote:
>> I still think I prefer a setup.py develop or develop.py to invoke the build system for development builds
> 
> It's possible we're not talking about the same thing -- I notice you
> keep mentioning "setup.py develop", which is entirely unrelated to the
> scenarios I'm talking about.  "setup.py develop" is for installing
> something *you* are working on/developing.  But depending on raw
> source doesn't imply that you would be editing or developing that
> source; it just means that you have a bleeding-edge dependency (which
> might in turn have others), adding to your management overhead if you
> have to know how to build an sdist for each of those dependencies
> whenever you need a refresh.
> 
> So, I'm not talking about scenarios where a user obtains a source
> checkout and does something with it, I'm talking about scenarios where
> the developer of a package wants to declare a *dependency* on
> *another* package that currently has to be fetched from revision
> control.  So, in order to install *their* package (e.g. to their
> staging/test server), the install system has to be able to fetch and
> build from raw sources.

I don't think you can, nor should you be able to, explicitly depend on something that is a VCS checkout.

Declared dependencies in the metadata should be "abstract", they are a name, possibly a version specifier but they are explicitly _not_ where you get that dependency from. They only become "concrete" when you resolve the abstract dependencies via an index (this index could be PyPI, it could be a directory on your machine, etc). This fits in very well with the idea of "Provides" as well, I do not depend on https://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz I depend on something that claims to be setuptools it could be https://bitbucket.org/tarek/distribute.

The point being me as the theoretical author of setuptools package author can't dictate where to install my package from.

> 
> 
>> Sorry my statement was a bit unclear, those people would all fall under actively working on that project (Zope in this case). I mean installs from VCS's should be discouraged for end users.
> 
> Define "end users".  ;-)
> 
> Here's a different example: there was a point at which I was actively
> developing PEAK-Rules and somebody else was actively developing
> something that used it.  That person wasn't developing PEAK-Rules, and
> I wasn't part of their project, but they wanted up-to-the-minute
> versions because I was making changes based on their use cases, which
> they needed right away.  Are they an "end user"?  ;-)
> 
> You could argue that, well, that's just one project, except that what
> if somebody *else* depends on *their* project, because they're also
> doing bleeding edge development?
> 
> Well, that happened, too, because the consumer of PEAK-Rules was doing
> a bleeding-edge library that *other* people were doing bleeding-edge
> development against.  So now there were two levels of dependency on
> raw sources.

I see, I was misunderstanding the use case. Like I said above though I don't think that a package author should dictate where you install X from.

I don't know easy_install or buildout very well, but when I need a bleeding-edge release of something in my dependency graph I use pip's requirements.txt file and add a ``-e <…>`` (which pip internally translates to checking the repo out and running setup.py develop). This is where, in my mind, this belongs because requirements.txt lists "concrete" dependencies (it is paired with a index url, defaulting to PyPI) and so I'm just listing another "concrete" dependency.

This does mean dependency graphs involving pre-released unpackaged dependencies are less friendly, but I think that's ok because:

* Users should opt into development releases
  - This thought is reflected in the PEP426 where it instructs installers to default to stable releases only unless the end user requests it either via flag, or by explicitly including it in the version spec.
* This is already outside of the packaging infrastructure/ecosystem. Sometimes I need system libraries installed too and I need to manually make sure they are installed. Python packaging can't solve every problem.
* I think this is an edge case (one I have hit myself) and I don't think it's a large enough use case to break the "abstract"-ness of the PKG-INFO/JSON/whatever metadata.

> 
> If you don't support these kinds of scenarios, you slow the
> community's development velocity.  Not too long ago, Richard Jones
> posted a graph on r/Python showing how package registration took off
> exponentially around the time easy_install was released.  I think that
> this is in large part due to the increased development velocity
> afforded by being able to depend on other packages at both development
> *and* deployment time.  Even though most packages don't depend on the
> bleeding edge (because they're not themselves the bleeding edge), for
> individual development it's a godsend to be able to depend on your
> *own* packages from revision control, without needing all kinds of
> manual rigamarole to use them.
> 
> (This is also really relevant for private and corporate-internal
> development scenarios.)



This was a hard email to write because I totally understand the motivation behind it, and I think it's a very attractive mis-feature. It sounds really good but I do not think it's a benefit is large enough to include it inside of "packaging" (the formats and minimum toolchain) with the negative qualities behind it. I do think this is a great value add on for an *installer* but that it should remain in the realms of installer specific (ala requirements.txt).

* I can't think of better terms than "abstract" and "concrete" and they don't perfectly describe the difference. 

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130327/492fe7ed/attachment-0001.pgp>