[Distutils] Known working sets II [was: Eggification redux]

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

(CC'ing the Python distutils list, which is where setuptools development
is discussed;  maybe I should be including the catalog-sig too, but I
don't read that list).

Philipp von Weitershausen wrote:
> (Also CCing zope3-dev where the first known working set discussion 
> started a while ago)
> 
> Tres Seaver wrote:
>> This is the "known good" problem.  I'm pretty convinced that adding some
>> kind of "PyPI subset", where gardeners for a given "package set" keep
>> the list of packages / versions known to work well together, is the only
>> way out of this issue.  E.g., a URL like:
>>
>>   http://pypi.python.org/pypi/release/zope3.4
>>
>> would be usable as an 'index-url' for setuptools:  when used this way,
>> setuptools would only find / install eggs from the "gardened" set,
>> rather than whatever anyone happened to have uploaded that day.
>>
>> If PyPI can't be tweaked to provide such a feature, we may need to come
>> up with some kind of mirroring scheme which does allow it.
> 
> This is certainly an interesting approach. I'd be curious how you would 
> garden this known working set. Martijn makes a pretty good case for 
> maintaining such working sets close to the package in question (e.g. the 
> grok egg, the Plone egg, etc.).

I would argue that this problem is too big for "developer convenience"
to drive it:  we need concerted effort from the different "communities
of interest" to manage the problem, in much the same way that Debian /
Fedora etc. manage their various distribution releases.

I see a PyPI subset implemented as follows:

 - Subset creator / owner defines the list of PyPI project names which
   are included in the subset.

 - For each project, the available releases known to PyPI fall into one
   of the following states:

   o 'unknown' is the default:  newly-uploaded distributions start here.

   o 'known_good' is a terminal state:  the creator / owner of the
     subset has blessed this version.

   o 'known_bad' is also a terminal state:  this version won't *ever*
     be compatible with the others in the subset.

 - Other users should be able to report "works for me" / "broken"
   against a given distribution *for that subset.*  Perhaps we can
   set up an RSS feed for each subset showning the "unknown" packages,
   so that people know they need testing.

 - Note that it would be possible (assuming setuptools requirements
   specs are sufficiently flexible) to generate a meta-egg from the
   "known_good" distribution list:  such an egg's 'install_requires'
   would need to list the "known_good" values, rather than attempting
   to do range arithmetic.

 - The subset homepage would be usable as 'index_url' for setuptools,
   so that folks wanting to 'easy_install' a package (or drive buildout)
   could restrict the available packages to the "known good" versions.

> I see two more solutions:
> 
> * A versions.cfg that's loaded via HTTP. zc.buildout actually supports 
> this now which makes it quite appealing. Also, far as I know, all major 
> deployers of Zope3 that use zc.buildout for deployment use this form of 
> pinning egg versions right now, which means it's actually being used out 
> there.

Locking down the file doesn't solve the problem of knowing that there
are new candidate / "unknown" distributions which need testing, nor of
colleting the "works for me" / "broken" information from users.  A
subset could certainly generate such a view, however, which would make
zc.buildout integration work on a par with the
'easy_install --index-url" usecase.

> * Adding version conflict resolution to zc.buildout and/or setuptools. 
> That way you could create meta eggs (e.g. the 'Zope' egg) with '==' 
> version specificers (for Grok, the 'grok' egg would function as the meta 
> egg as well). If this would cause version conflicts (and they often 
> occur in buildout due to the lack of a full dependency tree before 
> download), it should be possible to say which egg's versions are 
> authoritative.

As with an apt / yum repository, a subset could harvest and export the
full dependency graph information for all included distributions,
assuming that setuptools was willing to use such information rather than
incrementally discovering dependencies after download.  I'm not sure
there is much point in trying to compute such a "pickle" across the
whole of PyPI, however.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG+P/I+gerLs4ltQ4RAqnvAJ43njIxLFcs3CeBiPg59EsvIP+RhACg3HZT
/+6YTbnB1V50SkLtYzuNeRc=
=yVfp
-----END PGP SIGNATURE-----