-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (CC'ing the Python distutils list, which is where setuptools development is discussed; maybe I should be including the catalog-sig too, but I don't read that list). Philipp von Weitershausen wrote:

(Also CCing zope3-dev where the first known working set discussion started a while ago)

Tres Seaver wrote:

...

This is the "known good" problem. I'm pretty convinced that adding some kind of "PyPI subset", where gardeners for a given "package set" keep the list of packages / versions known to work well together, is the only way out of this issue. E.g., a URL like:

http://pypi.python.org/pypi/release/zope3.4

would be usable as an 'index-url' for setuptools: when used this way, setuptools would only find / install eggs from the "gardened" set, rather than whatever anyone happened to have uploaded that day.

If PyPI can't be tweaked to provide such a feature, we may need to come up with some kind of mirroring scheme which does allow it.

This is certainly an interesting approach. I'd be curious how you would garden this known working set. Martijn makes a pretty good case for maintaining such working sets close to the package in question (e.g. the grok egg, the Plone egg, etc.).

I would argue that this problem is too big for "developer convenience" to drive it: we need concerted effort from the different "communities of interest" to manage the problem, in much the same way that Debian / Fedora etc. manage their various distribution releases. I see a PyPI subset implemented as follows: - Subset creator / owner defines the list of PyPI project names which are included in the subset. - For each project, the available releases known to PyPI fall into one of the following states: o 'unknown' is the default: newly-uploaded distributions start here. o 'known_good' is a terminal state: the creator / owner of the subset has blessed this version. o 'known_bad' is also a terminal state: this version won't *ever* be compatible with the others in the subset. - Other users should be able to report "works for me" / "broken" against a given distribution *for that subset.* Perhaps we can set up an RSS feed for each subset showning the "unknown" packages, so that people know they need testing. - Note that it would be possible (assuming setuptools requirements specs are sufficiently flexible) to generate a meta-egg from the "known_good" distribution list: such an egg's 'install_requires' would need to list the "known_good" values, rather than attempting to do range arithmetic. - The subset homepage would be usable as 'index_url' for setuptools, so that folks wanting to 'easy_install' a package (or drive buildout) could restrict the available packages to the "known good" versions.

I see two more solutions:

* A versions.cfg that's loaded via HTTP. zc.buildout actually supports this now which makes it quite appealing. Also, far as I know, all major deployers of Zope3 that use zc.buildout for deployment use this form of pinning egg versions right now, which means it's actually being used out there.

Locking down the file doesn't solve the problem of knowing that there are new candidate / "unknown" distributions which need testing, nor of colleting the "works for me" / "broken" information from users. A subset could certainly generate such a view, however, which would make zc.buildout integration work on a par with the 'easy_install --index-url" usecase.

* Adding version conflict resolution to zc.buildout and/or setuptools. That way you could create meta eggs (e.g. the 'Zope' egg) with '==' version specificers (for Grok, the 'grok' egg would function as the meta egg as well). If this would cause version conflicts (and they often occur in buildout due to the lack of a full dependency tree before download), it should be possible to say which egg's versions are authoritative.

As with an apt / yum repository, a subset could harvest and export the full dependency graph information for all included distributions, assuming that setuptools was willing to use such information rather than incrementally discovering dependencies after download. I'm not sure there is much point in trying to compute such a "pickle" across the whole of PyPI, however. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFG+P/I+gerLs4ltQ4RAqnvAJ43njIxLFcs3CeBiPg59EsvIP+RhACg3HZT /+6YTbnB1V50SkLtYzuNeRc= =yVfp -----END PGP SIGNATURE-----