[Distutils] distutils2: depgraph with extras support and dynamic modification

[Sridhar Ratnakumar]

On 10/7/2010 1:31 AM, Tarek Ziadé wrote:
> On Tue, Oct 5, 2010 at 7:25 PM, Sridhar Ratnakumar
> <sridharr at activestate.com>  wrote:
>> re: http://bitbucket.org/tarek/distutils2/src/tip/distutils2/depgraph.py and http://tarekziade.wordpress.com/2010/10/03/a-quick-glimpse-at-distutils2-alpha3-part-2/
>>
>> Hi Tarek,
>>
>> I recently implemented a new dependency algorithm in PyPM. Incidentally, it is also called depgraph.py :-)
> Hehe
>
>> Our algorithm supports "extras" out of the box. It also enables dynamic readjustment of nodes/edges by incrementally adding requirements. Consider the case when you do:
>>
>>   pypm install fabric
>>
>> First, the depgraph is populated with installed packages. Then fabric's dependencies: paramiko and"pycrypto<2.1" are processed. paramiko depends on pycrypto, so pycrypto-2.3 (latest from pypi) is added. But when traversing back to fabric, we notice that pycrypto<2.1 is a requirement. So our algorithm takes care of such version requirements and adjusts things accordingly (effectively installing pycrypto-2.1).
>>
> Interesting !
>
>> I wonder if this would be of any use in distutils2. I'm sure the "extras" support would be useful to interoperate with legacy metadata. Even pip doesn't support extras right now, so this may be of incentive for users to move to distutils2 (or let pip use the new depgraph algorithm).  If this would be of benefit to distutils2, I can talk to Trent and Jeff about open sourcing our dependency algorithm implementation -- no guarantees though.
> That would be great indeed. We would love to see depgraph a basis for
> several installers. I am cc'ing Alexis who work on that part.
>

Hi Tarek/Alexis,

I've uploaded our MIT-licensed implementation of the algorithm here,
http://github.com/ActiveState/depgraph/blob/master/lib/depgraph.py

It contains the example usage (using fabric) at the bottom of the script.

Obviously, PyPM has the advantage of getting access to the index of 
available packages as a SQLite db[1], which PyPI/distutils2 doesn't -- 
so it will be tricky to integrate `get_available_distributions` into 
distutils2.

-srid

[1] We have our own internal mirror of PyPI complete with even packages 
without uploaded sdists, and metadata for all packages ... making it 
trivial to query any package (or its metadata) using a simple SQL 
statement (other than the index, all sources and metadata are stored as 
plain files). It is possible that we may open-source this 
implementation, provided there is an interest in first place.