[Catalog-sig] parsing setuptools style requires.txt

Dylan Jay djay at pretaweb.com
Tue Sep 13 03:53:37 CEST 2011

On 12/09/2011, at 11:01 PM, Tarek Ziadé wrote:

> On Mon, Sep 12, 2011 at 10:43 AM, Dylan Jay <djay at pretaweb.com> wrote:
> ...
>> then why not allow a change to parsing of setuptools style  
>> dependencies?
>> or perhaps point me to the discussion that explains what I'm missing.
>> Its also been mentioned that in order to make this parsing work you  
>> need to
>> run setup.py to get the requires.txt for setuptools packages and  
>> this is a
>> security concern. However many packages already have the egg-info  
>> commend
>> run before upload so there is no need to run setup.py. For those  
>> packages
>> where there is a need I think security concerns could be overcome  
>> with the
>> use of the restrictedpython package. Anything trying to import  
>> anything but
>> the bare minimum is skipped.
> the egg-info produced by setuptools is dependent on the platform it's
> run on, so running on the target client is what you need to get the
> right values for sure.

true. but I'd guess that 99% of eggs have the same set of requirements  
regardless of platform.
And if we had someway to label these requirements on pypi as "dodgy- 
setuptools" and should be used only as a hint rather the exact set of  
requirements for your particular platform?

> with PEP 345-style metadata, you have ways to define static
> per-platform metadata.
> In fact, one of the goal is to have setup.cfg published at pypi
>> My interest in this is the idea that we could get distutils2 and/or
>> zc.buildout to be able to download regular updates of metadata  
>> including
>> dependencies, then perhaps those tools could avoid certain kinds of  
>> conflict
>> errors which are a pain to debug without that information. For  
>> instance, the
>> current design of zc.buildout means that:
>> Download Bob. Bob 1.0 requires Fred >= 2.0.
>> Download Fred 3.0
>> Download Marry. Marry 1.0 requires Fred < 2.5
>> Conflict error. Marry 1.0 requires Fred < 2.5 but we already have  
>> Fred 3.0.
>> If instead we knew in advance of this conflict we could have chosen  
>> to
>> download Fred 2.4 or at least warned the user there was a potential  
>> conflict
>> and they should pick a compatible version. In the case of  
>> preinstalled
>> packages, it could offer to downgrade Fred from 3.0 to 2.4.
> I think Pip solves this by downloading all dependencies beforehand, no
> ? and it works for all platforms.

yes it can be solved by downloading all dependencies but you'd have to  
download all versions of all dependencies potentially to avoid a  
conflict in the worst case.

> The bottom line is that spending efforts on adding on server side
> platform-specific metadata that are not guaranteed to be accurate for
> your target platform, is not worth it since you can already do
> something accurate on client side at this time.

see above.

> PEP 345 will provide an improvement in the long term since you will be
> able to read dependencies, but also the file list, without having to
> download the package.
> IOW, installers will speed up because they will be able to build
> dependency graphs on the fly -- as long as all packages are PEP 345,
> which won't happen before a while.

can't wait, but in the meantime I can't see why installers can't use  
these hints to speed things up as long as they use it with the caveat  
that requirements could potentially differ once actually installed.

> Cheers
> Tarek
> -- 
> Tarek Ziadé | http://ziade.org

More information about the Catalog-SIG mailing list