[Catalog-sig] parsing setuptools style requires.txt

[Tarek Ziadé]

On Tue, Sep 13, 2011 at 3:53 AM, Dylan Jay <djay at pretaweb.com> wrote:
> On 12/09/2011, at 11:01 PM, Tarek Ziadé wrote:
>
>> On Mon, Sep 12, 2011 at 10:43 AM, Dylan Jay <djay at pretaweb.com> wrote:
>> ...
>>>
>>> then why not allow a change to parsing of setuptools style dependencies?
>>> or perhaps point me to the discussion that explains what I'm missing.
>>>
>>> Its also been mentioned that in order to make this parsing work you need
>>> to
>>> run setup.py to get the requires.txt for setuptools packages and this is
>>> a
>>> security concern. However many packages already have the egg-info commend
>>> run before upload so there is no need to run setup.py. For those packages
>>> where there is a need I think security concerns could be overcome with
>>> the
>>> use of the restrictedpython package. Anything trying to import anything
>>> but
>>> the bare minimum is skipped.
>>
>> the egg-info produced by setuptools is dependent on the platform it's
>> run on, so running on the target client is what you need to get the
>> right values for sure.
>
> true. but I'd guess that 99% of eggs have the same set of requirements
> regardless of platform.

where did you get this percentage from ?

I see this pattern more often than in 1% of the cases:

if sys.platform == 'win32':
    reqs.append('pywin32')

(I have more examples)


> And if we had someway to label these requirements on pypi as
> "dodgy-setuptools" and should be used only as a hint rather the exact set of
> requirements for your particular platform?

how would you detect them ?

...
>> I think Pip solves this by downloading all dependencies beforehand, no
>> ? and it works for all platforms.
>
> yes it can be solved by downloading all dependencies but you'd have to
> download all versions of all dependencies potentially to avoid a conflict in
> the worst case.
>
>>
>> The bottom line is that spending efforts on adding on server side
>> platform-specific metadata that are not guaranteed to be accurate for
>> your target platform, is not worth it since you can already do
>> something accurate on client side at this time.
>
> see above.
>
>
>>
>> PEP 345 will provide an improvement in the long term since you will be
>> able to read dependencies, but also the file list, without having to
>> download the package.
>>
>> IOW, installers will speed up because they will be able to build
>> dependency graphs on the fly -- as long as all packages are PEP 345,
>> which won't happen before a while.
>
> can't wait, but in the meantime I can't see why installers can't use these
> hints to speed things up as long as they use it with the caveat that
> requirements could potentially differ once actually installed.

a speed up with a potential non-working installation, does not sound
like a good idea me.

now, installers could keep a cache of platform-specific metadata on
the client side to build a dependency graph that's looked up and
completed on any new installation. That would save some calls and
downloads.

Cheers
Tarek

-- 
Tarek Ziadé | http://ziade.org