[Distutils] setup.cfg new format proposal

[Glyph Lefkowitz]

On Fri, Sep 11, 2009 at 1:40 PM, Jim Fulton <jim at zope.com> wrote:

> On Fri, Sep 11, 2009 at 11:49 AM, Tarek Ziadé <ziade.tarek at gmail.com>
> wrote:
> > On Fri, Sep 11, 2009 at 5:36 PM, Jim Fulton <jim at zope.com> wrote:
> >> On Fri, Sep 11, 2009 at 11:14 AM, Tarek Ziadé <ziade.tarek at gmail.com>
> wrote:
> >>> 2009/9/11  <exarkun at twistedmatrix.com>:
> >>>> On 02:33 pm, ziade.tarek at gmail.com wrote:
> >>>> I'm a little skeptical about creating a new mini language
> (particularly one
> >>>> with branching) for setup.cfg,
> >>
> >> Me too.
>

Me three.  Even after hearing the motivating use-cases, like:

> So you can for example define a list of dependencies that changes wether
> > you are under windows or linux, etc..
>
> OK. I've never had that situation. I'm sure someone has, but it
> doesn't see common.


Twisted actually has this.  On Windows, we need pywin32 to implement many of
the features which can be implemented with just the stdlib on Posix.  But
this is the only thing I can imagine computing based on the execution
environment.

> try to run this with lxml source tarball:
> >
> > $ python setup.py --name
> >
> > I don't know what it is doing, but a whole lot more that what it is
> > suppose to do
> > just to let me know the 'name' metadata.
> >
> > So if name was in setup.cfg, it would be better.
>

One big problem I have with this discussion is that it seems to be
approaching the whole problem backwards.  Distutils projects seem to have a
history of defining new file formats rather than or before new APIs.
 (PKG-INFO, MANIFEST.in, egg-info, etc etc).

I may be missing some context, but I did about an hour of googling around
before responding here, and for the life of me I can't find a simple answer
to the question: how do I currently programatically get the metadata out of
a distribution?

The answer seems to be a bizarre combination of running setup.py, maybe
reading PKG-INFO, maybe something to do with egg-info...?  Or maybe making
some direct calls with xmlrpclib?  Or setuptools?  Setuptools must be able
to do it internally, but I can't find an API in the documentation.

Before trying to make the metadata static, let's try to have the metadata,
for real, at all.  Here's an example of something I want to be able to do
with metadata:

from distutils.something import (
    load_metadata, pypi_metadata, installed_metadata)
meta = installed_metadata("Twisted")
meta2 = load_metadata("/home/glyph/Downloads/Twisted-x.y")
meta3 = pypi_metadata("Twisted")
if meta2.version > meta.version:
    print 'More recent version of Twisted already downloaded!'
if meta3.version > meta2.version:
    print 'More recent version of Twisted is available online!'


Important points here:

   - I can get the same metadata, in the same format, from PyPI, from a
   directory containing the distribution, and from my installed system.
   - There are supported functions that abstract this stuff away for me.
    Working with the packages, I don't have to know how the metadata is
   actually declared.
   - There is a single data structure, hopefully with some API documentation
   I can look at.

Now, maybe the data structure that I want already exists and I just don't
know about it (is it a distutils Distribution, maybe?) but it's certainly
hard to get a handle on one.

In order to be useful, the code that loads this metadata is going to need to
execute setup.py sometimes anyway.  Tarek, as you said in your blog post,
"there will always be edge cases".  So it doesn't seem too useful to be
focusing on getting rid of that requirement right away.  And when you
download from PyPI, you probably eventually have to execute a bunch of
untrusted code that somebody uploaded to a wiki, as root.  Again it would be
nice to fix that, but we have a long way to go :).

For what it's worth, on the implementation side of things, I think that the
library code on the client side should receive enough metadata that it can
do the job of applying the execution environment locally.  I don't
necessarily want to send pypi a bunch of information about my machine just
to figure out what distutils would tell me if I tried to install something.

If there were such an API then setup.cfg would simply be a small
implementation detail.  The problem with lxml will still need to be
addressed, by promoting awareness of the fact that your setup.py may be
executed to do things other than install or submit your package.  Before
adding conditional support to setup.cfg I would want to have a separate
'metadata.py' whose responsibilities were clearer: just produce the
metadata, don't do any pre-build steps.  Ideally that would be easy to load
correctly from setup.py as well.

Inspecting metadata could also be made easier by having an API more like
this for setup.py:

d = Distribution(... metadata ....)
if d.installing:
    my_project.custom_prepare_for_build()
d.setup()

since you can't really tell what setup() is going to do until you've already
called it.

So finally, if there's still no standard programmatic way to get at the
distribution metadata after setup.cfg is modified to have conditional
expressions in it, what exactly is the point?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20090914/519c2650/attachment.htm>