[Distutils] setup.cfg new format proposal

[Tarek Ziadé]

On Mon, Sep 14, 2009 at 4:13 PM, Glyph Lefkowitz
<glyph at twistedmatrix.com> wrote:
> I may be missing some context, but I did about an hour of googling around
> before responding here, and for the life of me I can't find a simple answer
> to the question: how do I currently programatically get the metadata out of
> a distribution?

You can't do it with Distutils to get metadata of installed distributions.

If you have the source, you can just do :

$ python setup.py --field

(where field is the metadata - PEP 314)

This need you are describing is exactly the purpose of PEP 376 in fact.


> The answer seems to be a bizarre combination of running setup.py, maybe
> reading PKG-INFO, maybe something to do with egg-info...?  Or maybe making
> some direct calls with xmlrpclib?  Or setuptools?  Setuptools must be able
> to do it internally, but I can't find an API in the documentation.
> Before trying to make the metadata static, let's try to have the metadata,
> for real, at all.  Here's an example of something I want to be able to do
> with metadata:
>
> from distutils.something import (
>     load_metadata, pypi_metadata, installed_metadata)
> meta = installed_metadata("Twisted")
> meta2 = load_metadata("/home/glyph/Downloads/Twisted-x.y")
> meta3 = pypi_metadata("Twisted")
> if meta2.version > meta.version:
>     print 'More recent version of Twisted already downloaded!'
> if meta3.version > meta2.version:
>     print 'More recent version of Twisted is available online!'
>
> Important points here:
>
> I can get the same metadata, in the same format, from PyPI, from a directory
> containing the distribution, and from my installed system.
> There are supported functions that abstract this stuff away for me.  Working
> with the packages, I don't have to know how the metadata is actually
> declared.
> There is a single data structure, hopefully with some API documentation I
> can look at.
>
> Now, maybe the data structure that I want already exists and I just don't
> know about it (is it a distutils Distribution, maybe?) but it's certainly
> hard to get a handle on one.

Basically, what you are describing is PKG-INFO, the file that contains all
metadata, and that's what PyPI sends you back through XLM RPC.

And the static setup.cfg [setup] section ressembles a lot at the final PKG-INGO
the Distribution class creates.

For installed packages, setuptools does provides a similar feature
through it's pkg_resources module, which knows how to look for the PKG-INFO file
of a given distribution in your system (several format exists, it can
be an egg-info file, or
a PKG-INFO file inside an egg info directory. PEP 376 tries
to put a standard on that)

So yes, your example would be what we would want, and if we add the
static setup.cfg thing in distutils, an API would be able to send you back the
metadata as they wil be in the PKG-INFO

> In order to be useful, the code that loads this metadata is going to need to
> execute setup.py sometimes anyway.

Not sure to understand, when installing you mean ?

> Tarek, as you said in your blog post,
> "there will always be edge cases".  So it doesn't seem too useful to be
> focusing on getting rid of that requirement right away.  And when you
> download from PyPI, you probably eventually have to execute a bunch of
> untrusted code that somebody uploaded to a wiki, as root.  Again it would be
> nice to fix that, but we have a long way to go :).
> For what it's worth, on the implementation side of things, I think that the
> library code on the client side should receive enough metadata that it can
> do the job of applying the execution environment locally.  I don't
> necessarily want to send pypi a bunch of information about my machine just
> to figure out what distutils would tell me if I tried to install something.

Right, that's a good point the execution could be on client-side,
that wouldn't matter.

I kinda like the zero download principle though, to query PyPI on distributions
that fits your target system.

> If there were such an API then setup.cfg would simply be a small
> implementation detail.  The problem with lxml will still need to be
> addressed, by promoting awareness of the fact that your setup.py may be
> executed to do things other than install or submit your package.  Before
> adding conditional support to setup.cfg I would want to have a separate
> 'metadata.py' whose responsibilities were clearer: just produce the
> metadata, don't do any pre-build steps.  Ideally that would be easy to load
> correctly from setup.py as well.

That wouldn't be different from setup.py I guess, because as long as you
let people write a Python module they can do whatever they want.

Now if we say  "metadata.py" is:
- a module that returns an instance of the Distribution class, that we
use in setup.py
- shall not have any dependency except vanilla python
- executed whith a restricted list of paths (basically, just Python's lib)

That would be better than the current situation.

But in any case, this is not opposed to what we've proposed for
setup.cfg, it could
be combined imho.

For clarity, maybe the [setup] section should be replaced by a plain PKG-INFO
file alongside setup.py and setup.cfg, that is loaded by the Distribution class
with the micro-language still present.

So we're "PKG-INFO"-ing at every stage (pypi, installed package,
source directory)


Tarek

-- 
Tarek Ziadé | http://ziade.org | オープンソースはすごい!