[Python-Dev] Distutils ML wrap-up: setup.cfg new format

[Tarek Ziadé]

Hello

Here's a wrapup of the Distutils-SIG discussion
we had on the "static metadata" topic.

I realize that it's a good thing to send in.
python-dev such wrapup on distutils design
decisions, to keep everyone informed and get
more feedback when required.

I will try to do it for every upcoming changes
that are not going in a PEP process (when it's not
a 'big' change). The rate of such mails should
not be very high. (around 1 mail in python-dev
for +150 mails in distutils-SIG)

If you feel that what we are about to change in distutils
is wrong, you can go ahead and help us by participating
in Distutils-ML, so we keep one and only one media
for these discussions.

The four sentences summary for people in a hurry:

    Getting metadata of a distribution that is not.
    installed means running its setup.py. This means.
    downloading the whole archive, and running.
    third party code on your system.

    To avoid it, we are adding a [setup] section in.
    setup.cfg where you can express the package metadata
    statically.

    Conditional sections, specific to some system.
    can be added to add some specific fields in [setup].

    At the end, you will be able to get metadata fields
    without running any third-party code, and possibly
    get only the distribution setup.cfg for this.

Now the long version.


Rational
========

Today, if you want to list all the metadata (PEP 314) of a.
distribution that is not installed, you need to use it's.
setup.py CLI.

So, basically, you download it, and run::

    $ python setup.py --name
    Foo

    $ python setup.py --version
    1.2

Where `name` and `version` are metadata fields. That's OK but as.
soon as the developers add more code in setup.py, this feature
might break or (worse) might do unwanted things on the target.
system.

Why should we run third-party code just to get its metadata ?

So we worked on a way to express those metadata statically,
by pushing them in `setup.cfg`. That's not hard, since all.
the metadata fields are static values.

Adding a setup section in setup.cfg
===================================

So the first thing we want to introduce is a [setup] section,
that may contain any field from the metadata::

    [setup]
    name: Foo
    version: 1.2

Distutils will look for them, and will use them. Of course
setup.py is still required, and in case an option that is.
a metadata field is passed to setup(), it will override one.
located in setup.cfg.

PEP 341 is coming up
====================

Remember the last Pycon Summit ? We said that we would
introduce a new metadata field to describe requirements..
That's basically what PEP 341 is about, and we are still.
working on it.

Basically, we will be able to write things like::

    requires: Twisted == 8.2.0

What takes us so long is that adding requirements like
this in the metadata requires more things:

- the ability to compare versions (this was described in.
  PEP 386 but not finished yet)

- the ability to make these requirements vary depending on.
  the target system

And the later makes our "setup.cfg" new [setup] section.
obsolete as soon as this new metadata field is added in.
Python.

So we need more that what I've shown in the previous section

Context-dependant sections
==========================

The second change we are going to introduce is context-dependant
sections in setup.cfg.

A context dependant section is a section with a condition that is.
used only if the execution environment meets its condition.

Here's an example::

    [setup]
    name: Foo
    version: 1.3

    [setup:sys_platform == 'win32']
    requires: pywin32
    requires: bar > 1.0

    [setup:os_machine == '64bits']
    requires: some_package

    [setup:python_version == '2.4' or python_version == '2.5']
    requires: some_package


Every [setup:condition] section will be added in [setup]
only if the condition is met.

The micro-language behind this is the simplest possible:
it compares only strings, with usual string operators,
and with the ability to combine expressions. It makes it also
easy to understand to non-pythoneers (os packagers).

The pseudo-grammar is (I don't know how to write those but you'll
get it hopefully)::

    comp: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'
    comparison: expr (comp_op expr)*
    expr: STRING
    test: or_test
    or_test: and_test ('or' and_test)*
    and_test: not_test ('and' not_test)*
    not_test: 'not' not_test | comparison

where STRING belongs to any of those:

- python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1])
- os_name = os.name
- sys_platform = sys.platform
- os_sysname = os.uname()[0].
- os_nodename = os.uname()[1]
- os_release = os.uname()[2].
- os_version = os.uname()[3]..
- os_machine = os.uname()[4].
- a free string, like '2.4', or 'win32'

Distutils will provide a function that is able to read the metadata
of a distribution, given a setup.cfg file, for the target environment::

    >>> from distutils.util import local_metadata
    >>> local_metadata('setup.cfg')
    <DistributionMetadata instance>

Meaning that a Vanilla Python will be able to read the metadata
of a package without running any code.

That's what distutils will use when setup.py is run.

That's what any package manager will be able to use to work.
with distributions.

Limitations
===========

If a distribution is unable for obscure reasons (or does not
wish) to set all metadata fields in setup.cfg, that's fine :.
the fields will be set to UNKNOWN when `local_metadata` is called.

When setup.py is run, options passed to setup() will complete
those.

When a package manager gets 'UNKOWN' values, it knows it might
need to do further processing by running setup.py.

Why this change is good
=====================

Once the requires metadata is added in PEP 341,
being able to query the metadata for a distribution can be done
without doing anything else than reading a static file and interpreting
its conditions in a restricted and secured manner.

So there's no need to run third-party code anymore.

Possible use cases:

- setup.cfg is published on PyPI. Packages managers like easy_install
  or pip will be able to list all the distributions needed and their
  versions for installing a distribution without having to download
  anything !

- OS packagers will be able to list the dependencies of a distribution
  without having to browse Python code.

And, again, this new [setup] section is backward-compatible, and will
be a good complement to the work done in PEP 376.

Thanks for reading so far. This code will be added in 2.7 and 3.2.

Tarek

-- 
Tarek Ziadé | http://ziade.org | オープンソースはすごい!