[Distutils] [Python-Dev] distlib updated with resources API

Daniel Holth dholth at gmail.com
Mon Oct 1 05:31:12 CEST 2012


On Sun, Sep 30, 2012 at 7:40 PM, Donald Stufft <donald.stufft at gmail.com> wrote:
> On Sunday, September 30, 2012 at 6:50 PM, David Cournapeau wrote:
>
> I am not sure I understand the argument: whatever the syntax, if the
> feature is there, you will have the same problem ? The fact that it is
> used by existing solutions tend to convince me it is not a problem
> (cabal works much better than distutils/etc... in pretty much every
> way, including installing things with dependency from their package
> servers, not to speak about RPM).
>
> I have a site that pulls data from python packages, one of my problems
> of late has been pulling requirements data out of a setuptools setup.py.
> Simply
> fetching the data for my current system is simple, however because of the
> use
> of conditionals that affect the final outcome of the metadata in order to
> make sure I get a true and complete accounting of all the metadata I would
> have to anticipate every combination of conditions used inside that file.

This is real setup.py problem and the packaging peps did a pretty good
job of making this better (but no one uses environment markers yet).
You might arrange to have it run on some number of common Pythons.

You could also parse the setup.py with the ast module to figure out
whether or not the install_requires was modified inside condition
statements.

You don't have to look at setup.py at all, you can just look for
*.egg-info/requires.txt and you will get the requirements as they
appeared on the publisher's machine. (pip always runs setup.py
egg_info again, to get trustworthy metadata, while it is resolving
deps).

> Now as far as I can tell, the bento.info conditionals have that same problem
> except that they have a much narrow scope, so they have a smaller matrix.
>
> How would I given a package that has an ``if os(windows)`` conditional, get
> a representation of that files metadata as if it were in windows? I would
> need
> to override what os(windows) is? Now if this same package also has
> conditonals
> for flags, the matrix becomes larger and larger and I need to either create
> my own custom parser that just ignores or annotates the conditionals or I
> need
> to use the official parser and run it through once per item in the matrix to
> get
> the "whole story". Am I incorrect in that?
>
> Additionally, conditionals as they are (to my understanding, having never
> used bento but having read the docs and some of it's source code)
> implemented
> in bento complicate the parser and confuse reading a file with understanding
> a file.
>
> What additional power or benefit does:
>
> Library:
>     InstallRequires:
>         docutils,
>         sphinx
>         if os(windows):
>             pywin32
>
> provide over:
>
> {
>     "Library": {
>         "InstallRequires": [
>             "docutils",
>             "sphinx",
>             "pywin32; sys.playform == win32"
>         ]
>     }
> }
>
> (Note I dislike the ; <foo> syntax too, but I dislike the conditionals
> more.)
>
> Maybe i'm missing some powerful construct in the bento.info that warrants
> throwing out a lot of hard work and testing of a lot of individuals for
> these
> common formats, however I just don't see it in merely providing a simple
> conditional statement. The ``bento.parser`` package contains ~1700 SLOC
> now without investigating that terribly deeply I'm willing to bet almost all
> of that could be thrown out in the use of a standard format.

David C. would probably be the first to admit that he is not
principally a compiler author. You are a Python programmer and you
want to use {} instead of     (indentation)? Don't answer that.

He should probably produce an AST (not "import ast") from the
bento.info with the conditionals, it sounds like it may currently
interpret while parsing, I haven't looked.

All the syntax arguments are a red herring. The other stuff
"separating build phases cleanly using an on-disk format" and
"extensible" is what makes Bento beautiful. Not having to write .ini
files is a definite plus though.

> Conditional markers do make the dependency harder to get right, as
> conditional in dependencies + provides is what makes dependency graph
> building complicated. It seems this can be solved neatly though (see
> http://www.slideshare.net/naderman/dependency-resolution-with-sat-symfony-live-2011-paris).

Well in pkg_resources the default environment for environment marker
functions (only supported in .dist-info style distributions) is:

_VARS = {'sys.platform': sys.platform,
         'python_version': '%s.%s' % sys.version_info[:2],
         # FIXME parsing sys.platform is not reliable, but there is no other
         # way to get e.g. 2.7.2+, and the PEP is defined with sys.version
         'python_full_version': sys.version.split(' ', 1)[0],
         'os.name': os.name,
         'platform.version': platform.version(),
         'platform.machine': platform.machine(),
         'platform.python_implementation': python_implementation(),
         'extra': None # wheel extension
        }

Your users could send you that dict, or you could just evaluate every
environment marker all as true and send them all the pkg-resources
files that they could possibly need.

Unfortunately no one is using environment markers.

If you would like to use environment markers for your requirements,
you can include in setup.cfg

[metadata]
requires-dist = argparse; python_version < '2.7'
    another_dep

and package your dist with "python setup.py bdist_wheel". The deps in
[metadata] will override the install_requires argument to setup.cfg.
When using the patched pip to install from the wheel file, if
markerlib and distribute >= 0.6.28 are installed, pip will resolve
dependencies based on those  markers.


More information about the Distutils-SIG mailing list