setup.cfg new format proposal
Hello here's a detailed proposal for the setup.cfg format. I've included all remarks, and tried to make the syntax as simple as possible. I've dropped the long_description stuff and the template language stuff. (people can do it as a post-release process) A/ Config file structure: """ [setup] foo: bar conditional-sections: one, two [one] condition: test [two] condition: test """ 2/ Execution environment: python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) os_name = os.name platform = sys.platform uname = os.uname() python_version_info = sys.version_info 3/ Grammar: comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not' comparison: expr (comp_op expr)* expr: 'python_version'|'python_version_info'|'os_name'|'uname'|'platform'|STRING test: or_test or_test: and_test ('or' and_test)* and_test: not_test ('and' not_test)* not_test: 'not' not_test | comparison 4/ Process distutils will interpret [setup] and act upon, to fill the Distribution object and its metadata attribute object. It will look only for Metadata fields and ignores the others. In case an option is passed in setup.py, it overrides the one in setup.cfg. Let me know if it's good enough. I won't write a PEP for this change, and I will just push it for information at python-dev as soon as I get enough +1 :) Regards Tarek -- Tarek Ziadé | http://ziade.org | オープンソースの岩！
On 02:33 pm, ziade.tarek@gmail.com wrote:
Hello
here's a detailed proposal for the setup.cfg format. I've included all remarks, and tried to make the syntax as simple as possible. I've dropped the long_description stuff and the template language stuff. (people can do it as a post-release process)
A/ Config file structure:
""" [setup] foo: bar conditional-sections: one, two
[one] condition: test
[two] condition: test """
The idea here is that the [setup] section includes everything in section [one] if all of the "condition" clauses in [one] are satisfied, and includes everything in section [two] if all of the "condition" clauses in [two] are satisfied?
2/ Execution environment:
python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) os_name = os.name platform = sys.platform uname = os.uname() python_version_info = sys.version_info
3/ Grammar:
comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not' comparison: expr (comp_op expr)* expr: 'python_version'|'python_version_info'|'os_name'|'uname'|'platform'|STRING test: or_test or_test: and_test ('or' and_test)* and_test: not_test ('and' not_test)* not_test: 'not' not_test | comparison
What's "STRING"? What's the precedence of and, or, and not? What does this mean? os_name > uname > 7 What does "is" mean? Is this all really Python, with lots of features thrown out?
4/ Process
distutils will interpret [setup] and act upon, to fill the Distribution object and its metadata attribute object. It will look only for Metadata fields and ignores the others.
In case an option is passed in setup.py, it overrides the one in setup.cfg.
Let me know if it's good enough. I won't write a PEP for this change, and I will just push it for information at python-dev as soon as I get enough +1 :)
I'm a little skeptical about creating a new mini language (particularly one with branching) for setup.cfg, but I haven't really been paying close attention to this discussion. Still, it might be good to give a brief summary of the justification for this someplace. Maybe you were already planning to do that. Jean-Paul
2009/9/11 <exarkun@twistedmatrix.com>:
On 02:33 pm, ziade.tarek@gmail.com wrote:
Hello
here's a detailed proposal for the setup.cfg format. I've included all remarks, and tried to make the syntax as simple as possible. I've dropped the long_description stuff and the template language stuff. (people can do it as a post-release process)
A/ Config file structure:
""" [setup] foo: bar conditional-sections: one, two
[one] condition: test
[two] condition: test """
The idea here is that the [setup] section includes everything in section [one] if all of the "condition" clauses in [one] are satisfied, and includes everything in section [two] if all of the "condition" clauses in [two] are satisfied?
yes, here's a concrete example : """ [setup] name: distutils version: 1.2 conditional-sections: py26, py3 [py26] condition: python_version <= 2.6 install_requires = lxml [py3] condition: python_version >= 3 install_requires = foo """
2/ Execution environment:
python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) os_name = os.name platform = sys.platform uname = os.uname() python_version_info = sys.version_info
3/ Grammar:
comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not' comparison: expr (comp_op expr)* expr: 'python_version'|'python_version_info'|'os_name'|'uname'|'platform'|STRING test: or_test or_test: and_test ('or' and_test)* and_test: not_test ('and' not_test)* not_test: 'not' not_test | comparison
What's "STRING"? What's the precedence of and, or, and not? What does this mean?
STRING is any string value that will be compared to one of the values provided in the environment, Also, I've just realized that uname() is a tuple, so I need to flatten it in strings to restrict the grammar: - os_sysname - os_nodename - os_release - os_version - os_machine
os_name > uname > 7
What does "is" mean?
That can be dropped since we have '==' and it's all strings. so: comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'
Is this all really Python, with lots of features thrown out?
I copy-pasted part of Python grammar and restricted to what we need yes, but the idea is just to have string comparisons with ands and ors
4/ Process
distutils will interpret [setup] and act upon, to fill the Distribution object and its metadata attribute object. It will look only for Metadata fields and ignores the others.
In case an option is passed in setup.py, it overrides the one in setup.cfg.
Let me know if it's good enough. I won't write a PEP for this change, and I will just push it for information at python-dev as soon as I get enough +1 :)
I'm a little skeptical about creating a new mini language (particularly one with branching) for setup.cfg, but I haven't really been paying close attention to this discussion. Still, it might be good to give a brief summary of the justification for this someplace. Maybe you were already planning to do that.
The idea is to be able to get with setup.cfg all the metadata without running any code from the distribution, and without installing it. Which is possible for most cases. (if not possible, setup.py can still be used) the mini-language is to be able to adapt these metadata depending on the execution context. I'll add a doc somewhere. Tarek
On 11 Sep, 2009, at 17:14, Tarek Ziadé wrote:
- os_sysname - os_nodename - os_release - os_version - os_machine
Os.uname() does have a usability issue: the various values don't have a clear meaning, nor even a clear mapping to "marketing" numbers. Examples: * os_sysname is "darwin" on a macosx machine, not something that would be clear to a novice OSX developer * os_release is not necessarily the same as the version of the OS but returns the version of the kernel. Examples of operating systems where the kernel version number and OS version number aren't the same: Solaris, MacOSX, and of course Linux os_nodename shouldn't be used in setup.cfg at all, I don't see a good reason to include the hostname in check conditions in setup.cfg. There's always the full setup.py if you want complicated conditions. Ronald
2009/9/11 <exarkun@twistedmatrix.com>:
On 02:33 pm, ziade.tarek@gmail.com wrote:
Hello
here's a detailed proposal for the setup.cfg format. I've included all remarks, and tried to make the syntax as simple as possible. I've dropped the long_description stuff and the template language stuff. (people can do it as a post-release process)
A/ Config file structure:
""" [setup] foo: bar conditional-sections: one, two
[one] condition: test
[two] condition: test """
The idea here is that the [setup] section includes everything in section [one] if all of the "condition" clauses in [one] are satisfied, and includes everything in section [two] if all of the "condition" clauses in [two] are satisfied?
yes, here's a concrete example : """ [setup] name: distutils version: 1.2 conditional-sections: py26, py3 [py26] condition: python_version <= 2.6 install_requires = lxml [py3] condition: python_version >= 3 install_requires = foo """
2/ Execution environment:
python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) os_name = os.name platform = sys.platform uname = os.uname() python_version_info = sys.version_info
3/ Grammar:
comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not' comparison: expr (comp_op expr)* expr: 'python_version'|'python_version_info'|'os_name'|'uname'|'platform'|STRING test: or_test or_test: and_test ('or' and_test)* and_test: not_test ('and' not_test)* not_test: 'not' not_test | comparison
What's "STRING"? What's the precedence of and, or, and not? What does this mean?
STRING is any string value that will be compared to one of the values provided in the environment, Also, I've just realized that uname() is a tuple, so I need to flatten it in strings to restrict the grammar: - os_sysname - os_nodename - os_release - os_version - os_machine
os_name > uname > 7
What does "is" mean?
That can be dropped since we have '==' and it's all strings. so: comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'
Is this all really Python, with lots of features thrown out?
I copy-pasted part of Python grammar and restricted to what we need yes, but the idea is just to have string comparisons with ands and ors
4/ Process
distutils will interpret [setup] and act upon, to fill the Distribution object and its metadata attribute object. It will look only for Metadata fields and ignores the others.
In case an option is passed in setup.py, it overrides the one in setup.cfg.
Let me know if it's good enough. I won't write a PEP for this change, and I will just push it for information at python-dev as soon as I get enough +1 :)
I'm a little skeptical about creating a new mini language (particularly one with branching) for setup.cfg, but I haven't really been paying close attention to this discussion. Still, it might be good to give a brief summary of the justification for this someplace. Maybe you were already planning to do that.
The idea is to be able to get with setup.cfg all the metadata without running any code from the distribution, and without installing it. Which is possible for most cases. (if not possible, setup.py can still be used) the mini-language is to be able to adapt these metadata depending on the execution context. I'll add a doc somewhere. Tarek
On Fri, Sep 11, 2009 at 11:14 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
2009/9/11 <exarkun@twistedmatrix.com>:
On 02:33 pm, ziade.tarek@gmail.com wrote: I'm a little skeptical about creating a new mini language (particularly one with branching) for setup.cfg,
Me too.
but I haven't really been paying close attention to this discussion.
Me neither. :)
Still, it might be good to give a brief summary of the justification for this someplace. Maybe you were already planning to do that.
The idea is to be able to get with setup.cfg all the metadata without running any code from the distribution, and without installing it.
Which is possible for most cases. (if not possible, setup.py can still be used)
the mini-language is to be able to adapt these metadata depending on the execution context. I'll add a doc somewhere.
Is that really necessary? I mean, if you have to adapt the meta data to the execution context, maybe you should fall back to writing a setup.py. I agree that more background and motivational information would be very helpful, as would some real world examples. I know that >90% (maybe 99% :) of the packages I distribute don't adapt to their execution context (beyond what distutils does internally) or need anything but meta data. Jim -- Jim Fulton
On Fri, Sep 11, 2009 at 5:36 PM, Jim Fulton <jim@zope.com> wrote:
On Fri, Sep 11, 2009 at 11:14 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
2009/9/11 <exarkun@twistedmatrix.com>:
On 02:33 pm, ziade.tarek@gmail.com wrote: I'm a little skeptical about creating a new mini language (particularly one with branching) for setup.cfg,
Me too.
but I haven't really been paying close attention to this discussion.
Me neither. :)
Still, it might be good to give a brief summary of the justification for this someplace. Maybe you were already planning to do that.
The idea is to be able to get with setup.cfg all the metadata without running any code from the distribution, and without installing it.
Which is possible for most cases. (if not possible, setup.py can still be used)
the mini-language is to be able to adapt these metadata depending on the execution context. I'll add a doc somewhere.
Is that really necessary? I mean, if you have to adapt the meta data to the execution context, maybe you should fall back to writing a setup.py.
The main background motivation is the future adding of "install_requires" in PEP 345. So you can for example define a list of dependencies that changes wether you are under windows or linux, etc.. The second motivation is : why do I need to run some code to get the metadata fields ? do I need to install a package to gets its metadata ? In 99% of the cases that would'nt be necessary. try to run this with lxml source tarball: $ python setup.py --name I don't know what it is doing, but a whole lot more that what it is suppose to do just to let me know the 'name' metadata. So if name was in setup.cfg, it would be better.
I agree that more background and motivational information would be very helpful, as would some real world examples. I know that >90% (maybe 99% :) of the packages I distribute don't adapt to their execution context (beyond what distutils does internally) or need anything but meta data.
That's why you could push your fields in the static setup.cfg. (and probably have an 'empty' setup.py file) I will prepare a document summarizing. Maybe a PEP would be required then if it's not an obvious change. Tarek
On Fri, Sep 11, 2009 at 11:49 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On Fri, Sep 11, 2009 at 5:36 PM, Jim Fulton <jim@zope.com> wrote:
On Fri, Sep 11, 2009 at 11:14 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
2009/9/11 <exarkun@twistedmatrix.com>:
On 02:33 pm, ziade.tarek@gmail.com wrote: I'm a little skeptical about creating a new mini language (particularly one with branching) for setup.cfg,
Me too.
but I haven't really been paying close attention to this discussion.
Me neither. :)
Still, it might be good to give a brief summary of the justification for this someplace. Maybe you were already planning to do that.
The idea is to be able to get with setup.cfg all the metadata without running any code from the distribution, and without installing it.
Which is possible for most cases. (if not possible, setup.py can still be used)
the mini-language is to be able to adapt these metadata depending on the execution context. I'll add a doc somewhere.
Is that really necessary? I mean, if you have to adapt the meta data to the execution context, maybe you should fall back to writing a setup.py.
The main background motivation is the future adding of "install_requires" in PEP 345.
So you can for example define a list of dependencies that changes wether you are under windows or linux, etc..
OK. I've never had that situation. I'm sure someone has, but it doesn't see common.
The second motivation is : why do I need to run some code to get the metadata fields ?
Turning the config file into code means you'd be running code. :) Still, I suspect that 99% (or some very high percentage) of cases wouldn't need conditional logic in meta data files.
do I need to install a package to gets its metadata ?
This is an important use case. It is orthogonal to defining packages with static meta data. ...
try to run this with lxml source tarball:
$ python setup.py --name
I don't know what it is doing, but a whole lot more that what it is suppose to do just to let me know the 'name' metadata.
So if name was in setup.cfg, it would be better.
Somewhat. I think the main concern though is to get the meta data without installing. In any case, I think a sufficiently large percentage of cases can be handled without logic in the setup.cfg.
I agree that more background and motivational information would be very helpful, as would some real world examples. I know that >90% (maybe 99% :) of the packages I distribute don't adapt to their execution context (beyond what distutils does internally) or need anything but meta data.
That's why you could push your fields in the static setup.cfg.
(and probably have an 'empty' setup.py file)
Yup. Don't get me wrong. I'm in favor of being able to define packages with static meta data. I just don't consider configuration files with logic in them to be static. :) Jim -- Jim Fulton
On Fri, Sep 11, 2009 at 7:40 PM, Jim Fulton <jim@zope.com> wrote:
The second motivation is : why do I need to run some code to get the metadata fields ?
Turning the config file into code means you'd be running code. :)
Well that's very different. You will be running a function that is provided in the stdlib, in a vanilla Python, and that will be restricted to string comparisons basically. In other words, it is more likely to be a translation of setup.cfg into a "localized" setup.cfg.
do I need to install a package to gets its metadata ?
This is an important use case. It is orthogonal to defining packages with static meta data.
Sure. but both can exist. although It would require a flag in setup.cfg indicating that it is not standalone. e.g. doesn't provides all metadata without requiring to execute some code in the distribution. If the flag is not present, you know you can work with setup.cfg alone.
Somewhat. I think the main concern though is to get the meta data without installing.
Well, without running the setup.py code at the minimum, to extract the metadata if they are platform-dependant. Which can start to happen more and more the day we add the famous "requires" in the metadata. For instance, how can I say that I need simplejson only if it's Python 2.5 or above ? People will have to download the package and run something to know it, where they could just read the setup.cfg file. The simplejson dependency will appear at PyPI *only* if the person that ran the register command used python 2.5 when he did... that's a problem indeed. Security-wise, if the micro-language is restricted to string comparisons, with a restricted list of variables, people and tool will be able to query setup.cfg with no risk.
In any case, I think a sufficiently large percentage of cases can be handled without logic in the setup.cfg.
yes
I agree that more background and motivational information would be very helpful, as would some real world examples. I know that >90% (maybe 99% :) of the packages I distribute don't adapt to their execution context (beyond what distutils does internally) or need anything but meta data.
That's why you could push your fields in the static setup.cfg.
(and probably have an 'empty' setup.py file)
Yup. Don't get me wrong. I'm in favor of being able to define packages with static meta data. I just don't consider configuration files with logic in them to be static. :)
Let's call it differently then, because if we want static metadata, we will have to provide that logic for the 1% that needs it. In any case, is such a feature is a problem for the 99% left ? You could still add your metadata in the [setup] section and never use that feature.
On Fri, Sep 11, 2009 at 1:40 PM, Jim Fulton <jim@zope.com> wrote:
On Fri, Sep 11, 2009 at 11:49 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On Fri, Sep 11, 2009 at 5:36 PM, Jim Fulton <jim@zope.com> wrote:
On Fri, Sep 11, 2009 at 11:14 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
2009/9/11 <exarkun@twistedmatrix.com>:
On 02:33 pm, ziade.tarek@gmail.com wrote: I'm a little skeptical about creating a new mini language (particularly one with branching) for setup.cfg,
Me too.
Me three. Even after hearing the motivating use-cases, like:
So you can for example define a list of dependencies that changes wether
you are under windows or linux, etc..
OK. I've never had that situation. I'm sure someone has, but it doesn't see common.
Twisted actually has this. On Windows, we need pywin32 to implement many of the features which can be implemented with just the stdlib on Posix. But this is the only thing I can imagine computing based on the execution environment.
try to run this with lxml source tarball:
$ python setup.py --name
I don't know what it is doing, but a whole lot more that what it is suppose to do just to let me know the 'name' metadata.
So if name was in setup.cfg, it would be better.
One big problem I have with this discussion is that it seems to be approaching the whole problem backwards. Distutils projects seem to have a history of defining new file formats rather than or before new APIs. (PKG-INFO, MANIFEST.in, egg-info, etc etc). I may be missing some context, but I did about an hour of googling around before responding here, and for the life of me I can't find a simple answer to the question: how do I currently programatically get the metadata out of a distribution? The answer seems to be a bizarre combination of running setup.py, maybe reading PKG-INFO, maybe something to do with egg-info...? Or maybe making some direct calls with xmlrpclib? Or setuptools? Setuptools must be able to do it internally, but I can't find an API in the documentation. Before trying to make the metadata static, let's try to have the metadata, for real, at all. Here's an example of something I want to be able to do with metadata: from distutils.something import ( load_metadata, pypi_metadata, installed_metadata) meta = installed_metadata("Twisted") meta2 = load_metadata("/home/glyph/Downloads/Twisted-x.y") meta3 = pypi_metadata("Twisted") if meta2.version > meta.version: print 'More recent version of Twisted already downloaded!' if meta3.version > meta2.version: print 'More recent version of Twisted is available online!' Important points here: - I can get the same metadata, in the same format, from PyPI, from a directory containing the distribution, and from my installed system. - There are supported functions that abstract this stuff away for me. Working with the packages, I don't have to know how the metadata is actually declared. - There is a single data structure, hopefully with some API documentation I can look at. Now, maybe the data structure that I want already exists and I just don't know about it (is it a distutils Distribution, maybe?) but it's certainly hard to get a handle on one. In order to be useful, the code that loads this metadata is going to need to execute setup.py sometimes anyway. Tarek, as you said in your blog post, "there will always be edge cases". So it doesn't seem too useful to be focusing on getting rid of that requirement right away. And when you download from PyPI, you probably eventually have to execute a bunch of untrusted code that somebody uploaded to a wiki, as root. Again it would be nice to fix that, but we have a long way to go :). For what it's worth, on the implementation side of things, I think that the library code on the client side should receive enough metadata that it can do the job of applying the execution environment locally. I don't necessarily want to send pypi a bunch of information about my machine just to figure out what distutils would tell me if I tried to install something. If there were such an API then setup.cfg would simply be a small implementation detail. The problem with lxml will still need to be addressed, by promoting awareness of the fact that your setup.py may be executed to do things other than install or submit your package. Before adding conditional support to setup.cfg I would want to have a separate 'metadata.py' whose responsibilities were clearer: just produce the metadata, don't do any pre-build steps. Ideally that would be easy to load correctly from setup.py as well. Inspecting metadata could also be made easier by having an API more like this for setup.py: d = Distribution(... metadata ....) if d.installing: my_project.custom_prepare_for_build() d.setup() since you can't really tell what setup() is going to do until you've already called it. So finally, if there's still no standard programmatic way to get at the distribution metadata after setup.cfg is modified to have conditional expressions in it, what exactly is the point?
On Mon, Sep 14, 2009 at 4:13 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
I may be missing some context, but I did about an hour of googling around before responding here, and for the life of me I can't find a simple answer to the question: how do I currently programatically get the metadata out of a distribution?
You can't do it with Distutils to get metadata of installed distributions. If you have the source, you can just do : $ python setup.py --field (where field is the metadata - PEP 314) This need you are describing is exactly the purpose of PEP 376 in fact.
The answer seems to be a bizarre combination of running setup.py, maybe reading PKG-INFO, maybe something to do with egg-info...? Or maybe making some direct calls with xmlrpclib? Or setuptools? Setuptools must be able to do it internally, but I can't find an API in the documentation. Before trying to make the metadata static, let's try to have the metadata, for real, at all. Here's an example of something I want to be able to do with metadata:
from distutils.something import ( load_metadata, pypi_metadata, installed_metadata) meta = installed_metadata("Twisted") meta2 = load_metadata("/home/glyph/Downloads/Twisted-x.y") meta3 = pypi_metadata("Twisted") if meta2.version > meta.version: print 'More recent version of Twisted already downloaded!' if meta3.version > meta2.version: print 'More recent version of Twisted is available online!'
Important points here:
I can get the same metadata, in the same format, from PyPI, from a directory containing the distribution, and from my installed system. There are supported functions that abstract this stuff away for me. Working with the packages, I don't have to know how the metadata is actually declared. There is a single data structure, hopefully with some API documentation I can look at.
Now, maybe the data structure that I want already exists and I just don't know about it (is it a distutils Distribution, maybe?) but it's certainly hard to get a handle on one.
Basically, what you are describing is PKG-INFO, the file that contains all metadata, and that's what PyPI sends you back through XLM RPC. And the static setup.cfg [setup] section ressembles a lot at the final PKG-INGO the Distribution class creates. For installed packages, setuptools does provides a similar feature through it's pkg_resources module, which knows how to look for the PKG-INFO file of a given distribution in your system (several format exists, it can be an egg-info file, or a PKG-INFO file inside an egg info directory. PEP 376 tries to put a standard on that) So yes, your example would be what we would want, and if we add the static setup.cfg thing in distutils, an API would be able to send you back the metadata as they wil be in the PKG-INFO
In order to be useful, the code that loads this metadata is going to need to execute setup.py sometimes anyway.
Not sure to understand, when installing you mean ?
Tarek, as you said in your blog post, "there will always be edge cases". So it doesn't seem too useful to be focusing on getting rid of that requirement right away. And when you download from PyPI, you probably eventually have to execute a bunch of untrusted code that somebody uploaded to a wiki, as root. Again it would be nice to fix that, but we have a long way to go :). For what it's worth, on the implementation side of things, I think that the library code on the client side should receive enough metadata that it can do the job of applying the execution environment locally. I don't necessarily want to send pypi a bunch of information about my machine just to figure out what distutils would tell me if I tried to install something.
Right, that's a good point the execution could be on client-side, that wouldn't matter. I kinda like the zero download principle though, to query PyPI on distributions that fits your target system.
If there were such an API then setup.cfg would simply be a small implementation detail. The problem with lxml will still need to be addressed, by promoting awareness of the fact that your setup.py may be executed to do things other than install or submit your package. Before adding conditional support to setup.cfg I would want to have a separate 'metadata.py' whose responsibilities were clearer: just produce the metadata, don't do any pre-build steps. Ideally that would be easy to load correctly from setup.py as well.
That wouldn't be different from setup.py I guess, because as long as you let people write a Python module they can do whatever they want. Now if we say "metadata.py" is: - a module that returns an instance of the Distribution class, that we use in setup.py - shall not have any dependency except vanilla python - executed whith a restricted list of paths (basically, just Python's lib) That would be better than the current situation. But in any case, this is not opposed to what we've proposed for setup.cfg, it could be combined imho. For clarity, maybe the [setup] section should be replaced by a plain PKG-INFO file alongside setup.py and setup.cfg, that is loaded by the Distribution class with the micro-language still present. So we're "PKG-INFO"-ing at every stage (pypi, installed package, source directory) Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
On Mon, 14 Sep 2009 10:13:55 -0400, Glyph Lefkowitz wrote:
I'm a little skeptical about creating a new mini language (particularly one with branching) for setup.cfg,
Me too.
Me three. Even after hearing the motivating use-cases, like:
So you can for example define a list of dependencies that changes wether
you are under windows or linux, etc..
Me four.
OK. I've never had that situation. I'm sure someone has, but it doesn't see common.
If you follow answers and questions posted to distutils list you notice after a while that windows users get less answers to questions on distutils relatated questions. Linux a little more. I have come to conclude that "edge-cases" just means windows or to a less degree linux users. It's a distutils sort of keyword for people that don't get acknowledged as even existing. Is it distutils platform bias? I wonder..
Twisted actually has this. On Windows, we need pywin32 to implement many of the features which can be implemented with just the stdlib on Posix. But this is the only thing I can imagine computing based on the execution environment.
It's true for many windows packages.
One big problem I have with this discussion is that it seems to be approaching the whole problem backwards. Distutils projects seem to have a history of defining new file formats rather than or before new APIs. (PKG-INFO, MANIFEST.in, egg-info, etc etc).
And also a history of being highly selective about who can be blessed to work on code. I think windows programmers have substantially reduced chances of making contributions.
I may be missing some context, but I did about an hour of googling around before responding here, and for the life of me I can't find a simple answer to the question: how do I currently programatically get the metadata out of a distribution?
That's easy.... Here's a workhorse file you can get from the python package manager.. http://pythonpkgmgr.svn.sourceforge.net/viewvc/pythonpkgmgr/Packaging.py?revision=103&view=markup At the command line: C:\Documents and Settings\DL\Desktop\pythonpkgmgr>python packaging.py --metadata pygame Package Manager Test - using configuration file C:\Documents and Settings\DL\Desktop\pythonpkgmgr\py thonpkgmgr.ini - Changing to python 2.6 in C:\Python26\ - checking package Metadata is as follows: ['Metadata-Version: 1.0\n', 'Name: pywin32\n', 'Version: 214\n', 'Summary: Pytho n for Window Extensions\n', 'Home-page: http://sourceforge.net/projects/pywin32/ \n', 'Author: Mark Hammond (et al)\n', 'Author-email: mhammond@users.sourceforge .net\n', 'License: PSF\n', 'Description: Python extensions for Microsoft Windows \n', ' Provides access to much of the Win32 API, the\n', ' ability to create and use COM objects, and the\n', ' Pythonwin environment.\n', 'Platform: UNKNOWN\n'] In code: pm = PackageManager() info = pm.package_information("pygame") print " - checking package %s" % package if "PKG-INFO" in info.keys(): print "Metadata is as follows:" print info["PKG-INFO"] else: print "No Metadata found"
The answer seems to be a bizarre combination of running setup.py, maybe reading PKG-INFO, maybe something to do with egg-info...? Or maybe making some direct calls with xmlrpclib? Or setuptools? Setuptools must be able to do it internally, but I can't find an API in the documentation.
Before trying to make the metadata static, let's try to have the
You're right. Distutils can't do it at the moment. metadata,
for real, at all. Here's an example of something I want to be able to do with metadata:
from distutils.something import ( load_metadata, pypi_metadata, installed_metadata) meta = installed_metadata("Twisted") meta2 = load_metadata("/home/glyph/Downloads/Twisted-x.y") meta3 = pypi_metadata("Twisted") if meta2.version > meta.version: print 'More recent version of Twisted already downloaded!' if meta3.version > meta2.version: print 'More recent version of Twisted is available online!'
+1
So finally, if there's still no standard programmatic way to get at the distribution metadata after setup.cfg is modified to have conditional expressions in it, what exactly is the point?
Originally, we could only have conditional 'code' in programs. But now there's a push to move it out of python code and create a new mini language (that isn't python) stored within configuration files. The end result will be two languages (python and the new language) sitting in two places (setup.py and setup.cfg) able to do exactly the same thing. I think that will get confusing.. especially for us windows users... We've never been allowed to have code in configuration files.. So I am watching to see how this one pans out... David
On Mon, 14 Sep 2009 07:13:55 -0700, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
So you can for example define a list of dependencies that changes wether
you are under windows or linux, etc..
OK. I've never had that situation. I'm sure someone has, but it doesn't see common.
Twisted actually has this. On Windows, we need pywin32 to implement many of the features which can be implemented with just the stdlib on Posix. But this is the only thing I can imagine computing based on the execution environment.
I believe virtualenv requires pywin32. Maybe several other Python packages too.
I may be missing some context, but I did about an hour of googling around before responding here, and for the life of me I can't find a simple answer to the question: how do I currently programatically get the metadata out of a distribution?
http://pypi.python.org/pypi/pkginfo/
[...] Before adding conditional support to setup.cfg I would want to have a separate 'metadata.py' whose responsibilities were clearer: just produce the metadata, don't do any pre-build steps. Ideally that would be easy to load correctly from setup.py as well.
I double users would follow this proposed convention. Once in a while, I stumble upon Python packages that do not even succeed at the "python setup.py" stage. Even if they did, what prevents them from making metadata.py a little more sophisticated so as to introduce inadvertent bugs on other platforms? (Just today, argparse's new release broke on linux and mac, but not on Windows .. due to regex parsing behavior) I vote for static metadata anyday.
Inspecting metadata could also be made easier by having an API more like this for setup.py:
d = Distribution(... metadata ....) if d.installing: my_project.custom_prepare_for_build() d.setup()
since you can't really tell what setup() is going to do until you've already called it.
I doubt this would happen in practice even if widely suggested by distutils/setuptools.
So finally, if there's still no standard programmatic way to get at the distribution metadata after setup.cfg is modified to have conditional expressions in it, what exactly is the point?
... Conditional metadata for Python versions does not have to be very complicated in my opinion. Simply have the following in setup.cfg, and thus in PKG-INFO: install_requires = ['lxml', 'multiprocessing[pyver<(2,6)]', 'argparse', 'pywin32[platform.name=windows]']
metadata.get_install_requires(for_pyver=(2,6), platform=Platform(name='windows', arch='x86')) ['lxml', 'argparse', 'pywin32']
I don't imagine fields other than `install_requires` to vary based on the target environment. Simple, no? -srid
At 01:46 PM 9/15/2009 -0700, Sridhar Ratnakumar wrote:
Conditional metadata for Python versions does not have to be very complicated in my opinion. Simply have the following in setup.cfg, and thus in PKG-INFO:
install_requires = ['lxml', 'multiprocessing[pyver<(2,6)]', 'argparse', 'pywin32[platform.name=windows]']
If you spelled that [python-lt-26] and [platform-windows], it'd even be syntax compatible with today's setuptools; there'd just need to be a way to specify default "extra" tags to be always installed. (Technically, '.', and '_' characters would be usable as well as '-'.)
I don't imagine fields other than `install_requires` to vary based on the target environment.
I have a package or two that change py_modules and the like based on environment, but I could probably refactor them to use separate installation.
Sridhar Ratnakumar wrote:
I may be missing some context, but I did about an hour of googling around before responding here, and for the life of me I can't find a simple answer to the question: how do I currently programatically get the metadata out of a distribution?
Yeah, but you have to remember to sacrifice the right goats to end up with the right things in whatever files pkginfo uses :-( Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote:
Sridhar Ratnakumar wrote:
I may be missing some context, but I did about an hour of googling around before responding here, and for the life of me I can't find a simple answer to the question: how do I currently programatically get the metadata out of a distribution? http://pypi.python.org/pypi/pkginfo/
Yeah, but you have to remember to sacrifice the right goats to end up with the right things in whatever files pkginfo uses :-(
You bothering the goats again, Chris? ;) pkginfo uses the standard information generated by the 'sdist', 'develop' and 'bdist_egg' commenads, and can even introspect installed packages on Python 2.6.x. It is documented here: http://packages.python.org/pkginfo/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKs+b9+gerLs4ltQ4RApBrAKCvmK97Xd1rdrrDVgPkiDWhHA1bzACfTlHh I9kbSb19HRCfXyaQJ4RaSSE= =HNfh -----END PGP SIGNATURE-----
Tres Seaver wrote:
You bothering the goats again, Chris? ;)
pkginfo uses the standard information generated by the 'sdist', 'develop' and 'bdist_egg' commenads, and can even introspect installed packages on Python 2.6.x. It is documented here:
Yep, I know all this, I experienced some weirdness when working on dev packages with the right metadata not being used when I was generating sphinx docs using pkginfo to get version information. I'll post again when/if I can reproduce this easily... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Fri, Sep 11, 2009 at 5:49 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
I agree that more background and motivational information would be very helpful, as would some real world examples. I know that >90% (maybe 99% :) of the packages I distribute don't adapt to their execution context (beyond what distutils does internally) or need anything but meta data.
That's a first draft: http://tarekziade.wordpress.com/2009/09/12/static-metadata-for-distutils/ I blogged it to get some more feedback.
On Sat, Sep 12, 2009 at 02:52:17AM +0200, Tarek Ziadé wrote:
On Fri, Sep 11, 2009 at 5:49 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
I agree that more background and motivational information would be very helpful, as would some real world examples. I know that >90% (maybe 99% :) of the packages I distribute don't adapt to their execution context (beyond what distutils does internally) or need anything but meta data.
That's a first draft:
http://tarekziade.wordpress.com/2009/09/12/static-metadata-for-distutils/
Your example shows an empty setup() call but does not say which package/module is supposed to be installed. I'm confused now to how much you would like in setup.cfg. Is the idea to leave the package/module/extension module/scripts etc in setup.py? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Sat, Sep 12, 2009 at 4:42 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
On Sat, Sep 12, 2009 at 02:52:17AM +0200, Tarek Ziadé wrote:
On Fri, Sep 11, 2009 at 5:49 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
I agree that more background and motivational information would be very helpful, as would some real world examples. I know that >90% (maybe 99% :) of the packages I distribute don't adapt to their execution context (beyond what distutils does internally) or need anything but meta data.
That's a first draft:
http://tarekziade.wordpress.com/2009/09/12/static-metadata-for-distutils/
Your example shows an empty setup() call but does not say which package/module is supposed to be installed.
I'll complete it, right.
I'm confused now to how much you would like in setup.cfg. Is the idea to leave the package/module/extension module/scripts etc in setup.py?
The target is to push all fields that belong to the Metadata (see PEP 314) in the static file. So anything else required by a command would stay in setup.py Regards Tarek
On Fri, Sep 11, 2009 at 05:14:58PM +0200, Tarek Ziadé wrote: [...]
That can be dropped since we have '==' and it's all strings.
'2.6' < '2.10'? Although not close you must be wary of this. Bring on PEP 386 ;-) Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Fri, Sep 11, 2009 at 6:32 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
On Fri, Sep 11, 2009 at 05:14:58PM +0200, Tarek Ziadé wrote: [...]
That can be dropped since we have '==' and it's all strings.
'2.6' < '2.10'?
Although not close you must be wary of this. Bring on PEP 386 ;-)
Right. Maybe we can drop < >, >= and >=, completely. In real world use cases, we can use "or" to make a section work for 2.6, 3.1, etc.. for python versions. I can't think of a use case where "in" and "==" is not enough.
On Fri, Sep 11, 2009 at 07:02:53PM +0200, Tarek Ziadé wrote:
On Fri, Sep 11, 2009 at 6:32 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
On Fri, Sep 11, 2009 at 05:14:58PM +0200, Tarek Ziadé wrote: [...]
That can be dropped since we have '==' and it's all strings.
'2.6' < '2.10'?
Although not close you must be wary of this. Bring on PEP 386 ;-)
Right. Maybe we can drop < >, >= and >=, completely.
In real world use cases, we can use "or" to make a section work for 2.6, 3.1, etc.. for python versions.
I can't think of a use case where "in" and "==" is not enough.
Version comparison would be nice though. Taking your example from your blog post: if I where to try to install that on Python 2.4 I'd have to edit the condition of the [py25] section to "in ('2.4', '2.5')". Maybe this is what you want though, in case you explicitly don't support for 2.4. OTOH it's nice if things would just work in case of developers simply not having tried earlier versions. This also shows how "in" implies a tuple or a list as data type, your mini-language was trying to avoid that. Or did I miss something? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Sat, Sep 12, 2009 at 4:50 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
I can't think of a use case where "in" and "==" is not enough.
Version comparison would be nice though. Taking your example from your blog post: if I where to try to install that on Python 2.4 I'd have to edit the condition of the [py25] section to "in ('2.4', '2.5')".
or simply do: """ condition: python_version == '2.4' or python_version == '2.5' """ It's more verbose, but I don't think it's too bad
Maybe this is what you want though, in case you explicitly don't support for 2.4. OTOH it's nice if things would just work in case of developers simply not having tried earlier versions.
This also shows how "in" implies a tuple or a list as data type, your mini-language was trying to avoid that. Or did I miss something?
I was thinking about the "in" operator exclusively restricted to strings. for example: """ condition: 'i386' in os_machine """ In real python, that would be: 'i386' in os.uname()[-1]
Floris Bruynooghe wrote:
On Fri, Sep 11, 2009 at 07:02:53PM +0200, Tarek Ziadé wrote:
On Fri, Sep 11, 2009 at 6:32 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
On Fri, Sep 11, 2009 at 05:14:58PM +0200, Tarek Ziadé wrote: [...]
That can be dropped since we have '==' and it's all strings. '2.6' < '2.10'?
Although not close you must be wary of this. Bring on PEP 386 ;-) Right. Maybe we can drop < >, >= and >=, completely.
In real world use cases, we can use "or" to make a section work for 2.6, 3.1, etc.. for python versions.
I can't think of a use case where "in" and "==" is not enough.
Version comparison would be nice though. Taking your example from your blog post: if I where to try to install that on Python 2.4 I'd have to edit the condition of the [py25] section to "in ('2.4', '2.5')". Maybe this is what you want though, in case you explicitly don't support for 2.4. OTOH it's nice if things would just work in case of developers simply not having tried earlier versions.
This also shows how "in" implies a tuple or a list as data type, your mini-language was trying to avoid that. Or did I miss something?
A lot of these problems would go away if we could just treat "Python" as a package like every other package when it comes to dependencies. Why is that so hard to do? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Fri, 18 Sep 2009 11:17:40 +0100, Chris Withers <chris@simplistix.co.uk> wrote:
A lot of these problems would go away if we could just treat "Python" as a package like every other package when it comes to dependencies.
Why is that so hard to do?
It's not hard at all. But somebody needs to show how easy it is to do first, and then once that happens, it can be 'forked'. Perphaps implemented differently (white-boxed) from the original. So that somebody else can put their name on the idea. Forking in distutils is perphaps the new way of having innovation and progress. Don't worry Chris, if your idea is good enough, it is going to get forked... David
David Lyon wrote:
Don't worry Chris, if your idea is good enough, it is going to get forked...
Sorry, I misheard that last word, in which case I agree ;-) Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Fri, Sep 18, 2009 at 11:17:40AM +0100, Chris Withers wrote:
Floris Bruynooghe wrote:
On Fri, Sep 11, 2009 at 07:02:53PM +0200, Tarek Ziadé wrote:
On Fri, Sep 11, 2009 at 6:32 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
On Fri, Sep 11, 2009 at 05:14:58PM +0200, Tarek Ziadé wrote: [...]
That can be dropped since we have '==' and it's all strings. '2.6' < '2.10'?
Although not close you must be wary of this. Bring on PEP 386 ;-) Right. Maybe we can drop < >, >= and >=, completely.
In real world use cases, we can use "or" to make a section work for 2.6, 3.1, etc.. for python versions.
I can't think of a use case where "in" and "==" is not enough.
Version comparison would be nice though. Taking your example from your blog post: if I where to try to install that on Python 2.4 I'd have to edit the condition of the [py25] section to "in ('2.4', '2.5')". Maybe this is what you want though, in case you explicitly don't support for 2.4. OTOH it's nice if things would just work in case of developers simply not having tried earlier versions.
This also shows how "in" implies a tuple or a list as data type, your mini-language was trying to avoid that. Or did I miss something?
A lot of these problems would go away if we could just treat "Python" as a package like every other package when it comes to dependencies.
How would that help in specifying dependencies based on a python version? The conditional syntax as discussed so far does not allow for conditions based on versions of other packages detected. And doing so would seem like a lot more complicated then what's being discussed so far (at first sight anyway, haven't tought too deeply about this yet). Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Fri, Sep 11, 2009 at 05:14:58PM +0200, Tarek Ziadé wrote:
Also, I've just realized that uname() is a tuple, so I need to flatten it in strings to restrict the grammar:
- os_sysname - os_nodename - os_release - os_version - os_machine
When I asked for this I was imagening (wrongly) way more to go in setup.cfg then just the meta-data. To be honest in that case I don't think I've got a use for os.uname() to be available. While nothing is wrong with having it, if no one else thinks they need it maybe it's unnecessary complication? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Tue, Sep 15, 2009 at 5:46 PM, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
On Fri, Sep 11, 2009 at 05:14:58PM +0200, Tarek Ziadé wrote:
Also, I've just realized that uname() is a tuple, so I need to flatten it in strings to restrict the grammar:
- os_sysname - os_nodename - os_release - os_version - os_machine
When I asked for this I was imagening (wrongly) way more to go in setup.cfg then just the meta-data. To be honest in that case I don't think I've got a use for os.uname() to be available. While nothing is wrong with having it, if no one else thinks they need it maybe it's unnecessary complication?
I guess we could leave them since they don't hurt, and see after a version if they are not used by people. os_machine help though, knowing if it's a 32 or 64 bits machine
Regards Floris
-- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
On Tue, 15 Sep 2009 16:46:59 +0100, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
On Fri, Sep 11, 2009 at 05:14:58PM +0200, Tarek Ziadé wrote:
Also, I've just realized that uname() is a tuple, so I need to flatten it in strings to restrict the grammar:
- os_sysname - os_nodename - os_release - os_version - os_machine
When I asked for this I was imagening (wrongly) way more to go in setup.cfg then just the meta-data. To be honest in that case I don't think I've got a use for os.uname() to be available. While nothing is wrong with having it, if no one else thinks they need it maybe it's unnecessary complication?
I think it is an unneccessary complication. And imo the discussion has gone way off track.. The use case isn't abstract. All this discussion is about trying to rewrite two lines of code. -- setup.py -- """ if sys.platform == 'win32': setup.dependencies.add('win32com','162') setup() """ That would be the simplest way to do it in code. It appears as if "if sys.platform == 'win32':" is an evil line of code - that certain people want to go to great lengths to stomp out. It's one line of code for crying out loud... We don't need a mini-language just because we don't like writing it the shortest way... talk about platform bias gone utterly crazy... David
On Sep 15, 2009, at 7:52 PM, David Lyon wrote:
And imo the discussion has gone way off track..
The use case isn't abstract. All this discussion is about trying to rewrite two lines of code.
-- setup.py -- """ if sys.platform == 'win32': setup.dependencies.add('win32com','162')
setup()
"""
That would be the simplest way to do it in code.
It appears as if "if sys.platform == 'win32':" is an evil line of code - that certain people want to go to great lengths to stomp out.
It's one line of code for crying out loud...
We don't need a mini-language just because we don't like writing it the shortest way...
talk about platform bias gone utterly crazy...
Assuming that it really is that simple... +1 on everything above. A 1 line change is much better than a 30 message debate, BNF diagram, and DSL for a simple case. If it doesn't work out for some reason, some _real_ reason, debate it then in context, if it ever comes up again. S
On 12:09 am, ssteinerx@gmail.com wrote:
On Sep 15, 2009, at 7:52 PM, David Lyon wrote:
And imo the discussion has gone way off track..
The use case isn't abstract. All this discussion is about trying to rewrite two lines of code.
-- setup.py -- """ if sys.platform == 'win32': setup.dependencies.add('win32com','162')
setup()
"""
That would be the simplest way to do it in code.
It appears as if "if sys.platform == 'win32':" is an evil line of code - that certain people want to go to great lengths to stomp out.
It's one line of code for crying out loud...
We don't need a mini-language just because we don't like writing it the shortest way...
talk about platform bias gone utterly crazy...
Assuming that it really is that simple...
+1 on everything above.
A 1 line change is much better than a 30 message debate, BNF diagram, and DSL for a simple case. If it doesn't work out for some reason, some _real_ reason, debate it then in context, if it ever comes up again.
I don't think it's really this simple. The point is to have static definition of the package information. Having a Python program define this information at runtime does not satisfy this requirement. You can say that the simple sys.platform test is not evil, but how does that lead to a solution allowing PyPI to tell users what the dependencies for a package are on their platform? Still, it may be that if the only use case here is conditionally declaring a dependency if the install target is Windows, a simpler solution might suffice. I won't try to describe such a solution. Instead, I'll ask if that premise seems valid. What use cases do we have? There's the one described above, which lots of people have been talking about. I think there's another one related to target Python version - eg, on Python 2.3, depend on simplejson, but on Python 2.6, don't. What else? Jean-Paul
On Wed, 16 Sep 2009 00:42:24 -0000, exarkun@twistedmatrix.com wrote:
I don't think it's really this simple.
If it isn't, then it should be made to be that simple...
The point is to have static definition of the package information.
It would be a huge win for everybody if we could get over this hurdle.
Having a Python program define this information at runtime does not satisfy this requirement.
I agree with that. That's why I suggested it in the first place.
You can say that the simple sys.platform test is not evil, but how does that lead to a solution allowing PyPI to tell users what the dependencies for a package are on their platform?
I've been ardently campaigning for the static metadata to include platform specific package definition for a number of months. I'll just assume you missed my posts. Here I'll just redescribe a typical win32/linux situation. On win32 the odbc is built in, but you need win32com to access it. On Linux, you don't have odbc so you need pyodbc. """ [setup] name = myframework version = 1.1 [dependencies] packages = cherrypy, 1.1 objectmapper, 1.5 [dependencies win32] packages = win32com [dependencies linux2] packages = pyodbc """ It is really very simple.
Still, it may be that if the only use case here is conditionally declaring a dependency if the install target is Windows, a simpler solution might suffice. I won't try to describe such a solution. Instead, I'll ask if that premise seems valid.
Yes. The premise is true for both windows and linux. They're similar but different.
What use cases do we have? There's the one described above, which lots of people have been talking about. I think there's another one related to target Python version - eg, on Python 2.3, depend on simplejson, but on Python 2.6, don't. What else?
Ok. Not what else, here's the complete list: 1) Additional packages on old python versions . eg need lxml if xml support isn't available. simplejson. 2) Additional packages due to o/s ommission. eg win32com on on windows and pyodbc on linux. They are the typical use cases. It's either one or two lines in a config file or one or two lines in the setup.py script.. So it is either 1) """ [dependencies linux2] packages = pyodbc """ or.. 2) """ if setup.platform == "linux2": setup.dependencies.append('pyodbc') """ Anyway, the real issue here is philosophy. Eliminating platform specifics vs making it simple for real world use cases. I'm also pretending that I didn't look in distutils code and see how many "darwin" references there actually are. But it isn't about that.. just being able to conditionally install extra packages as required based on whichever of the three major o/s the user happened to choose. David
David Lyon kirjoitti:
On Wed, 16 Sep 2009 00:42:24 -0000, exarkun@twistedmatrix.com wrote:
I don't think it's really this simple.
If it isn't, then it should be made to be that simple...
The point is to have static definition of the package information.
It would be a huge win for everybody if we could get over this hurdle.
Having a Python program define this information at runtime does not satisfy this requirement.
I agree with that. That's why I suggested it in the first place.
You can say that the simple sys.platform test is not evil, but how does that lead to a solution allowing PyPI to tell users what the dependencies for a package are on their platform?
I've been ardently campaigning for the static metadata to include platform specific package definition for a number of months. I'll just assume you missed my posts.
Here I'll just redescribe a typical win32/linux situation. On win32 the odbc is built in, but you need win32com to access it. On Linux, you don't have odbc so you need pyodbc.
""" [setup] name = myframework version = 1.1
[dependencies] packages = cherrypy, 1.1 objectmapper, 1.5
[dependencies win32] packages = win32com
[dependencies linux2] packages = pyodbc
"""
It is really very simple.
Still, it may be that if the only use case here is conditionally declaring a dependency if the install target is Windows, a simpler solution might suffice. I won't try to describe such a solution. Instead, I'll ask if that premise seems valid.
Yes. The premise is true for both windows and linux. They're similar but different.
What use cases do we have? There's the one described above, which lots of people have been talking about. I think there's another one related to target Python version - eg, on Python 2.3, depend on simplejson, but on Python 2.6, don't. What else?
Ok. Not what else, here's the complete list:
1) Additional packages on old python versions . eg need lxml if xml support isn't available. simplejson.
2) Additional packages due to o/s ommission. eg win32com on on windows and pyodbc on linux.
They are the typical use cases.
It's either one or two lines in a config file or one or two lines in the setup.py script..
So it is either
1) """ [dependencies linux2] packages = pyodbc """
or..
2) """ if setup.platform == "linux2": setup.dependencies.append('pyodbc') """
Anyway, the real issue here is philosophy. Eliminating platform specifics vs making it simple for real world use cases.
I'm also pretending that I didn't look in distutils code and see how many "darwin" references there actually are. But it isn't about that.. just being able to conditionally install extra packages as required based on whichever of the three major o/s the user happened to choose.
Make sure you don't forget about Java (which is a "platform" of its own) and .NET.
David
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Sep 15, 2009, at 8:42 PM, exarkun@twistedmatrix.com wrote:
What use cases do we have? There's the one described above, which lots of people have been talking about. I think there's another one related to target Python version - eg, on Python 2.3, depend on simplejson, but on Python 2.6, don't. What else?
Ok, so let's work backwards from use cases to tests to code. Sounds like something I heard once, somewhere... Do we start by breaking it down by: Python Version Platform and drill down from there? Seems to me, from there there are lots of details to handle but, without those big chunks out of the way, there's really no way to meaningfully discuss any further details. Anyone have any more "big chunks?" S
On Wed, Sep 16, 2009 at 2:42 AM, <exarkun@twistedmatrix.com> wrote:
What use cases do we have? There's the one described above, which lots of people have been talking about. I think there's another one related to target Python version - eg, on Python 2.3, depend on simplejson, but on Python 2.6, don't. What else?
So if I resume, so far the uses cases are: - the OS given by os.name and sys.platform (linux/mac/windows/riscos/ce/ etc..) - the architecture, given by os.uname() (32/64 bits) - the python version, given sys.version_info Tarek
On Tue, 15 Sep 2009 17:42:24 -0700, <exarkun@twistedmatrix.com> wrote:
What use cases do we have? There's the one described above, which lots of people have been talking about. I think there's another one related to target Python version - eg, on Python 2.3, depend on simplejson, but on Python 2.6, don't. What else?
Today I stumbled on a project called PyGUI that depends on packages based on the platform. http://www.cosc.canterbury.ac.nz/greg.ewing/python_gui/ """ MacOSX - requires PyObjC (tested with PyObjC-1.2) Linux - requires pygtk (tested with pygtk-2.14.0 and Gtk+-2.14.7) Windows - requires pywin32 (tested with pywin32-213) and ctypes (included with Python 2.5 and later) """ If we use setuptools' extras like syntax, this would be: extras_require={'platform=macosx': ['pyobjc'], 'platform=win32': ['pywin32', 'ctypes'] 'platform=unix': ['pygtk']} -srid
On Fri, Sep 18, 2009 at 8:13 PM, Sridhar Ratnakumar <sridharr@activestate.com> wrote:
On Tue, 15 Sep 2009 17:42:24 -0700, <exarkun@twistedmatrix.com> wrote:
What use cases do we have? There's the one described above, which lots of people have been talking about. I think there's another one related to target Python version - eg, on Python 2.3, depend on simplejson, but on Python 2.6, don't. What else?
Today I stumbled on a project called PyGUI that depends on packages based on the platform.
http://www.cosc.canterbury.ac.nz/greg.ewing/python_gui/
""" MacOSX - requires PyObjC (tested with PyObjC-1.2) Linux - requires pygtk (tested with pygtk-2.14.0 and Gtk+-2.14.7) Windows - requires pywin32 (tested with pywin32-213) and ctypes (included with Python 2.5 and later) """
If we use setuptools' extras like syntax, this would be:
extras_require={'platform=macosx': ['pyobjc'], 'platform=win32': ['pywin32', 'ctypes'] 'platform=unix': ['pygtk']}
I think you miss the "included with Python 2.5 and later" part Anyway, I find it rather verbose compared to what we have proposed so far in setup.cfg I think the latest proposal, where the condition is located in the section name is good. So I'll just BDFL a bit here to move things forward. I am adding it in Distutils, with a notification mail in python-dev, and a blog entry describing the result. Thanks all for the feedback ! -- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
On Sun, Sep 13, 2009 at 2:56 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Tarek Ziadé wrote:
[setup] foo: bar conditional-sections: one, two
Is conditional-sections: really necessary? Isn't the presence of a conditions: line in a section enough to mark it as conditional?
I find having a conditional section explicitely defined better that doing a convention on names because if someone implements a command that uses that name (it's likely to be improbable but...) it will break his ability to use setup.cfg to give options to his commands Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
Tarek Ziadé wrote:
I find having a conditional section explicitely defined better that doing a convention on names because if someone implements a command that uses that name (it's likely to be improbable but...) it will break his ability to use setup.cfg to give options to his commands
I don't follow that. If a "conditions:" line is being used to express the condition, then you can't use the name "conditions" for anything else anyway. -- Greg
On Tue, Sep 15, 2009 at 10:47 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Tarek Ziadé wrote:
I find having a conditional section explicitely defined better that doing a convention on names because if someone implements a command that uses that name (it's likely to be improbable but...) it will break his ability to use setup.cfg to give options to his commands
I don't follow that. If a "conditions:" line is being used to express the condition, then you can't use the name "conditions" for anything else anyway.
setupt.cfg is also used by commands that is. Imagine a command with the name "foo", that has an option called "condition". The setup.cfg file might look like this: """ [global] verbose: 1 [setup] name: MyDistribution version: 1.0 [only_windows] condition: sys_platform == 'win32' requires: pywin32 [foo] verbose: 1 condition: badababdalolo """" Here, the foo section is not supposed to be used by the setup section, it's just a section for the command value, which is allowed (see http://docs.python.org/distutils/configfile.html) Now Distutils, if told to grab all sections that have a "condition" variable in them, will fail at reading "badababdalolo". With an explicit list of the conditional sections, the probelm can't occur, because it's explicit: """ [global] verbose: 1 [setup] name: MyDistribution version: 1.0 conditional-sections: only_windows [only_windows] condition: sys_platform == 'win32' requires: pywin32 [foo] verbose: 1 condition: badababdalolo """" Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
Tarek Ziadé wrote:
Imagine a command with the name "foo", that has an option called "condition".
The setup.cfg file might look like this:
""" [global] verbose: 1
[setup] name: MyDistribution version: 1.0
[only_windows] condition: sys_platform == 'win32' requires: pywin32
[foo] verbose: 1 condition: badababdalolo """"
Here, the foo section is not supposed to be used by the setup section, it's just a section for the command value, which is allowed (see http://docs.python.org/distutils/configfile.html)
Ooh, nasty. I would -want- this case to error out because it could introduce a hard to find problem. Perhaps that condition: is intended for use by [setup] but a tired person forgot to list it in [setup]. He keeps looking at his setup.cfg and cannot see why it doesn't use his condition. I'd rather a big error popped up than a silent ambiguity. -Jeff
On Tue, Sep 15, 2009 at 1:34 PM, Jeff Rush <jeff@taupro.com> wrote:
Tarek Ziadé wrote:
Imagine a command with the name "foo", that has an option called "condition".
The setup.cfg file might look like this:
""" [global] verbose: 1
[setup] name: MyDistribution version: 1.0
[only_windows] condition: sys_platform == 'win32' requires: pywin32
[foo] verbose: 1 condition: badababdalolo """"
Here, the foo section is not supposed to be used by the setup section, it's just a section for the command value, which is allowed (see http://docs.python.org/distutils/configfile.html)
Ooh, nasty. I would -want- this case to error out because it could introduce a hard to find problem. Perhaps that condition: is intended for use by [setup] but a tired person forgot to list it in [setup]. He keeps looking at his setup.cfg and cannot see why it doesn't use his condition. I'd rather a big error popped up than a silent ambiguity.
Another options is to prefix those sections that are to be used with [setup]: """ [global] verbose: 1 [setup] name: MyDistribution version: 1.0 [setup:only_windows] condition: sys_platform == 'win32' requires: pywin32 [setup:only_mac] condition: sys_platform == 'darwin' requires: pywin32 [foo] verbose: 1 condition: badababdalolo """"
-Jeff
-- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
On Tue, Sep 15, 2009 at 01:38:22PM +0200, Tarek Ziadé wrote:
Another options is to prefix those sections that are to be used with [setup]:
""" [global] verbose: 1
[setup] name: MyDistribution version: 1.0
[setup:only_windows] condition: sys_platform == 'win32' requires: pywin32
[setup:only_mac] condition: sys_platform == 'darwin' requires: pywin32
[foo] verbose: 1 condition: badababdalolo """"
That's quite nice, +1 And is it worth to consider erroring out when you have a [setup:.*] section without a "condition: ..." line? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Tarek Ziadé wrote:
Imagine a command with the name "foo", that has an option called "condition".
If you really want to keep the entire keyword namespace available for use as a section sees fit, it would be better to put the conditions inside the section headers, as I suggested earlier. I think that would also make it easier to see which sections are conditional when looking through the file. -- Greg
On Wed, Sep 16, 2009 at 2:30 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Tarek Ziadé wrote:
Imagine a command with the name "foo", that has an option called "condition".
If you really want to keep the entire keyword namespace available for use as a section sees fit, it would be better to put the conditions inside the section headers, as I suggested earlier.
I think that would also make it easier to see which sections are conditional when looking through the file.
yes, that's another option. It just feel weird to my eyes but I guess that's just me.. So this would be: """ [global] verbose: 1 [setup] name: MyDistribution version: 1.0 [setup:sys_platform == 'win32'] requires: pywin32 [setup:sys_platform == 'darwin'] requires: foo [setup:'linux' in sys_platform] requires: bar [foo] verbose: 1 condition: badababdalolo """" I am not including 'requires' in the section like you have previously shown, since it's a metadata field and since we will probably have other fields in these section (see PEP 341 proposals like 'obsoles') Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
On Tue, 15 Sep 2009 02:00:41 -0700, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
[only_windows] condition: sys_platform == 'win32' requires: pywin32
I am ambivalent about using sys.platform. How do I specify Linux? >>> sys.platform 'linux2' "linux2"? What are the possible values for `sys.platform`? Is it even a well-defined list? What if Linux 3.x comes out .. and the package would stop using the conditional dependent package (that is dependent on 'linux2')? cf. http://code.google.com/p/platinfo/issues/detail?id=1 -srid
2009/9/15 Sridhar Ratnakumar <sridharr@activestate.com>:
On Tue, 15 Sep 2009 02:00:41 -0700, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
[only_windows] condition: sys_platform == 'win32' requires: pywin32
I am ambivalent about using sys.platform. How do I specify Linux?
>>> sys.platform 'linux2'
"linux2"? What are the possible values for `sys.platform`? Is it even a well-defined list?
Yes of course, http://docs.python.org/library/sys.html os.uname() also give you more details on the platform/
What if Linux 3.x comes out .. and the package would stop using the conditional dependent package (that is dependent on 'linux2')?
The same thing that what would happen when Python 4 is out I guess, all distributions will have to change their dependencies list (thus the setup.cfg file) accordingly in their next release.
-srid
-- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
Sridhar Ratnakumar wrote:
sys.platform 'linux2'
"linux2"? What are the possible values for `sys.platform`?
This is why I usually write Python code that checks sys.platform like if sys.platform.startswith("linux"): ... So the condition language for the metadata needs a "startswith" operator, I think. -- Greg
On Wed, Sep 16, 2009 at 10:21 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Sridhar Ratnakumar wrote:
>>> sys.platform 'linux2'
"linux2"? What are the possible values for `sys.platform`?
This is why I usually write Python code that checks sys.platform like
if sys.platform.startswith("linux"): ...
So the condition language for the metadata needs a "startswith" operator, I think.
What about 'in' ? 'linux' in sys.platform should cover all cases
-- Greg _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé | http://ziade.org | オープンソースはすごい!
participants (15)
-
Alex Grönholm -
Chris Withers -
David Lyon -
exarkun＠twistedmatrix.com -
Floris Bruynooghe -
Glyph Lefkowitz -
Greg Ewing -
Jeff Rush -
Jim Fulton -
P.J. Eby -
Ronald Oussoren -
Sridhar Ratnakumar -
ssteinerX＠gmail.com -
Tarek Ziadé -
Tres Seaver