[RFC] Recentering the static metadata need : PKG-INFO
Hey, Marc-André Lemburg gave me a key points about the static metadata discussions we have (wether its PEP 390 or around it) He said that the important thing was to have the context-dependant markers in PKG-INFO, and that having in described in setup.cfg or in setup.py by any way is not the important thing. And he is right ! So, PEP 390 proposes a new setup.cfg format, that's fine. And others are thinking about other ways to build metadata, that's great. But what really matters at the end is to provide a new Metadata format where these marker are present, so any consumer can play with them (and Distutils provide an API to play with them : the DistributionMetadata class) So please, have a look at the new section I've added here with the syntax I am proposing : http://www.python.org/dev/peps/pep-0390/#impact-on-pkg-info-generation-and-p... That will probably re-center the debate in creating a Metadata 1.2 format that includes these markers. As someone asked, here is the principal uses cases of having static metadata that includes markers like this: - pip, or easy_install, or any client code, will be able to query PyPI, to get the metadata of a package. *and* will be able to have it depending on a particular environment. - it will be possible to generate and provide alongside the source code the PKG-INFO file, like how install_egg_info does at installation time, exactly like how MANIFEST is built. If that sounds like the right path (it does to me), we could start to work on a PEP 314 update, and add in the work what was started last year: - revision of "requires", "obsoletes", etc.. - link to PEP 386 Regards Tarek
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tarek Ziadé wrote:
Hey,
Marc-André Lemburg gave me a key points about the static metadata discussions we have (wether its PEP 390 or around it)
He said that the important thing was to have the context-dependant markers in PKG-INFO, and that having in described in setup.cfg or in setup.py by any way is not the important thing. And he is right !
So, PEP 390 proposes a new setup.cfg format, that's fine. And others are thinking about other ways to build metadata, that's great.
But what really matters at the end is to provide a new Metadata format where these marker are present, so any consumer can play with them (and Distutils provide an API to play with them : the DistributionMetadata class)
So please, have a look at the new section I've added here with the syntax I am proposing :
http://www.python.org/dev/peps/pep-0390/#impact-on-pkg-info-generation-and-p...
That will probably re-center the debate in creating a Metadata 1.2 format that includes these markers.
As someone asked, here is the principal uses cases of having static metadata that includes markers like this:
- pip, or easy_install, or any client code, will be able to query PyPI, to get the metadata of a package. *and* will be able to have it depending on a particular environment.
- it will be possible to generate and provide alongside the source code the PKG-INFO file, like how install_egg_info does at installation time, exactly like how MANIFEST is built.
If that sounds like the right path (it does to me), we could start to work on a PEP 314 update, and add in the work what was started last year:
- revision of "requires", "obsoletes", etc.. - link to PEP 386
Shouldn't this point at PEP 345, which already defines a metadata version 1.2? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrYhz8ACgkQ+gerLs4ltQ7HxACeP6BVcIIze+WKjjCe/1ED9ko8 oYIAnAu9KC/10+4u8rJ8S9bUr1V7MbLj =qsDv -----END PGP SIGNATURE-----
On Fri, Oct 16, 2009 at 4:46 PM, Tres Seaver <tseaver@palladion.com> wrote:
Shouldn't this point at PEP 345, which already defines a metadata version 1.2?
Right sorry, I messed up with the PEPs numbers. PEP 314 is the final 1.1, and PEP 345 is the draft for 1.2, where we should work to change PKG-INFO. Tarek
On Fri, Oct 16, 2009 at 8:54 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Marc-André Lemburg gave me a key points about the static metadata discussions we have (wether its PEP 390 or around it)
He said that the important thing was to have the context-dependant markers in PKG-INFO, and that having in described in setup.cfg or in setup.py by any way is not the important thing. And he is right !
Excellent, this resolves my own concern about the discussion as well. Putting this in PKG-INFO is something I can concretely make use of, regardless of how it is generated. PKG-INFO is already somewhat flawed as a format for holding the data, in particularly maintaining indentation for Description. I think adding a general "...; <condition>" places another syntactic constraint, where no field can have ";" in it. Ideally I'd like to see both cases resolved. And certainly I don't want to end up in a place where weird bugs emerge if I put ";" in a field (especially since many are free text). -- Ian Bicking | http://blog.ianbicking.org | http://topplabs.org/civichacker
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ian Bicking wrote:
On Fri, Oct 16, 2009 at 8:54 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Marc-André Lemburg gave me a key points about the static metadata discussions we have (wether its PEP 390 or around it)
He said that the important thing was to have the context-dependant markers in PKG-INFO, and that having in described in setup.cfg or in setup.py by any way is not the important thing. And he is right !
Excellent, this resolves my own concern about the discussion as well. Putting this in PKG-INFO is something I can concretely make use of, regardless of how it is generated.
PKG-INFO is already somewhat flawed as a format for holding the data, in particularly maintaining indentation for Description. I think adding a general "...; <condition>" places another syntactic constraint, where no field can have ";" in it. Ideally I'd like to see both cases resolved. And certainly I don't want to end up in a place where weird bugs emerge if I put ";" in a field (especially since many are free text).
The only fields where I see this syntax making any sense are the "requirements" ones: - - The ones specified in PEP 314: Requires, Provides, Obsoletes - - The new ones proposed in PEP 345[1]: Requires-Dist, Provides-Dist, Obsoletes-Dist [1] As amended here: http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?revision=73235&view=markup Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrYtiYACgkQ+gerLs4ltQ5XhACeJ1XHIreqAU4IipKoa0UHH2dO ADQAnRddgnKzi6iWw5Nd9T9cCOnbFE5S =wsCj -----END PGP SIGNATURE-----
On Fri, Oct 16, 2009 at 8:06 PM, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Ian Bicking wrote:
On Fri, Oct 16, 2009 at 8:54 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Marc-André Lemburg gave me a key points about the static metadata discussions we have (wether its PEP 390 or around it)
He said that the important thing was to have the context-dependant markers in PKG-INFO, and that having in described in setup.cfg or in setup.py by any way is not the important thing. And he is right !
Excellent, this resolves my own concern about the discussion as well. Putting this in PKG-INFO is something I can concretely make use of, regardless of how it is generated.
PKG-INFO is already somewhat flawed as a format for holding the data, in particularly maintaining indentation for Description. I think adding a general "...; <condition>" places another syntactic constraint, where no field can have ";" in it. Ideally I'd like to see both cases resolved. And certainly I don't want to end up in a place where weird bugs emerge if I put ";" in a field (especially since many are free text).
The only fields where I see this syntax making any sense are the "requirements" ones:
- - The ones specified in PEP 314: Requires, Provides, Obsoletes - - The new ones proposed in PEP 345[1]: Requires-Dist, Provides-Dist, Obsoletes-Dist
Yes, that would have to be a subset of the fields, and ";" makes PKG-INFO still RFC 232 compatible. Tarek
On Fri, 16 Oct 2009 15:54:29 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
But what really matters at the end is to provide a new Metadata format where these marker are present, so any consumer can play with them (and Distutils provide an API to play with them : the DistributionMetadata class)
I'm fine with this. Although I suggest that we could clean up the language a bit using some of the conditional compound ideas from what I'm working on.
So please, have a look at the new section I've added here with the syntax I am proposing :
http://www.python.org/dev/peps/pep-0390/#impact-on-pkg-info-generation-and-p...
Metadata-Version: 1.2 Name: distribute Version: 0.6.4 ... Requires: pywin32, bar > 1.0; sys_platform == 'win32' Requires: foo; os_machine == 'i386' Requires: bar; python_version == '2.4' or python_version == '2.5' Requires: baz; 'linux' in sys_platform Obsoletes = pywin31; sys_platform == 'win32' ... Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Python Software Foundation License something like this (as best I can read the other format): Metadata-Version: 1.2 Name: distribute Version: 0.6.4 ... Requires: (windows) pywin32, bar > 1.0 Requires: (linux-suse-kde-64) foo Requires: (python < 2.5) bar Requires: (linux) baz Obsoletes = (windows) pywin31 ... Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Python Software Foundation License Regards David
On Sat, 17 Oct 2009 01:15:25 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On Sat, Oct 17, 2009 at 12:39 AM, David Lyon <david.lyon@preisshare.net> wrote:
Requires: (linux-suse-kde-64) foo
How this expression would be verified on the target system ?
I'm working on coding it. Basically the strings are a in a list. # -- These strings describe different platforms standard_platform_bits = ('windows','linux','mac', 'xp','vista', 'os/x', '32','64', 'kde','gnome', 'wx', 'ubuntu','debian','suse','redhat','gentoo', 'centos','symbian' ) Then they go into a dictionary: platform_bits = {'windows' = False, 'linux' = False, 'mac' = False, 'xp' = False, 'vista' = False, 'os/x' = False, '32' = False, '64' = False, 'kde' = False, 'gnome' = False, 'wx' = False, 'gtk' = False, 'ubuntu' = False, 'debian' = False, 'suse' = False, 'redhat' = False, 'gentoo' = False, 'centos' = False, 'symbian = False' } # -- See what our platform is def build_platform_bit_map(): if sys.platform == 'darwin': self.platform_bits['mac'] = True # -- Further sub-bit determination .. else if sys.platform == 'win32': self.platform_bits['windows'] = True # -- Further sub-bit determination .. else if sys.platform == 'linux2': # -- Further sub-bit determination .. I'm working on the rest. David
On Sat, Oct 17, 2009 at 1:38 AM, David Lyon <david.lyon@preisshare.net> wrote:
# -- See what our platform is def build_platform_bit_map():
if sys.platform == 'darwin': self.platform_bits['mac'] = True
# -- Further sub-bit determination ..
else if sys.platform == 'win32': self.platform_bits['windows'] = True
# -- Further sub-bit determination .. else if sys.platform == 'linux2':
# -- Further sub-bit determination
So far.. so good. It's basically what we proposed earlier in PEP 390, but for the distribution, the desktop (kde, etc)... That would require a new stdlib module that would be very hard to implement and maintain since it's a fast moving ground.
On Sat, 17 Oct 2009 01:53:48 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
So far.. so good. It's basically what we proposed earlier in PEP 390, but for the distribution, the desktop (kde, etc)... .. That would require a new stdlib module that would be very hard to implement and maintain since it's a fast moving ground.
I didn't see your pep for a new hard to implement stdlib module.. What number PEP was that? :-) Jokes aside, these desktops are easy to check for with one line of python code. It's either an environment variable or a configuration file. For example: if os.environ['KDEDIR'] <> '': That's a pretty good check to see if KDE is installed and running and won't bomb out, even if you try running it on windows. Most of these environment variables haven't changed in the last 10 years or more. I'd assert that there are some very static environment variables in these linux window managers. Regards David
On Sat, Oct 17, 2009 at 3:03 AM, David Lyon <david.lyon@preisshare.net> wrote:
On Sat, 17 Oct 2009 01:53:48 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
So far.. so good. It's basically what we proposed earlier in PEP 390, but for the distribution, the desktop (kde, etc)... .. That would require a new stdlib module that would be very hard to implement and maintain since it's a fast moving ground.
I didn't see your pep for a new hard to implement stdlib module..
What number PEP was that? :-)
There's no PEP for that because I don't want to add such a module. The context dependant marker are just relying on os, platform and sys.
It's either an environment variable or a configuration file.
For example:
if os.environ['KDEDIR'] <> '':
That's a pretty good check to see if KDE is installed and running and won't bomb out, even if you try running it on windows.
Most of these environment variables haven't changed in the last 10 years or more. I'd assert that there are some very static environment variables in these linux window managers.
If you can find a way to express conditions based on os.environ, maybe os.environ could be added to the variables that are used when checking the markers on a given platform. e.g. like = 'KDEDIR' in os.environ (so meaning we introduce sequence comparisons/operators in those markers) But we can't possibly add in the PKG-INFO description, a hardcoded list of plaftorms like you have shown (kde, ubuntu, etc). Do you have an example for a platform that would depend on a configuration file ? Tarek.
On Sat, 17 Oct 2009 18:19:44 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
If you can find a way to express conditions based on os.environ, maybe os.environ could be added to the variables that are used when checking the markers on a given platform.
I'm not sure that you should try to do that. It would make things too complicated for the developer. I'm not sure that they would like to guess environment variables at an end-users system. It's different if the api does it internally. The developer then doesn't see how the check for kde or gnome is actually performed.
e.g. like = 'KDEDIR' in os.environ (so meaning we introduce sequence comparisons/operators in those markers)
I suggest that you just add a variable for 'desktop' or something to your pep and allow them to check for that. I've told you an easy way to check (environment variables) so there's no reason you couldn't do it the same way.
Do you have an example for a platform that would depend on a configuration file ?
What I meant there was that desktops such as gnome or kde or flux or others can also be detected internally by the presence of their configuration files on linux. http://www.ibiblio.org/oswg/oswg-nightly/oswg/en_US.ISO_8859-1/articles/gdm-... For example, if there is a /etc/gdm/gdm.conf it is pretty safe to assume that gnome is installed on that system. So in your PEP, your could have code.. if sys.platform == 'linux2': gnome_installed = os.path.exists('/etc/gdm/gdm.conf') Then exposed that as a variable in your condition: 'if desktop in "gnome|windows"' etc Regards David
On Sat, Oct 17, 2009 at 06:00:51PM -0400, David Lyon wrote:
On Sat, 17 Oct 2009 18:19:44 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
e.g. like = 'KDEDIR' in os.environ (so meaning we introduce sequence comparisons/operators in those markers) [...] For example, if there is a /etc/gdm/gdm.conf it is pretty safe to assume that gnome is installed on that system.
So in your PEP, your could have code..
if sys.platform == 'linux2': gnome_installed = os.path.exists('/etc/gdm/gdm.conf')
Then exposed that as a variable in your condition:
'if desktop in "gnome|windows"' etc
All this seems like a bad idea to me. Thinking from a GNU/Linux distro point of view now you'd need to add a build-depends to e.g. GDM *and* KDM only to get the correct .desktop file or something like that. And even worse, as proposed it would probably get confused when both are installed. There must be better ways of saying if you want to have (e.g.) a .desktop file installed. Also the choice you give to a Python package author is suddenly massive: linux-gnome-kde-32-whatever. Just how is a Python package author supposed to get this right? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Mon, 19 Oct 2009 10:18:57 +0100, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
All this seems like a bad idea to me. Thinking from a GNU/Linux distro point of view now you'd need to add a build-depends to e.g. GDM *and* KDM only to get the correct .desktop file or something like that. And even worse, as proposed it would probably get confused when both are installed.
I don't see why. An application/package might use KDE and might want to check for it. An application/package might want to use GNOME and want to check for it. Why can't developers check what the platform is? There's no forcing developers to use these checks. They're only there to assist in overcoming platform specific installation issues.
There must be better ways of saying if you want to have (e.g.) a .desktop file installed.
Ok.. so throw me some ideas..
Also the choice you give to a Python package author is suddenly massive: linux-gnome-kde-32-whatever. Just how is a Python package author supposed to get this right?
Typically, the don't specify anything. They just assume everything works on every platform until their test process tells them it isn't so. In that case, this mechanism allows them to provide workarounds for problem cases. I cannot see how that is a bad idea... David
Floris Bruynooghe wrote:
For example, if there is a /etc/gdm/gdm.conf it is pretty safe to assume that gnome is installed on that system.
So in your PEP, your could have code..
if sys.platform == 'linux2': gnome_installed = os.path.exists('/etc/gdm/gdm.conf')
Then exposed that as a variable in your condition:
'if desktop in "gnome|windows"' etc
Agreed, this is too fragile and complicated. Anyone who has done packaging knows how fragile those things are: some buggy packages do not remove some files, some corner cases you did not think about are always coming. At least for things like platform, arch and os, things are quite stable. For everything else, either there is a way to define our own variable whose values are handled by outside code (say setup.py or command line), or you don't make them available at all in the static file. I don't see other solution, and the packaging solutions I know do not allow this kind of things. For any new feature in distutils, it should almost be mandatory to look at what other people did to the same problem space. Good design only comes with experience in that space domain, IMO. David
On Tue, 20 Oct 2009 12:41:52 +0900, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
'if desktop in "gnome|windows"' etc
Agreed, this is too fragile and complicated.
I don't understand which part is fragile. Detecting if gnome or windows exists? or implementing this via Tareks proposal (sort of shown as best as I understand it above). My proposal has always been: [dependencies windows] packages = win32com If you are really suggesting that detecting windows is so fragile that it really shouldn't be supported I really don't have much further to say. Windows is fragile... linux is fragile.. python packaging is fragile..
At least for things like platform, arch and os, things are quite stable.
Ok, so you agree with everything up to defining the window manager. That's good. Detecting the actual window manager is a fairly trivial detail. In reality.
For everything else, either there is a way to define our own variable whose values are handled by outside code (say setup.py or command line), or you don't make them available at all in the static file. I don't see other solution, and the packaging solutions I know do not allow this kind of things. For any new feature in distutils, it should almost be mandatory to look at what other people did to the same problem space. Good design only comes with experience in that space domain, IMO.
I do many desktop software installs. I have a good working knowledge of the issues. Ok - I'm agreeing. So lets make all the python packaging as good as it is in perl. Really, you're only objection comes down to the tags 'gnome' and 'kde' if I'm not mistaken. To me they aren't big issues. Ok, so we have to ommit 4 lines of code. I'm not bothered. David
On Sun, Oct 18, 2009 at 12:00 AM, David Lyon <david.lyon@preisshare.net> wrote:
So in your PEP, your could have code..
if sys.platform == 'linux2': gnome_installed = os.path.exists('/etc/gdm/gdm.conf')
Then exposed that as a variable in your condition:
'if desktop in "gnome|windows"' etc
From the stdlib PoV, exposing new variables like "gnome" isn't appropriate, because we don't want to keep track of all desktop systems in the stdlib. That's impossible.
So we have to work at providing tools for the developers in the conditions, for them to express them. So exposing os.environ for your use case could help. I am not sure about file presence though. More generally, I feel uncomfurtable to give too much power in the PKG-INFO syntax, I think things like os.platform() are enough to describe conditional fields in the metadata. I can see a use case for building Extensions and package_data etc.. But what is the use case to detect gnome for the metadata fields ? Tarek
On Mon, 19 Oct 2009 13:46:11 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
From the stdlib PoV, exposing new variables like "gnome" isn't appropriate, because we don't want to keep track of all desktop systems in the stdlib. That's impossible.
Well, the list isn't infinite. And under Linux there are only two major ones existing in any great quantity unless I am mistaken. They are Gnome and KDE. The rest.. aren't so important for detection I would suggest.. ok - somebody stick a stake in my heart... haha
So exposing os.environ for your use case could help. I am not sure about file presence though.
As I said before.. I'm not so keen on that.
More generally, I feel uncomfurtable to give too much power in the PKG-INFO syntax, I think things like os.platform() are enough to describe conditional fields in the metadata.
But what is the use case to detect gnome for the metadata fields ?
Perphaps somebody is writing something that only works in gnome. The true rational is slightly different. Under windows there are distinct window manager versions. Eg XP, Vista, Windows 7, win-ce. They *do* have a big impact on python programs and change configurations significantly. So my rationale is simply to have the same thing under linux where there are two major desktops, KDE and GNOME. Maybe on the Mac there are differences to the python developer and we shouldn't forget those also. I'm proposing detecting less than 10 different desktops in all. And detecting those with two lines of code each. So 20 lines of code.. About 20 lines of new code shouldn't be so hard... I would have thought. David
participants (6)
-
David Cournapeau -
David Lyon -
Floris Bruynooghe -
Ian Bicking -
Tarek Ziadé -
Tres Seaver