Re: [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals
At 07:55 PM 10/1/2008 +0200, Tarek Ziadé wrote:
On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby <pje@telecommunity.com> wrote:
I'm -1 on all of the above. I think we need a standard for tools interop (ala WSGI), not implementation tweaks for the existing tools. I also think that a concrete metadata format proposal is premature at this time; we've barely begun to gather -- let alone specify -- our requirements for that metadata. (Essentially, only version dependencies have been discussed, AFAICT.)
What are the other important points we need to discuss at this point in your opinion ?
What information needs to be conveyed by a "build" tool to an "install" tool, and vice versa. For example, an install tool needs to know what files are documentation, which are sample data, and what is part of the library proper, as well as what things are scripts (and in what language those scripts are written, e.g. Python, shell, batch, etc.). Some install tools need to know about icons, menus, registry info, cron jobs, etc. (These are perhaps more properly the domain of applications than libraries, but I'm going to assume that these things are in scope.) The way that this information is communicated needs to be extensible, so that optional metadata for debs and msi's and rpm's and whathaveyou can be incorporated, without needing to modify the standard -- especially if the APIs for reading and writing this data are in the stdlib. There needs to be a way for install tools to ask a source package to "configure" itself, possibly specifying options (and a way for it to find out what those options are, to be able to present them to the user). Then there needs to be a way for the install tool to ask the package to build itself with the configured options, and a standard for how the build tool(s) communicate errors or other issues back to the install tools. There needs to be a way, of course, for the package to specify what build tools it needs in order to be built, and for those to plug in to the (again stdlib-contained) build API. There needs to be a better API for querying C configuration and compiler details, that's separate from the distutils "ccompiler" stuff. Last, but not least, there needs to be a definition of core build and install tools to be both an example/reference implementation of the standard, and to provide the basics needed by the core. I think I've mentioned all of these previously in the thread. I also think that as a matter of technicalities, these things are not difficult to achieve... but if it they are only achieved *technically*, then the result is a failure, not a success. In order for the *real* goal to be achieved (i.e., a flourishing build/install system for Python), widespread participation and buy-in is required. If the OS people or the big package people (e.g. Zope Corp., Enthought) or the distutils aficionados are left out, then the result won't get used. I think the best way to ensure that nobody is left out, is to get them to participate in the design of a standard that ensures that *they* will be able to control their destiny, by creating their own build plugins and/or install tools... or at least having a robust choice of alternatives. We need a consensus "de jure" standard (ala WSGI), rather than just an ongoing "de facto" standard (ala distutils/setuptools), or we're not making any substantial progress, just handing the reins over to somebody else.
Enforcements: - a binary distribution cannot be uploaded if a source distrbution has not been previously provided for the version
Note that this doesn't allow closed-source packages to be uploaded; thus it would need to be a warning, rather than a requirement.
Right. do you agree it is something useful to do ?
Sure.
New features: - we should be able to download the metadata of a package without downloading the package - PyPI should display the install and test dependencies in the UI
It could only do this for specific binaries, since dependencies can be dynamic.
What dynamic means here ? the python module to static file process or more ? can you provide an example ?
Dependencies can be platform-specific as well as python-version specific. If you want ctypes, you would depend on it in python 2.4 but not in 2.5. Similarly, if on some platform a certain library is required to implement a feature that is natively accessible on other platforms. In these cases, you would have logic in setup.py to detect this and choose the appropriate dependencies.
On Wed, Oct 01, 2008 at 02:28:07PM -0400, Phillip J. Eby wrote:
In order for the *real* goal to be achieved (i.e., a flourishing build/install system for Python), widespread participation and buy-in is required. If the OS people or the big package people (e.g. Zope Corp., Enthought) or the distutils aficionados are left out, then the result won't get used.
[...]
We need a consensus "de jure" standard (ala WSGI), rather than just an ongoing "de facto" standard (ala distutils/setuptools), or we're not making any substantial progress, just handing the reins over to somebody else.
Nice words. I like very much the sound of this discussion. Ga�l
On Wed, Oct 1, 2008 at 8:28 PM, Phillip J. Eby <pje@telecommunity.com> wrote:
At 07:55 PM 10/1/2008 +0200, Tarek Ziadé wrote:
On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby <pje@telecommunity.com> wrote:
I'm -1 on all of the above. I think we need a standard for tools interop (ala WSGI), not implementation tweaks for the existing tools. I also think that a concrete metadata format proposal is premature at this time; we've barely begun to gather -- let alone specify -- our requirements for that metadata. (Essentially, only version dependencies have been discussed, AFAICT.)
What are the other important points we need to discuss at this point in your opinion ?
What information needs to be conveyed by a "build" tool to an "install" tool, and vice versa.
For example, an install tool needs to know what files are documentation, which are sample data, and what is part of the library proper, as well as what things are scripts (and in what language those scripts are written, e.g. Python, shell, batch, etc.). Some install tools need to know about icons, menus, registry info, cron jobs, etc. (These are perhaps more properly the domain of applications than libraries, but I'm going to assume that these things are in scope.)
The way that this information is communicated needs to be extensible, so that optional metadata for debs and msi's and rpm's and whathaveyou can be incorporated, without needing to modify the standard -- especially if the APIs for reading and writing this data are in the stdlib.
So basically with this system, this would mean that an ini-file in my package would be marked as a configuration file for the installer API, and that third party tools would decide what to do with this file. This would mean that my program would have to access to that file through an API as well to get back to the ini-file in the system. how would it work ?
I think I've mentioned all of these previously in the thread. I also think that as a matter of technicalities, these things are not difficult to achieve... but if it they are only achieved *technically*, then the result is a failure, not a success.
In order for the *real* goal to be achieved (i.e., a flourishing build/install system for Python), widespread participation and buy-in is required. If the OS people or the big package people (e.g. Zope Corp., Enthought) or the distutils aficionados are left out, then the result won't get used.
Well yes, that is basically what we are trying to build since a few days, but threads are not linear so people cant' keep up and jump in like that. So maybe you mentioned that idea before in the thread, but if we want people to buy the idea, it should be put in the wiki imho, even prematurely, and built it there, starting from a small set of points. I mean, you said earlier that it is premature to write a concrete document but it's hard to follow in threads the ideas proposed. That was the idea of my early proposal : start a synthetic list of problems and for each problem possible solutions. Then discuss them in the ML and make the right one raise. That is just my 2 cents on how the discussions go
I think the best way to ensure that nobody is left out, is to get them to participate in the design of a standard that ensures that *they* will be able to control their destiny, by creating their own build plugins and/or install tools... or at least having a robust choice of alternatives.
We need a consensus "de jure" standard (ala WSGI), rather than just an ongoing "de facto" standard (ala distutils/setuptools), or we're not making any substantial progress, just handing the reins over to somebody else.
+10
Enforcements: - a binary distribution cannot be uploaded if a source distrbution has not been previously provided for the version
Note that this doesn't allow closed-source packages to be uploaded; thus it would need to be a warning, rather than a requirement.
Right. do you agree it is something useful to do ?
Sure.
Ok so maybe this is *one* problem we can already solve at PyPI with a patch.
- PyPI should display the install and test dependencies in the UI
It could only do this for specific binaries, since dependencies can be dynamic.
What dynamic means here ? the python module to static file process or more ? can you provide an example ?
Dependencies can be platform-specific as well as python-version specific. If you want ctypes, you would depend on it in python 2.4 but not in 2.5. Similarly, if on some platform a certain library is required to implement a feature that is natively accessible on other platforms. In these cases, you would have logic in setup.py to detect this and choose the appropriate dependencies.
ok, right it is not possible with what we have now. If each dependency was marked as platform-specific or python-version specific in the metadata when it is necessary, we would know them all without calling extra detection code to build them. I hate the idea of dynamic metadata in fact. I can't express precisely why at that point.
-- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/
On Oct 1, 2008, at 19:10 PM, Tarek Ziadé wrote:
I hate the idea of dynamic metadata in fact. I can't express precisely why at that point.
Me too and me too. Perhaps it would help to distinguish between requiring a certain functionality and requiring a specific codebase which implements that functionality. For example: distribution A requires the functionality of ctypes. That part is statically, declaratively always true. However, distribution A doesn't necessarily require a *distribution* named "ctypes". If you are running on Python 2.6, then that functionality is already present. If there is a new distribution out there named "new_ctypes" which provides the same functionality and the same interface but is a completely different code base, then the presence of "new_ctypes" satisfies distribution A's requirements. The former question is simple, static, and declarative. The latter question isn't. In most cases there is only one implementation of a given interface, so we make do by equating the interface with the implementation. I wonder how Debian and Fedora handle this sort of issue? Regards, Zooko --- http://allmydata.org -- Tahoe, the Least-Authority Filesystem http://allmydata.com -- back up all your files for $5/month
zooko wrote:
On Oct 1, 2008, at 19:10 PM, Tarek Ziadé wrote:
I hate the idea of dynamic metadata in fact. I can't express precisely why at that point.
Me too and me too.
Perhaps it would help to distinguish between requiring a certain functionality and requiring a specific codebase which implements that functionality.
For example: distribution A requires the functionality of ctypes. That part is statically, declaratively always true.
However, distribution A doesn't necessarily require a *distribution* named "ctypes". If you are running on Python 2.6, then that functionality is already present. If there is a new distribution out there named "new_ctypes" which provides the same functionality and the same interface but is a completely different code base, then the presence of "new_ctypes" satisfies distribution A's requirements.
The former question is simple, static, and declarative. The latter question isn't.
In most cases there is only one implementation of a given interface, so we make do by equating the interface with the implementation.
I wonder how Debian and Fedora handle this sort of issue?
With python modules we just require one thing providing the interface. Let's say that elementtree was merged into python-2.5. And let's say that we got python-2.5 as the default python in Fedora 7. Since we only have one version of python in any release of Fedora we do something like this: Require: python %if 0%{?fedora} < 7 Require: python-elementtree %endif We are thinking of enhancing what dependency information we Require and Provide (the problem being... we want to do this automatically.) If we get that working, we could do things like: Require: python(elementtree) and in Fedora 6, python-elementtree would have: Provide: python(elementtree) whereas Fedora 7+, the python package would have: Provide: python(elementtree) Note that this information is not as easy to get to as the metadata provided by eggs so we're still trying to come up with a script that will generate this data automatically. -Toshio
On Sun, Oct 05, 2008 at 10:13:37AM -0600, zooko wrote:
On Oct 1, 2008, at 19:10 PM, Tarek Ziadé wrote: For example: distribution A requires the functionality of ctypes. That part is statically, declaratively always true.
However, distribution A doesn't necessarily require a *distribution* named "ctypes". If you are running on Python 2.6, then that functionality is already present. If there is a new distribution out there named "new_ctypes" which provides the same functionality and the same interface but is a completely different code base, then the presence of "new_ctypes" satisfies distribution A's requirements.
The former question is simple, static, and declarative. The latter question isn't.
In most cases there is only one implementation of a given interface, so we make do by equating the interface with the implementation.
I wonder how Debian and Fedora handle this sort of issue?
I think Josselin mentioned this already for Debian's case, but basically when python2.4 was default packages did say they wanted the python-elementtree package. So when python2.5 was packaged it said "Provides: python-elementree" and all packages depending on python-elementree will now be happy with just python2.5 installed. It's slightly more complicated then that in reality, the full Provides of python2.5 is: Provides: python2.5-celementtree, python2.5-cjkcodecs, python2.5-ctypes, python2.5-elementtree, python2.5-plistlib, python2.5-wsgiref The Debian infrastructure has various ways of helping packages cope with the versioned package names in the above (and it was all still very much a mess in python2.4 times). Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
participants (6)
-
Floris Bruynooghe -
Gael Varoquaux -
Phillip J. Eby -
Tarek Ziadé -
Toshio Kuratomi -
zooko