Pre-pre-PEP: Requirements for the Python BUILDS Specification
I'm thinking about putting together a pre-PEP for a "Build Utilities, Installation Locations, & Distribution Standards" (BUILDS) specification. But first, I want to throw out a few ideas to test the waters, and to give a general idea of what the first PEP would cover. The basic idea for the first PEP is to: 1. Give an overview of the current situation (problems w/distutils and setuptools, mainly, but also some of the successes) 2. Comment on some lessons learned from the WSGI definition process and WSGI in the field, and how they can be applied to the BUILDS process 3. Lay out high-level goals for the BUILDS project, such as: * distributing responsibility/authority for build tools development * adding extensibility to installation processes, * providing a 100% open playing field for build & install tools, * interoperability with the existing "egg" infrastructure, and * interoperability (but not 100% backward-compatibility) with distutils * allowing an incremental transition to the new standard 4. Lay out *non-goals* for the BUILDS project, such as trying to get developers to become system packagers or doing anything that requires them to change the runtime contents of their projects (as opposed to merely porting their setup.py, setup.cfg, etc.), defining and implementing the "perfect" system, etc. 5. Define rigorous terminology to be used for discussion of requirements and design, including such terms as "project", "release", "distribution", "system package", "installed distribution", etc. (This is incredibly important, because the discussions we're having now are already having Tower-of-Babel confusions.) 6. Sketch an overall design concept (build libraries in the stdlib establishing a Python API for invoking build tools, that in turn build an installation manifest to be used by installation tools), but without specifying actual APIs, manifest format, or a full enumeration of release metadata. 7. Present a vision/plan for how migration can occur from current tools. 8. Set the scope for the PEPs that should follow, on the installation manifest format, build architecture, build tool API, compiler/config infrastructure, etc. Whew. As you can see, just defining the problem adequately is a big job, and it may take a while to get even this first PEP right. So, I'd like some feedback, if anybody has some ideas about what else needs to be added to this. I also don't want to end up writing all of the PEPs, or else it may be a year or so before they all get done. ;-) I also think we're going to want a working group for this, or maybe multiple working groups, and it might be best not to use the general distutils-SIG for discussion past the first PEP, to allow people to filter threads better. Anyway. Thoughts, comments, volunteers, anyone?
Phillip,
You said it was too early, 2 days ago when I sent a similar (less detailed)
mail where I expressed the need to start writing things down in Python Wiki
.
But this work has already started, by Thoshio, me, and some others. mail.
http://wiki.python.org/moin/Distribute/Functionality for instance.
or here for the overview, http://wiki.python.org/moin/Distribute
we are also hanging in #distutils
Can't you join ? and help on it ?
We where waiting for that in fact..
Also , I am -1 on having another Mailing List for that. Distutils is the
place to be imho. Since main talks are about that.
And the working group is there : people that are talking for now a few
weeks, a bit before you joined the discussions.
Tarek
2008/10/3 Phillip J. Eby
I'm thinking about putting together a pre-PEP for a "Build Utilities, Installation Locations, & Distribution Standards" (BUILDS) specification. But first, I want to throw out a few ideas to test the waters, and to give a general idea of what the first PEP would cover.
The basic idea for the first PEP is to:
1. Give an overview of the current situation (problems w/distutils and setuptools, mainly, but also some of the successes)
2. Comment on some lessons learned from the WSGI definition process and WSGI in the field, and how they can be applied to the BUILDS process
3. Lay out high-level goals for the BUILDS project, such as:
* distributing responsibility/authority for build tools development * adding extensibility to installation processes, * providing a 100% open playing field for build & install tools, * interoperability with the existing "egg" infrastructure, and * interoperability (but not 100% backward-compatibility) with distutils * allowing an incremental transition to the new standard
4. Lay out *non-goals* for the BUILDS project, such as trying to get developers to become system packagers or doing anything that requires them to change the runtime contents of their projects (as opposed to merely porting their setup.py, setup.cfg, etc.), defining and implementing the "perfect" system, etc.
5. Define rigorous terminology to be used for discussion of requirements and design, including such terms as "project", "release", "distribution", "system package", "installed distribution", etc. (This is incredibly important, because the discussions we're having now are already having Tower-of-Babel confusions.)
6. Sketch an overall design concept (build libraries in the stdlib establishing a Python API for invoking build tools, that in turn build an installation manifest to be used by installation tools), but without specifying actual APIs, manifest format, or a full enumeration of release metadata.
7. Present a vision/plan for how migration can occur from current tools.
8. Set the scope for the PEPs that should follow, on the installation manifest format, build architecture, build tool API, compiler/config infrastructure, etc.
Whew. As you can see, just defining the problem adequately is a big job, and it may take a while to get even this first PEP right. So, I'd like some feedback, if anybody has some ideas about what else needs to be added to this.
I also don't want to end up writing all of the PEPs, or else it may be a year or so before they all get done. ;-) I also think we're going to want a working group for this, or maybe multiple working groups, and it might be best not to use the general distutils-SIG for discussion past the first PEP, to allow people to filter threads better.
Anyway. Thoughts, comments, volunteers, anyone?
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Tarek Ziadé - Directeur Technique INGENIWEB (TM) - SAS 50000 Euros - RC B 438 725 632 Bureaux de la Colline - 1 rue Royale - Bâtiment D - 9ème étage 92210 Saint Cloud - France Phone : 01.78.15.24.00 / Fax : 01 46 02 44 04 http://www.ingeniweb.com - une société du groupe Alter Way
On Fri, Oct 03, 2008 at 12:35:11PM -0400, Phillip J. Eby wrote:
I'm thinking about putting together a pre-PEP for a "Build Utilities, Installation Locations, & Distribution Standards" (BUILDS) specification.
Hehe, clever name!
The basic idea for the first PEP is to: [...] 2. Comment on some lessons learned from the WSGI definition process and WSGI in the field, and how they can be applied to the BUILDS process
That will be interesting to read. Not being a web developer I have no idea how WSGI came to be and what exactly it does. [...]
4. Lay out *non-goals* for the BUILDS project, [...] or doing anything that requires them to change the runtime contents of their projects (as opposed to merely porting their setup.py, setup.cfg, etc.)
Does this mean actively avoiding an API that would allow developers to access certain types of data files (I'm thinking of the discussion about locale data and not putting anything else but .py/.pyc/.pyo files in packages) or merely making sure the existing way (of shipping data files in packages and finding them by os.path and __file__) keeps working?
5. Define rigorous terminology to be used for discussion of requirements and design, including such terms as "project", "release", "distribution", "system package", "installed distribution", etc. (This is incredibly important, because the discussions we're having now are already having Tower-of-Babel confusions.)
Yay, please make sure the terminology finally allows us to talk about packages in the .deb and .rpm meaning too. The current distutils terminology doesn't and it really doesn't help.
I also think we're going to want a working group for this, or maybe multiple working groups, and it might be best not to use the general distutils-SIG for discussion past the first PEP, to allow people to filter threads better.
Hmm, this all seems quite on topic for distutils-sig. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
At 10:25 PM 10/6/2008 +0100, Floris Bruynooghe wrote:
On Fri, Oct 03, 2008 at 12:35:11PM -0400, Phillip J. Eby wrote:
I'm thinking about putting together a pre-PEP for a "Build Utilities, Installation Locations, & Distribution Standards" (BUILDS) specification.
Hehe, clever name!
The basic idea for the first PEP is to: [...] 2. Comment on some lessons learned from the WSGI definition process and WSGI in the field, and how they can be applied to the BUILDS process
That will be interesting to read. Not being a web developer I have no idea how WSGI came to be and what exactly it does.
WSGI came about because I was frustrated with the non-progress of the Web-SIG in creating standard request and response objects; it was quickly obvious to me that it would never go anywhere, due to socio/psych/economic considerations. I proposed it in order to break the logjam by having a standard that more or less allowed everybody to do whatever they wanted and still be able to co-operate... by arranging things so that the costs and benefits of adoption were balanced for all parties. In the case of BUILDS, I propose to do the same: define a standard whose cost/benefit ratios are ideally balanced for each participant. This does not, by the way, mean that everybody ends up with the same cost/benefit ratio; it simply means that the cost/benefit ratios are best for those people whose participation is most required for the standard to be widely adopted. You can see that this is also what I did in the design of easy_install and setuptools, except that in that effort I only considered developers and users, not system packagers. I propose to rectify that with BUILDS, although developers are still the most important audience a new standard must satisfy, in the sense that they require the best cost/benefit ratio for widespread adoption to occur. (And this is of course in system packagers' best interest, since widespread adoption of the standard should make their lives easier, as long as the standard provides an improvement over the status quo for them.) There are of course some other differences between this effort and those of WSGI and setuptools, but that's the 30-second summary. :)
[...]
4. Lay out *non-goals* for the BUILDS project, [...] or doing anything that requires them to change the runtime contents of their projects (as opposed to merely porting their setup.py, setup.cfg, etc.)
Does this mean actively avoiding an API that would allow developers to access certain types of data files (I'm thinking of the discussion about locale data and not putting anything else but .py/.pyc/.pyo files in packages) or merely making sure the existing way (of shipping data files in packages and finding them by os.path and __file__) keeps working?
I would actively avoid it for a "BUILDS 1.0" spec, because on any platform where the people building installation tools care about relocating these files, symlinks are available, so both sides can be happy without needing a new API. That is, unless I have misunderstood Josselin and Toshio, I understand symlinks to currently be an acceptable compromise. (For example, Debian uses them to relocate .pyc files currently.)
5. Define rigorous terminology to be used for discussion of requirements and design, including such terms as "project", "release", "distribution", "system package", "installed distribution", etc. (This is incredibly important, because the discussions we're having now are already having Tower-of-Babel confusions.)
Yay, please make sure the terminology finally allows us to talk about packages in the .deb and .rpm meaning too.
The term I use for such packages is "system packages", btw.
Le lundi 06 octobre 2008 à 21:33 -0400, Phillip J. Eby a écrit :
Does this mean actively avoiding an API that would allow developers to access certain types of data files (I'm thinking of the discussion about locale data and not putting anything else but .py/.pyc/.pyo files in packages) or merely making sure the existing way (of shipping data files in packages and finding them by os.path and __file__) keeps working?
I would actively avoid it for a "BUILDS 1.0" spec, because on any platform where the people building installation tools care about relocating these files, symlinks are available, so both sides can be happy without needing a new API.
I fail to see what this would bring over the current situation, then.
That is, unless I have misunderstood Josselin and Toshio, I understand symlinks to currently be an acceptable compromise. (For example, Debian uses them to relocate .pyc files currently.)
Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers don’t do it. Furthermore, you seem to be unaware of the amount of abuse that was produced by the mere idea of using __file__ to locate data. It is not only about loading a few image files, let me show you some cases. Case 1: pastescript. This module ships templates of python modules, that are named with the .py extension but are not usable as is from the interpreter, and shipped in the modules directory. How in the world can our tools know that they shouldn’t be byte-compiled and that they should be relocated to somewhere in /usr/share/$package? Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can’t understand how people can think of (and implement) such broken solutions. Case 3: GStreamer. The modules installation directory is detected at compile time and then hardcoded in the modules themselves so that some subdirectories can be added to sys.path upon loading. In the end, the modules are not relocatable. I keep wondering what Python module developers will invent next to complicate packaging even more. Frankly, we’d lose much less hair if such things were simply impossible. At least, it should be documented that such practice is wrong, so that we can actually consider it buggy and make our tools simpler. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.
On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette
Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can’t understand how people can think of (and implement) such broken solutions.
Hi Josselin, The facts you have presented above are not correct. Jean-Paul
On Tue, Oct 07, 2008 at 08:01:28AM -0400, Jean-Paul Calderone wrote:
On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette
wrote: Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can’t understand how people can think of (and implement) such broken solutions.
The facts you have presented above are not correct.
It would be helpful if you could correct them in that case. Cheers Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Tue, 7 Oct 2008 13:12:03 +0100, Floris Bruynooghe
On Tue, Oct 07, 2008 at 08:01:28AM -0400, Jean-Paul Calderone wrote:
On Tue, 07 Oct 2008 09:57:56 +0200, Josselin Mouette
wrote: Case 2: twisted. Plugins, consisting of a few .py files, are shipped in a plugins/ subdirectory, and its content is dynamically added to sys.path after some sanity checks (which fail when you add namespace packages). The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory. When there are solutions as simple as versioning the plugin directory, I can’t understand how people can think of (and implement) such broken solutions.
The facts you have presented above are not correct.
It would be helpful if you could correct them in that case.
I don't think the details of the plugin system are relevant to the topic under discussion here. The installation requirements are not unusual for the most part - that a directory full of .py files be copied to the install location, just as any package would be. The one unusual thing is that non-Twisted package X might want to copy files into twisted/plugins/ (if it is providing plugins). For details, see the Twisted plugin documentation: http://twistedmatrix.com/projects/core/documentation/howto/plugin.html Or this discussion in the Debian issue tracker: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=474630 I would be happy to answer any questions not addressed in the documentation, but perhaps off-list if they are not directly relevant to distribution topics. Jean-Paul
Le mardi 07 octobre 2008 à 08:58 -0400, Jean-Paul Calderone a écrit :
I don't think the details of the plugin system are relevant to the topic under discussion here. The installation requirements are not unusual for the most part - that a directory full of .py files be copied to the install location, just as any package would be. The one unusual thing is that non-Twisted package X might want to copy files into twisted/plugins/ (if it is providing plugins).
This is not unusual, but other frameworks that do the same don’t break when you add namespace packages. In fact, the very fact that you can’t add namespace packages shows that you are abusing the python modules directory: you are shipping a twisted/plugins/foo/foo.py file which is not meant to be available as a twisted.plugins.foo.foo module. The ability to install several incompatible versions of Twisted on the system is the reason I was given by glyph; if there is another reason, I’d be happy to hear the explanation, but the result is the same: we had to code a special case for Twisted. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.
On Tue, 07 Oct 2008 15:19:57 +0200, Josselin Mouette
Le mardi 07 octobre 2008 à 08:58 -0400, Jean-Paul Calderone a écrit :
I don't think the details of the plugin system are relevant to the topic under discussion here. The installation requirements are not unusual for the most part - that a directory full of .py files be copied to the install location, just as any package would be. The one unusual thing is that non-Twisted package X might want to copy files into twisted/plugins/ (if it is providing plugins).
This is not unusual, but other frameworks that do the same don’t break when you add namespace packages. In fact, the very fact that you can’t add namespace packages shows that you are abusing the python modules directory: you are shipping a twisted/plugins/foo/foo.py file which is not meant to be available as a twisted.plugins.foo.foo module.
I think you meant "shipping a twisted/plugins/foo.py file which is not meant to be available as a twisted.plugins.foo module", since as far as I know, no one is putting packages (as opposed to modules) into the twisted.plugins package (and I don't think it works, either).
The ability to install several incompatible versions of Twisted on the system is the reason I was given by glyph; if there is another reason, I’d be happy to hear the explanation, but the result is the same: we had to code a special case for Twisted.
Above correction aside, your understanding of the system is still flawed (I am fully aware that this is likely due to our failure to document the system sufficiently, so please don't interpret that as an attack). I think that you also misunderstood the explanation that Glyph provided. In your earlier message, you wrote this:
The reason is to allow plugins for several incompatible Twisted versions to be installed in the *same* directory.
However, no such feature is offered by the Twisted plugin system. :) The issue which arose when Debian packaged Twisted and (for example) a new version of Nevow which provides a plugin to Twisted is that Nevow's plugin was installed to a location which was not the location intended or expected by the Twisted developers or the Nevow developers. The expectation of the Twisted developers was that the installed directory structure for Twisted files would look like this: /usr/lib/python2.5/site-packages/twisted/ /usr/lib/python2.5/site-packages/twisted/__init__.py /usr/lib/python2.5/site-packages/twisted/plugins/ /usr/lib/python2.5/site-packages/twisted/plugins/__init__.py (obviously this is only part of what a Twisted installation would look like) The expectations of the Nevow developers was that a file included in Nevow, nevow_widget.py, would be installed to /usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py On Debian, difficulties were encountered because, while Twisted was installed more or less to the expected location, the file in Nevow was instead installed to /var/lib/python-support/python2.5/twisted/plugins/nevow_widget.py and the infrastructure for installing the file to this location also creates /var/lib/python-support/python2.5/twisted/ /var/lib/python-support/python2.5/twisted/__init__.py /var/lib/python-support/python2.5/twisted/plugins/ /var/lib/python-support/python2.5/twisted/plugins/__init__.py This is problematic because there are now two Twisted packages in the module search path. Python's module system can't do anything with the second one (which one is second depends on the order of sys.path). If the version created for nevow_widget.py came first, the Twisted package itself would basically be completely broken. Fortunately, it doesn't, so that doesn't happen. However, since nevow_widget.py is in the second version, the only way it can be imported is if some extra help is supplied to the import system. In previous releases of Twisted, this help was given by twisted/plugins/ __init__.py. It included an extremely primitive implementation of the concept of namespace packages (_extremely_ primitive). It would find all directories named "twisted/plugins" which were in the module search path and add them to its own __path__. The purpose of this feature wasn't to support installations, however. The purpose of this feature was as a developer convenience. It allows developers to edit plugins in a non- installed location (for example, their home directory) and then use them without copying them to the installed Twisted's twisted/plugins/ directory. In a more recent release of Twisted, this feature was tweaked slightly. The reason for the change is that there was a significant bug in the old implementation of namespace packages. If a developer was using their own copy of Twisted (for example, an SVN checkout in their home directory) instead of the installed version, the namespace package implementation would still load plugins from the *installed* version of Twisted. In the best possible case, this gave you two copies of every plugin. In the worst case, the installed version of Twisted wasn't compatible with the other version of Twisted and the plugins would raise exceptions. This is the failure condition which I believe you misinterpreted as meaning we were trying to support any kind of versioning features. We're not. We're only trying to not run code that the user (/developer) doesn't want run. So, to fix this, we tweaked the namespace package setup code so that it will skip any "twisted/plugins/" directory which contains an "__init__.py", since that is a relatively strong hint that it's a real package, not one of the developer-convenience plugin locations (which are not real packages). Debian's Nevow package then broke, since Debian installed an __init__.py when installing Nevow's nevow_widget.py file. The ideal fix, from my perspective as a Twisted developer, is to install the nevow_widget.py file into /usr/lib/python2.5/site-packages/twisted/plugins/ along with all the other Twisted plugin files (perhaps using a symlink, I can't think of any reason why that would make a difference, particularly since all the other .py files in that area are symlinks). In other words, Debian took advantage of a feature which was intended to be used only as a convenience for development and was never intended to assist installation. Have I succeeded in explaining why the Twisted plugin system isn't making any unusual requirements of the installation system nor using the Python module system in an unusual way? Can I clarify any parts of this further? Can I expect to see any improvements to the Debian packaging of Twisted and projects which supply plugins to Twisted? :) Jean-Paul
Le mardi 07 octobre 2008 à 10:02 -0400, Jean-Paul Calderone a écrit :
The expectations of the Nevow developers was that a file included in Nevow, nevow_widget.py, would be installed to
/usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py
This expectation is wrong. You’re shipping it as a Python module, the only thing you can expect is to be able to import it. This wouldn’t happen if you were shipping plugins in a specific, private, plugins directory.
The ideal fix, from my perspective as a Twisted developer, is to install the nevow_widget.py file into
/usr/lib/python2.5/site-packages/twisted/plugins/
As explained on the debian-python list (see http://lists.debian.org/debian-python/2008/05/msg00032.html), there are strong reasons for not putting managed modules in the same directory as modules included in the Python distribution. So you can’t expect modules to be in a specific directory (but this shouldn’t be a problem since these are not modules, hmmm?). (I have some ideas for keeping all files at the same place and moving only .pyc files to /var, but it requires patching the interpreter and the Debian Python maintainer disagrees, so we’ll have to stick to the current situation for now.)
along with all the other Twisted plugin files (perhaps using a symlink, I can't think of any reason why that would make a difference, particularly since all the other .py files in that area are symlinks).
In other words, Debian took advantage of a feature which was intended to be used only as a convenience for development and was never intended to assist installation.
The fact that we use a .pth to specify the different module hierarchy is an implementation detail but is indeed abusing this feature; we should be shipping a modified setup.py instead. However this has nothing to do with the inability of Twisted to cope with multiple module paths. Let’s take another example that is not Debian specific: what if I want to install a plugin to /usr/local? The fact that you are abusing the python modules directory forces me to install it to /usr/local/python2.5/site-packages/twisted/plugins instead of e.g. /usr/local/share/twisted/plugins, where a normal application or library would be looking for its plugins. And there, I have to deal with namespaces and other python module specificities that should be irrelevant for plugins. In the beginning, Debian only expected the Python modules directory to contain, well, Python modules. It seems even that is too much to expect, since module developers are taking advantage of a set of features (module file location) that were initially meant for introspection.
Have I succeeded in explaining why the Twisted plugin system isn't making any unusual requirements of the installation system nor using the Python module system in an unusual way?
Not really. You have succeeded in explaining what is the exact nature of the unusual requirements you make, though. Thanks for that, as it is much clearer than the original explanations I received.
Can I expect to see any improvements to the Debian packaging of Twisted and projects which supply plugins to Twisted? :)
Until you fix your plugin system, we are bound to treat Twisted packages as a special case. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.
On Tue, 07 Oct 2008 17:11:19 +0200, Josselin Mouette
Le mardi 07 octobre 2008 à 10:02 -0400, Jean-Paul Calderone a écrit :
The expectations of the Nevow developers was that a file included in Nevow, nevow_widget.py, would be installed to
/usr/lib/python2.5/site-packages/twisted/plugins/nevow_widget.py
This expectation is wrong. You’re shipping it as a Python module, the only thing you can expect is to be able to import it.
Leaving aside whether or not I think that is a reasonable restriction to impose, I don't see the relevance. The only reason the module can be imported on Debian now is because of the special measures included (by upstream developers, not by Debian) in twisted/plugins/__init__.py. I can't understand how this could be part of Debian's Python packaging policy, so perhaps I'm missing something very fundamental.
[snip - "specific, private, plugins directory" which I may not understand, but doubt is relevant]
The ideal fix, from my perspective as a Twisted developer, is to install the nevow_widget.py file into
/usr/lib/python2.5/site-packages/twisted/plugins/
As explained on the debian-python list (see http://lists.debian.org/debian-python/2008/05/msg00032.html), there are strong reasons for not putting managed modules in the same directory as modules included in the Python distribution. So you can’t expect modules to be in a specific directory (but this shouldn’t be a problem since these are not modules, hmmm?).
They're modules. They get imported. That makes them modules, right? exarkun@charm:~$ python Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import twisted.plugins.nevow_widget >>> Do we have the same definition of "module"? I chose the phrase "The ideal fix, from my perspective as a Twisted developer" very intentionally. If the file ends up somewhere else but it's still an importable module, fine - if, from a Debian packager's perspective that's better, fine. I don't believe that the current situation is better from a Debian packager's perspective, though, since it involves carrying a patch that reverts an upstream bugfix.
[snip - .pyc files in /var - not directly relevant] [snip - use of .pth file - not directly relevant]
However this has nothing to do with the inability of Twisted to cope with multiple module paths. Let’s take another example that is not Debian specific: what if I want to install a plugin to /usr/local? The fact that you are abusing the python modules directory forces me to ^^^^^^^^^^^^ install it to /usr/local/python2.5/site-packages/twisted/plugins instead ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ of e.g. /usr/local/share/twisted/plugins, where a normal application or ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ library would be looking for its plugins. And there, I have to deal with namespaces and other python module specificities that should be irrelevant for plugins.
No, this is wrong. Install it to /usr/local/share/ if you want. It will work just fine. Why do you think it won't? I'd rather you _didn't_, at least not without a couple other changes, and I don't understand why you would _want_ to, but it's possible.
In the beginning, Debian only expected the Python modules directory to contain, well, Python modules. It seems even that is too much to expect, since module developers are taking advantage of a set of features (module file location) that were initially meant for introspection.
I assume you're talking about the "Python modules directory" in the install target (not, say, in an upstream VCS). Twisted plugins are modules, so Debian's expectations (be they valid or otherwise, I make no judgement) aren't violated by them (oops, except for dropin.cache files, but we haven't really talked about those yet and I don't think they're actually contentious).
Have I succeeded in explaining why the Twisted plugin system isn't making any unusual requirements of the installation system nor using the Python module system in an unusual way?
Not really. You have succeeded in explaining what is the exact nature of the unusual requirements you make, though. Thanks for that, as it is much clearer than the original explanations I received.
I'm glad this was clearer. I'm not sure I've yet succeeded in entirely explaining the system. I hope this follow up goes a little bit further toward that goal.
[snip - I can't expect to see any improvements yet]
Jean-Paul
Le mardi 07 octobre 2008 à 11:44 -0400, Jean-Paul Calderone a écrit :
They're modules. They get imported. That makes them modules, right?
Fine. I hadn’t understood that they are actually meant to be imported as is (which makes you right in your assertions that the explanations weren’t completely clear to me yet). If they are modules, meant to be imported, they actually have their place in the modules directory, so please disregard what I’ve said about private directories. So, if they are modules, they sometimes need namespace packages to be imported properly. However you intentionally do not import modules that have associated namespace packages, which is the exact opposite of what we expect for other modules. Let me comment on your previous email with that in mind. In previous releases of Twisted, this help was given by twisted/plugins/ __init__.py. It included an extremely primitive implementation of the concept of namespace packages (_extremely_ primitive). It would find all directories named "twisted/plugins" which were in the module search path and add them to its own __path__. The purpose of this feature wasn't to support installations, however. The purpose of this feature was as a developer convenience. It allows developers to edit plugins in a non- installed location (for example, their home directory) and then use them without copying them to the installed Twisted's twisted/plugins/ directory. OK, I get that now. You need to do that because importing plugins.something in the twisted/ directory will only look for subdirectories of the same installation path. In a more recent release of Twisted, this feature was tweaked slightly. The reason for the change is that there was a significant bug in the old implementation of namespace packages. If a developer was using their own copy of Twisted (for example, an SVN checkout in their home directory) instead of the installed version, the namespace package implementation would still load plugins from the *installed* version of Twisted. In the best possible case, this gave you two copies of every plugin. In the worst case, the installed version of Twisted wasn't compatible with the other version of Twisted and the plugins would raise exceptions. About this issue, I still think you need to version your plugins directory if the plugins are not compatible across versions. Otherwise - and this is something that will hit distributions hard - you may install a an incompatible version of, say, Nevow, and get your installation fucked up. However, in the end, I think you’ve been hit by an issue that was only sporadically met until now: managing files belonging to the same module hierarchy with two different tools doesn’t always work. And this is something that we need to manage inside Debian (which is more a political problem than a technical one).
Do we have the same definition of "module"?
I think so.
I chose the phrase "The ideal fix, from my perspective as a Twisted developer" very intentionally. If the file ends up somewhere else but it's still an importable module, fine - if, from a Debian packager's perspective that's better, fine. I don't believe that the current situation is better from a Debian packager's perspective, though, since it involves carrying a patch that reverts an upstream bugfix.
In the end, I don’t think this patch was added as there is another workaround. Either way, it may break again if you change the way you look for plugins, so you are right to expect that they land in the same directory. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.
Josselin Mouette
About this issue, I still think you need to version your plugins directory if the plugins are not compatible across versions.
You've referred to this a few times now without explaining exactly what you expect the developer to do. So, just in case: I presume you mean that, in the case where a program's plugins are not compatible with plugins for previous versions of the same program, the developer should always seek (and attempt to install) plugins for version N in a differently-named directory from plugins for version M and version P. That is, that a directory expected to contain plugins for version N should contain *only* plugins for version N, and never for other versions. Yes? -- \ “Pinky, are you pondering what I'm pondering?” “I think so, | `\ Brain, but isn't that why they invented tube socks?” —_Pinky | _o__) and The Brain_ | Ben Finney
Le mercredi 08 octobre 2008 à 08:44 +1100, Ben Finney a écrit :
Josselin Mouette
writes: About this issue, I still think you need to version your plugins directory if the plugins are not compatible across versions.
You've referred to this a few times now without explaining exactly what you expect the developer to do. So, just in case:
I presume you mean that, in the case where a program's plugins are not compatible with plugins for previous versions of the same program, the developer should always seek (and attempt to install) plugins for version N in a differently-named directory from plugins for version M and version P.
That is, that a directory expected to contain plugins for version N should contain *only* plugins for version N, and never for other versions.
Yes. Sorry for omitting this detailed explanation. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.
On Wed, 08 Oct 2008 00:03:16 +0200, Josselin Mouette
Le mercredi 08 octobre 2008 à 08:44 +1100, Ben Finney a écrit :
Josselin Mouette
writes: About this issue, I still think you need to version your plugins directory if the plugins are not compatible across versions.
You've referred to this a few times now without explaining exactly what you expect the developer to do. So, just in case:
I presume you mean that, in the case where a program's plugins are not compatible with plugins for previous versions of the same program, the developer should always seek (and attempt to install) plugins for version N in a differently-named directory from plugins for version M and version P.
That is, that a directory expected to contain plugins for version N should contain *only* plugins for version N, and never for other versions.
Yes. Sorry for omitting this detailed explanation.
What if we consider the scenario without two different versions? The plugin duplication issue happens if you have a site-wide installation of Twisted X.Y and an extra (eg, per-user) copy of Twisted X.Y, the same version, explicitly in PYTHONPATH. Versioning the plugin directory wouldn't help here, because both copies would appear to be the right version for whichever instance of the install gets used. It might be possible to avoid duplication by inspecting the names carefully and avoiding loading anything that looks like it was already loaded. But I can still think of problem cases - the site-wide installation has plugins X, Y, and Z and my personal installation has plugins A, B, and C, and I use my personal installation with the express intent of not having X, Y, and Z made available. If the site-wide plugins directory is considered, then I get all 6 - not what I wanted. Jean-Paul
At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote:
Le lundi 06 octobre 2008 à 21:33 -0400, Phillip J. Eby a écrit :
Does this mean actively avoiding an API that would allow developers to access certain types of data files (I'm thinking of the discussion about locale data and not putting anything else but .py/.pyc/.pyo files in packages) or merely making sure the existing way (of shipping data files in packages and finding them by os.path and __file__) keeps working?
I would actively avoid it for a "BUILDS 1.0" spec, because on any platform where the people building installation tools care about relocating these files, symlinks are available, so both sides can be happy without needing a new API.
I fail to see what this would bring over the current situation, then.
That is, unless I have misunderstood Josselin and Toshio, I understand symlinks to currently be an acceptable compromise. (For example, Debian uses them to relocate .pyc files currently.)
Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers donât do it.
Which is why the idea for the BUILDS spec to include a way for automated tools to do it, so that you won't have to do it manually. Or are you saying that that isn't an improvement over the current situation? Yes, it's true that I'm saying that developers should not be *required* to add the extra data to their packages, but that they should be *able* to, and if it is trivial to add the extra data, most should accept patches or respond to requests to do so. Right now, it's not even *possible* for them to do so, however. Also, I wasn't aware of those peculiarities regarding Twisted and GStreamer -- neither one uses setuptools. But certainly they should make interesting test cases for any proposed standard, wrt to finding ways to satisfy both parties. Please understand that I'm not saying we must be 100% backward compatible with 100% of the packages; we just need to be compatible enough with enough of the packages for network effects to drive the adoption of the rest. At the same time, the people I'd most like to see on the PEP team from the developer-user side would definitely include folks from Twisted, Numpy, Enthought, and Zope, as they are the folks who have most stressed the distutils to the limits and beyond. Just getting you and them in the "same room" so to speak seems to already be producing some benefits.
Le mardi 07 octobre 2008 à 14:40 -0400, Phillip J. Eby a écrit :
At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote:
Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers don’t do it.
Which is why the idea for the BUILDS spec to include a way for automated tools to do it, so that you won't have to do it manually.
If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way.
Or are you saying that that isn't an improvement over the current situation?
It is. I didn’t understand you wanted to automate the symlinks creation. It indeed means less burden on the maintainers, but it would be a shame to keep the same mess in binary packages, while having a tool to do that would allow to make things cleaner.
Yes, it's true that I'm saying that developers should not be *required* to add the extra data to their packages, but that they should be *able* to, and if it is trivial to add the extra data, most should accept patches or respond to requests to do so.
Right now, it's not even *possible* for them to do so, however.
It is possible - if they use autoconf instead. BTW, I would consider it a good approach to try bringing BUILDS on par with the autotools capabilities. These tools have serious drawbacks, but they were written with the ability to work for distributors in mind.
At the same time, the people I'd most like to see on the PEP team from the developer-user side would definitely include folks from Twisted, Numpy, Enthought, and Zope, as they are the folks who have most stressed the distutils to the limits and beyond. Just getting you and them in the "same room" so to speak seems to already be producing some benefits.
Looks like a sane approach. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.
At 08:58 PM 10/7/2008 +0200, Josselin Mouette wrote:
Le mardi 07 octobre 2008 à 14:40 -0400, Phillip J. Eby a écrit :
At 09:57 AM 10/7/2008 +0200, Josselin Mouette wrote:
Symlinks are a real pain to handle. We can use them transparently for .pyc files, but if we want to relocate data files to some other directories, currently it has to be done by hand, and this is why most maintainers donât do it. .
Which is why the idea for the BUILDS spec to include a way for automated tools to do it, so that you won't have to do it manually.
If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way.
I'm not sure I follow you. What I'm proposing is that: 1. A BUILDS "manifest" for a project should list the files included in the project by what "kind" of file they are, as well as by inter-file relationships (e.g., "these 3 files are part of Python package 'foo.bar' within this project, and they are static data used at runtime but not read or written"). 2. BUILDS-compatible install tools will be free to install those files wherever they like (or create .debs, .rpms, etc. that do), and use symlinks so that the installed project can find them in the places that the developer wanted to find them. Is this what you're asking for? Because it's what I've been trying to say we should do.
Or are you saying that that isn't an improvement over the current situation?
It is. I didnât understand you wanted to automate the symlinks creation. It indeed means less burden on the maintainers, but it would be a shame to keep the same mess in binary packages, while having a tool to do that would allow to make things cleaner.
I'm not sure I understand you here, either. I'm saying that a BUILDS-compatible install tools for Debian and RPM systems should create system packages with the stuff wherever the OS people want it to go, *and* symlinks so the Python code can find it. And that this should be an automated process, to the extent possible/practical. For example, if Debian needs metadata that's not in the core BUILDS metadata, it should be possible within BUILDS to include it, so that it can be contributed upstream. Likewise, it should be possible for a BUILDS install tool to use a third-party supplement to get that data, so that when you're initially porting a project, you could write a file with the additional data while doing the initial port, then submit that file upstream.
Yes, it's true that I'm saying that developers should not be *required* to add the extra data to their packages, but that they should be *able* to, and if it is trivial to add the extra data, most should accept patches or respond to requests to do so.
Right now, it's not even *possible* for them to do so, however.
It is possible - if they use autoconf instead.
Sure, but who's going to convince them to do *that*? ;-) Less flippantly, our goal here is to have something that has a better cost/benefit ratio for adoption by a Python developer who has not seen any pressing need for autoconf in their life to date. And if somebody wants to make a BUILDS tool to generate autoconf sources from a project manifest, then great!
BTW, I would consider it a good approach to try bringing BUILDS on par with the autotools capabilities. These tools have serious drawbacks, but they were written with the ability to work for distributors in mind.
Sure. I like stealing designs and requirements definition from mature projects, even if you might argue that I chose poor role models for setuptools. (Java and OSGi.) However, since autotools compatibility isn't a goal in itself, I think I'll leave it to those people who have interest/experience to propose what specific autotools features they'd like to see in BUILDS, and the use cases they have for them. If I were to just skim the docs trying to figure that out, I might think that something cool in it is actually useful, or vice versa. ;-) So, it's better here I think to hear from experienced users such as yourself.
Le mardi 07 octobre 2008 à 16:51 -0400, Phillip J. Eby a écrit :
If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way.
I'm not sure I follow you. What I'm proposing is that:
1. A BUILDS "manifest" for a project should list the files included in the project by what "kind" of file they are, as well as by inter-file relationships (e.g., "these 3 files are part of Python package 'foo.bar' within this project, and they are static data used at runtime but not read or written").
2. BUILDS-compatible install tools will be free to install those files wherever they like (or create .debs, .rpms, etc. that do), and use symlinks so that the installed project can find them in the places that the developer wanted to find them.
Is this what you're asking for? Because it's what I've been trying to say we should do.
What I am afraid of is that, by adding just another layer, you will introduce new problems while fixing existing ones. Currently we already have a hard time maintaining a symlink farm, and adding a second symlink farm on top of it is not going to make things more understandable (currently, package maintainers already have a hard time understanding how to deal with Python modules) nor more reliable. The first kind of issues I can think of is that dpkg handles symlinks to directories very badly, so you’re likely to run into issues that have to be dealt with by hand by maintainers. I haven’t thought about it much, but I have the feeling that other things will break. In the end, if you are designing a new packaging specification and a new tool, I think you’d better take this as an occasion to do the right thing instead of adding this new layer.
For example, if Debian needs metadata that's not in the core BUILDS metadata, it should be possible within BUILDS to include it, so that it can be contributed upstream. Likewise, it should be possible for a BUILDS install tool to use a third-party supplement to get that data, so that when you're initially porting a project, you could write a file with the additional data while doing the initial port, then submit that file upstream.
If there is a need for Debian-specific metadata, it will go in the debian/ directory. It looks interesting to integrate it upstream only if it is relevant for other distributors.
Less flippantly, our goal here is to have something that has a better cost/benefit ratio for adoption by a Python developer who has not seen any pressing need for autoconf in their life to date. And if somebody wants to make a BUILDS tool to generate autoconf sources from a project manifest, then great!
No, no. Please don’t :) But if you can express in the project manifest the same flexibility as with autoconf (or even the tenth of it, we don’t need *that* much flexibility), that should be enough.
BTW, I would consider it a good approach to try bringing BUILDS on par with the autotools capabilities. These tools have serious drawbacks, but they were written with the ability to work for distributors in mind.
Sure. I like stealing designs and requirements definition from mature projects, even if you might argue that I chose poor role models for setuptools. (Java and OSGi.)
However, since autotools compatibility isn't a goal in itself, I think I'll leave it to those people who have interest/experience to propose what specific autotools features they'd like to see in BUILDS, and the use cases they have for them. If I were to just skim the docs trying to figure that out, I might think that something cool in it is actually useful, or vice versa. ;-) So, it's better here I think to hear from experienced users such as yourself.
There is no use in being compatible with the autotools, the point is about stealing good ideas. The first thing that comes to mind, and that we already discussed at large, is the ability to specify at install time where data, documentation, manual pages and so on have to go. Another thing that I consider a good design (when expunged of the useless stuff in it) is the config.h file. It is a file that is generated at configure time and which contains macros you may need inside your code: package version, installation directories, enabled features, etc. What some Python projects using autoconf are doing, and I’d like to see in BUILDS, is the generation of a config.py file at installation time. It would be useful to specify install-time stuff like installation directories, optional features and installation parameters. For example it could contain: version = "1.2.3" builddate = "Tue 7 Oct 2008, 23:12 +0100" templates_dir = "/usr/share/foobar/templates" has_libamazing = True # _foobar.so is built with libamazing support page_footer = "Powered by foobar" For Python-only packages, you can easily ship a default config.py file that works without installation: version = get_version_from_changelog () templates_dir = os.path.join(os.path.basedir(__file__),"templates") page_footer = "Foobar (development version)" By doing such a thing, you are guaranteed to obtain easily the data you need from within the Python application: import config template = file(os.path.join(config.templates_dir), "foo.template") I may be biased because I’m used to work like this, but I find this easier for both the developer and the distributor. In my personal experience, it is also more reliable. Another thing from the autotools that I’d like to see is seamless integration with pkg-config, for modules with C extensions. Bonus points for being able to use not only compilation flags and installation paths, but also macros you need to use inside the code (they could go in config.py). I’ll tell you if I can think of other things. Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.
At 12:43 AM 10/8/2008 +0200, Josselin Mouette wrote:
Le mardi 07 octobre 2008 à 16:51 -0400, Phillip J. Eby a écrit :
If you write a tool to do that, why not make it simply move files properly and let the code locate them, instead of adding yet another layer on top of the existing stuff? The tool will not be more complicated this way.
I'm not sure I follow you. What I'm proposing is that:
1. A BUILDS "manifest" for a project should list the files included in the project by what "kind" of file they are, as well as by inter-file relationships (e.g., "these 3 files are part of Python package 'foo.bar' within this project, and they are static data used at runtime but not read or written").
2. BUILDS-compatible install tools will be free to install those files wherever they like (or create .debs, .rpms, etc. that do), and use symlinks so that the installed project can find them in the places that the developer wanted to find them.
Is this what you're asking for? Because it's what I've been trying to say we should do.
What I am afraid of is that, by adding just another layer, you will introduce new problems while fixing existing ones. Currently we already have a hard time maintaining a symlink farm, and adding a second symlink farm on top of it is not going to make things more understandable (currently, package maintainers already have a hard time understanding how to deal with Python modules) nor more reliable.
I guess I'm confused a bit here, since the idea is not to have maintainers need to understand anything but how to be able to tell upstream, "hey, you didn't configure file X correctly". At least, that's what I'd hope would be all that's needed. :-)
The first kind of issues I can think of is that dpkg handles symlinks to directories very badly, so youâre likely to run into issues that have to be dealt with by hand by maintainers. I havenât thought about it much, but I have the feeling that other things will break.
Actually, until I saw the Twisted plugins directory use case, I hadn't actually seen any use cases for symlinks to directories that couldn't be handled by using a directory of symlinks instead. Are there any other Python packages you know of that would need a symlink to a directory?
In the end, if you are designing a new packaging specification and a new tool,
The BUILDS manifest is neither packaging nor a tool; it's just information about a bunch of files. ;-) We don't want to design *a* tool, we want to make it possible for tools to be written. Currently, there are a whole bunch of installation and maintenance tools out there built on top of setuptools, but they aren't necessarily very compatible with each other, and building tools that approach things very differently from setuptools is hard. The idea of a specification is to allow installation tools to be built that apply different policies on different platforms, not to build The One True Tool[tm]. In other words, the goal isn't to "do the right thing", but to allow various useful things to be done.
For example, if Debian needs metadata that's not in the core BUILDS metadata, it should be possible within BUILDS to include it, so that it can be contributed upstream. Likewise, it should be possible for a BUILDS install tool to use a third-party supplement to get that data, so that when you're initially porting a project, you could write a file with the additional data while doing the initial port, then submit that file upstream.
If there is a need for Debian-specific metadata, it will go in the debian/ directory. It looks interesting to integrate it upstream only if it is relevant for other distributors.
If I have to explicitly mark my locale files for Debian to be happy, then that *is* Debian-specific metadata from my perspective as a cross-platform developer. That is, I didn't say anything about metadata that was unique to Debian, just metadata needed by a Debian BUILDS installer to be happy, as opposed to a generic BUILDS installer that doesn't care about the FHS. (For example, an Enthought installer for Windows that's also installing its own Python interpreter might want to consume BUILDS but it sure as heck wouldn't be following FHS standards; it'd have its own.)
There is no use in being compatible with the autotools, the point is about stealing good ideas.
The first thing that comes to mind, and that we already discussed at large, is the ability to specify at install time where data, documentation, manual pages and so on have to go.
The idea of the BUILDS "install locations" (IL) manifest is just that it lists the contents of the project distribution, and describes the logical locations that things go in. So, part of the BUILDS spec would be what categories exist for the above. It would then be the responsibility of people building BUILDS-consuming install tools to define the actual locations. I say it that way mainly because I'm not volunteering my time to actually build such an install tool, nor am I necessarily volunteering to build *any* tools. No sense in replacing me as the setuptools bottleneck to be the bottleneck on new tools. The whole idea is to get system packagers and major developers together to agree on how to communicate installation requirements to installation tools, and then have everybody go back and update their tools to generate or consume installation data in accordance with the spec. Of course, it's likely that rather than everybody creating their own install tools, there will probably be some grouping and alliances along platform and/or use-case lines. For the stdlib, I expect there will be a simple installer with configurable locations for installing things, maybe with some ability to be extended to handle more complex policies. There would probably also need to be BUILDS-to-wininst, and BUILDS-to-msi converters.
What some Python projects using autoconf are doing, and Iâd like to see in BUILDS, is the generation of a config.py file at installation time. It would be useful to specify install-time stuff like installation directories, optional features and installation parameters.
For example it could contain: version = "1.2.3" builddate = "Tue 7 Oct 2008, 23:12 +0100" templates_dir = "/usr/share/foobar/templates" has_libamazing = True # _foobar.so is built with libamazing support page_footer = "Powered by foobar"
For Python-only packages, you can easily ship a default config.py file that works without installation: version = get_version_from_changelog () templates_dir = os.path.join(os.path.basedir(__file__),"templates") page_footer = "Foobar (development version)"
By doing such a thing, you are guaranteed to obtain easily the data you need from within the Python application: import config template = file(os.path.join(config.templates_dir), "foo.template")
I may be biased because Iâm used to work like this, but I find this easier for both the developer and the distributor. In my personal experience, it is also more reliable.
It looks like a great idea, and I'm all for recommending people use it, or something like it. I'm also fine with BUILDS having support for some sort of "install-time editing" like this. I just don't want it to be a requirement that everybody change their ways. Not everybody even uses pkg_resources' API for accessing data files yet, after all. In my experience with setuptools adoption, anything that makes you have to change something outside of the packaging part is a significant barrier. People wonder if they can rely on it, whether it will lock them in to something, they need to do testing, etc. Sure, these things aren't REALLY that big of a hassle, but the psychological effect is to create hesitation. But if you can move your code without changing your code, it creates a sense of safety, and provides incremental rewards for incremental efforts. Thus, people who first moved to setuptools without needing to change their actual package code, were then much more likely to adopt the API later. The same will be true for this.
Another thing from the autotools that Iâd like to see is seamless integration with pkg-config, for modules with C extensions. Bonus points for being able to use not only compilation flags and installation paths, but also macros you need to use inside the code (they could go in config.py).
Sorry, what is pkg-config?
Le mardi 07 octobre 2008 à 19:25 -0400, Phillip J. Eby a écrit :
At 12:43 AM 10/8/2008 +0200, Josselin Mouette wrote:
What I am afraid of is that, by adding just another layer, you will introduce new problems while fixing existing ones. Currently we already have a hard time maintaining a symlink farm, and adding a second symlink farm on top of it is not going to make things more understandable (currently, package maintainers already have a hard time understanding how to deal with Python modules) nor more reliable.
I guess I'm confused a bit here, since the idea is not to have maintainers need to understand anything but how to be able to tell upstream, "hey, you didn't configure file X correctly". At least, that's what I'd hope would be all that's needed. :-)
First, it is wrong to expect maintainers to not understand what they do. It is a guaranteed recipe for failure. Second, it is not realistic to assume that all the maintainer will do is forward bugs to the upstream developers, and that upstream developers will fix it in the minute – especially for packaging issues, which developers generally disregard. The maintainer often needs to patch the sources or the installation scripts so that the software installs properly, and to do so he needs to understand what he does.
The first kind of issues I can think of is that dpkg handles symlinks to directories very badly, so you’re likely to run into issues that have to be dealt with by hand by maintainers. I haven’t thought about it much, but I have the feeling that other things will break.
Actually, until I saw the Twisted plugins directory use case, I hadn't actually seen any use cases for symlinks to directories that couldn't be handled by using a directory of symlinks instead. Are there any other Python packages you know of that would need a symlink to a directory?
Yes, you can arrange for not meeting this specific issue. What I’m saying is not that such issues don’t have workarounds, I’m saying that we will discover new ones, again and again, as long as the only thing we do to fix each issue is add another carpet to hide the dust.
The idea of the BUILDS "install locations" (IL) manifest is just that it lists the contents of the project distribution, and describes the logical locations that things go in. So, part of the BUILDS spec would be what categories exist for the above.
If the only thing it does is allowing to move files and symlink them, I frankly doubt the benefits will outperform the cost of maintaining this new symlink farm.
What some Python projects using autoconf are doing, and I’d like to see in BUILDS, is the generation of a config.py file at installation time. It would be useful to specify install-time stuff like installation directories, optional features and installation parameters.
It looks like a great idea, and I'm all for recommending people use it, or something like it. I'm also fine with BUILDS having support for some sort of "install-time editing" like this. I just don't want it to be a requirement that everybody change their ways. Not everybody even uses pkg_resources' API for accessing data files yet, after all.
You don’t have to make it a requirement. If you agree with the idea of such a feature, a good approach might be to: * always generate this file; * provide the possibility to easily install it; * let developers gradually start using it directly in the code.
Another thing from the autotools that I’d like to see is seamless integration with pkg-config, for modules with C extensions. Bonus points for being able to use not only compilation flags and installation paths, but also macros you need to use inside the code (they could go in config.py).
Sorry, what is pkg-config?
pkg-config is becoming the de-facto standard for providing information about installed development libraries. With it you can easily: * obtain compile-time parameters for dependent programs (CFLAGS and LIBS); * require a range of versions for a number of libraries you depend on; * obtain any parameter that the developer wanted you to know (for example, the name of a dbus service you need to start, the path to a helper utility, or the installation path for plugins); * with the automake integration, enable programs or features based on the availability of dependencies. For a Python example: you want to know where to install your definition files for GNOME Python bindings? Just use: pkg-config --variable=codegendir pygtk-2.0 Cheers, -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `- our own. Resistance is futile.
2008/10/7 Phillip J. Eby
In the case of BUILDS, I propose to do the same: define a standard whose cost/benefit ratios are ideally balanced for each participant. This does not, by the way, mean that everybody ends up with the same cost/benefit ratio; it simply means that the cost/benefit ratios are best for those people whose participation is most required for the standard to be widely adopted.
You can see that this is also what I did in the design of easy_install and setuptools, except that in that effort I only considered developers and users, not system packagers.
I'd argue (you may differ) that the most significant area where you missed the mark on user benefits with easy_install and setuptools is the lack of easy *uninstall* and easy *list* options. Most of the issues I hear from users about setuptools (filtered by my prejudices, admittedly) is that there's no management options (which brings in the system packagers, and their concerns). Can I suggest that this be included somehow in the new spec, so that metadata is available to make wtiting uninstallers and listers as easy as writing installers? Paul.
participants (8)
-
Ben Finney
-
David Cournapeau
-
Floris Bruynooghe
-
Jean-Paul Calderone
-
Josselin Mouette
-
Paul Moore
-
Phillip J. Eby
-
Tarek Ziade