splitting out bdist_* (was: interminable 'setuptools' thread)
On 07:59 pm, fdrake@acm.org wrote:
I'm actually in favor of removing the bdist_* from the standard library, and allowing 3rd-party tools to implement whatever they need for the distros. But I don't think what you're presenting there supports it.
I do think that it's relevant that the respective operating system packagers don't find bdist_rpm, bdist_deb, et. al. useful. It's not very useful to have a bdist_deb that nobody actually builds debs with. This has been a problem for me, personally, since debian has made various ad-hoc change to Twisted or Twisted-based packages to break our plugin system, since the distutils metadata has been insufficient for their purposes. If the deb-generating stuff were in a separate project with a faster release cycle that were easier to contribute packages to, perhaps debian packagers could be convinced to contribute their build- hacks there (and bdist_deb could invoke debhelper, or vice-versa). It would be great if someone could volunteer to maintain this stuff independently, put it in a Launchpad project or something. IMHO it would be better for the community at large if this were spun as "increasing the release speed" of the bdist_* family, rather than "removing", which seems to me like it would generate another teacup- tempest on the blogowebs. Of course I'm not volunteering, but I will be best friends forever with whoever does this PR/maintenance :). Given that py2exe and py2app (which includes "bdist_mpkg") are both based on distutils, it seems like we're on the way to independent maintenance anyway. Perhaps bdist_wininst/_msi could be donated to the py2exe project if they would be willing to maintain it, and the new project for _deb and _rpm could be called "py2packman" or something.
glyph@divmod.com schrieb:
On 07:59 pm, fdrake@acm.org wrote:
I'm actually in favor of removing the bdist_* from the standard library, and allowing 3rd-party tools to implement whatever they need for the distros. But I don't think what you're presenting there supports it.
I do think that it's relevant that the respective operating system packagers don't find bdist_rpm, bdist_deb, et. al. useful. It's not very useful to have a bdist_deb that nobody actually builds debs with. This has been a problem for me, personally, since debian has made various ad-hoc change to Twisted or Twisted-based packages to break our plugin system, since the distutils metadata has been insufficient for their purposes. If the deb-generating stuff were in a separate project with a faster release cycle that were easier to contribute packages to, perhaps debian packagers could be convinced to contribute their build- hacks there (and bdist_deb could invoke debhelper, or vice-versa).
It would be great if someone could volunteer to maintain this stuff independently, put it in a Launchpad project or something. IMHO it would be better for the community at large if this were spun as "increasing the release speed" of the bdist_* family, rather than "removing", which seems to me like it would generate another teacup- tempest on the blogowebs. Of course I'm not volunteering, but I will be best friends forever with whoever does this PR/maintenance :).
Given that py2exe and py2app (which includes "bdist_mpkg") are both based on distutils, it seems like we're on the way to independent maintenance anyway. Perhaps bdist_wininst/_msi could be donated to the py2exe project if they would be willing to maintain it, and the new project for _deb and _rpm could be called "py2packman" or something.
Well, py2exe is Windows only. And I know that people used bdist_wininst to create windows installers on Linux. -- Thanks, Thomas
On 2009-03-27 21:49, glyph@divmod.com wrote:
On 07:59 pm, fdrake@acm.org wrote:
I'm actually in favor of removing the bdist_* from the standard library, and allowing 3rd-party tools to implement whatever they need for the distros. But I don't think what you're presenting there supports it.
I do think that it's relevant that the respective operating system packagers don't find bdist_rpm, bdist_deb, et. al. useful. It's not very useful to have a bdist_deb that nobody actually builds debs with. This has been a problem for me, personally, since debian has made various ad-hoc change to Twisted or Twisted-based packages to break our plugin system, since the distutils metadata has been insufficient for their purposes. If the deb-generating stuff were in a separate project with a faster release cycle that were easier to contribute packages to, perhaps debian packagers could be convinced to contribute their build- hacks there (and bdist_deb could invoke debhelper, or vice-versa).
It would be great if someone could volunteer to maintain this stuff independently, put it in a Launchpad project or something. IMHO it would be better for the community at large if this were spun as "increasing the release speed" of the bdist_* family, rather than "removing", which seems to me like it would generate another teacup- tempest on the blogowebs. Of course I'm not volunteering, but I will be best friends forever with whoever does this PR/maintenance :).
Given that py2exe and py2app (which includes "bdist_mpkg") are both based on distutils, it seems like we're on the way to independent maintenance anyway. Perhaps bdist_wininst/_msi could be donated to the py2exe project if they would be willing to maintain it, and the new project for _deb and _rpm could be called "py2packman" or something.
Do you really think that splitting up the distutils package is going to create a better user experience ? What would benefit the bdist_* commands is externalized maintenance, ie. have more frequent releases on PyPI, but still ship the most up-to-date versions with core distutils in each new Python release. BTW: py2exe and py2app solve a different set of problems than distutils is trying to solve. They are about packaging complete applications, not individual packages, so I don't see how they relate to the bdist_* commands. But perhaps I'm missing some context. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 27 2009)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
I do think that it's relevant that the respective operating system packagers don't find bdist_rpm, bdist_deb, et. al. useful. It's not very useful to have a bdist_deb that nobody actually builds debs with.
I think that conclusion is invalid: just because the distributions don't use it doesn't mean that nobody uses it. As a data point, there are 16 packages on PyPI that release RPMs (I haven't checked how they actually built them, though).
This has been a problem for me, personally, since debian has made various ad-hoc change to Twisted or Twisted-based packages to break our plugin system, since the distutils metadata has been insufficient for their purposes. If the deb-generating stuff were in a separate project with a faster release cycle that were easier to contribute packages to, perhaps debian packagers could be convinced to contribute their build- hacks there (and bdist_deb could invoke debhelper, or vice-versa).
I don't think this would happen. For .deb, you can't simply have "deb-generating stuff" - you have to acually manually package each file. The only way to get that out of the hands of the Debian maintainer would be if the package author provided the necessary data, which in turn requires that the "deb generating stuff" is readily available to the package author. In fact, .deb is a proof that it does *not* help to have the package commands outside distutils. For .deb, the command actually *is* outside distutils (there is no bdist_deb in distutils) - and it hasn't helped.
It would be great if someone could volunteer to maintain this stuff independently, put it in a Launchpad project or something.
Perhaps. However, for none of the bdist commands, anybody actually volunteered to maintain them outside of distutils. I would not want to see any of them removed until I know whom to blame when they die after being removed :-)
Given that py2exe and py2app (which includes "bdist_mpkg") are both based on distutils, it seems like we're on the way to independent maintenance anyway. Perhaps bdist_wininst/_msi could be donated to the py2exe project if they would be willing to maintain it, and the new project for _deb and _rpm could be called "py2packman" or something.
Perhaps. I'm skeptical that they want it. Also notice that bdist_wininst works fine on Linux, and I'm skeptical that Linux users would have easy access to it if it became part of py2exe. Regards, Martin
Martin v. Löwis wrote:
I do think that it's relevant that the respective operating system packagers don't find bdist_rpm, bdist_deb, et. al. useful. It's not very useful to have a bdist_deb that nobody actually builds debs with.
I think that conclusion is invalid: just because the distributions don't use it doesn't mean that nobody uses it. As a data point, there are 16 packages on PyPI that release RPMs (I haven't checked how they actually built them, though).
And I personally use bdist_rpm for my work, which I distribute to a farm of servers under my control. So no doubt it's used.
In fact, .deb is a proof that it does *not* help to have the package commands outside distutils. For .deb, the command actually *is* outside distutils (there is no bdist_deb in distutils) - and it hasn't helped.
It proves that it doesn't help given the current state of affairs. I suspect that if all of this additional information needed to build a .deb (for example) could be included as metadata in the python package (using the word "package" loosely), that it would be. It would make the ultimate packager's life easier, and it's no real burden for the original author.
Eric Smith writes:
And I personally use bdist_rpm for my work, which I distribute to a farm of servers under my control. So no doubt it's used.
Sure, but use for internal distribution is very different from to problem its being asked to solve now, isn't it? IIUC, you're basically using RPM as an installer for a standalone application, where you set policy at both ends, packaging and installation. This may be a group of modules which may have internal interdependencies, but very likely the environment doesn't change much. Pretty much anything will do in that kind of situation, and I don't think it would matter to you if there are zero, one, or twelve such tools in stdlib, as long as there's one you like in PyPI somewhere.
[That .deb tools are necessarily maintained outside of bdist] proves that [external maintenance] doesn't help given the current state of affairs. I suspect that if all of this additional information needed to build a .deb (for example) could be included as metadata in the python package (using the word "package" loosely), that it would be. It would make the ultimate packager's life easier, and it's no real burden for the original author.
I'm not sure what you mean by "it would be". Are you referring to what I believe to be true, that because Python and Python-based apps need to integrate with the rest of the OS, it's quite burdensome for module authors to include the necessary information, which is likely to vary from packaging tool to packaging tool, and according to policy even among packagers using the same tool? Or do you mean that it would be useful, so it would be nice if such information were included, and that as you see it the required effort would be small?
2009/3/28 Stephen J. Turnbull <stephen@xemacs.org>:
Sure, but use for internal distribution is very different from to problem its being asked to solve now, isn't it? IIUC, you're basically using RPM as an installer for a standalone application, where you set policy at both ends, packaging and installation. This may be a group of modules which may have internal interdependencies, but very likely the environment doesn't change much. Pretty much anything will do in that kind of situation, and I don't think it would matter to you if there are zero, one, or twelve such tools in stdlib, as long as there's one you like in PyPI somewhere.
I myself would never use such a tool, unless sanctioned by my OS vendor, because I would not trust it not to break my system. But I think bdist_rpm and bdist_deb solve a real deficiency: no uninstallation feature. Thinking of it, that's exactly why I like bdist_wininst so much when I am on windows (and because the consequences of a bad installer from bdist_wininst seem minimal on windows, seems everything is one directory). David
Stephen J. Turnbull wrote:
Eric Smith writes:
And I personally use bdist_rpm for my work, which I distribute to a farm of servers under my control. So no doubt it's used.
Sure, but use for internal distribution is very different from to problem its being asked to solve now, isn't it? IIUC, you're basically using RPM as an installer for a standalone application, where you set policy at both ends, packaging and installation. This may be a group of modules which may have internal interdependencies, but very likely the environment doesn't change much. Pretty much anything will do in that kind of situation, and I don't think it would matter to you if there are zero, one, or twelve such tools in stdlib, as long as there's one you like in PyPI somewhere.
I was just pointing out that bdist_rpm has users, and it's not likely to be abandoned. It's certainly true that different users have different use cases for it. It's this lack of understanding of all the use cases that most concerns me about this overall effort. How can we know if we succeeded if we don't all agree on the state we're trying to get to? To be concrete, I think distutils should support (among other things): - entry points for plugins - entry points as used for producing console and windowed "scripts" - dependency declarations for other python packages - dependency declarations for non-python packages But maybe these goals aren't shared by others, and don't fall into anyone else's "make distutils better" concept. PJE pointed out "A library that targets Python 2.4 and 2.5 and uses wsgiref, sqlite, ctypes, or ElementTree, for example, may have different dependencies depending on the version it is being installed in." Is that something we want to support?
[That .deb tools are necessarily maintained outside of bdist] proves that [external maintenance] doesn't help given the current state of affairs. I suspect that if all of this additional information needed to build a .deb (for example) could be included as metadata in the python package (using the word "package" loosely), that it would be. It would make the ultimate packager's life easier, and it's no real burden for the original author.
I'm not sure what you mean by "it would be". Are you referring to what I believe to be true, that because Python and Python-based apps need to integrate with the rest of the OS, it's quite burdensome for module authors to include the necessary information, which is likely to vary from packaging tool to packaging tool, and according to policy even among packagers using the same tool? Or do you mean that it would be useful, so it would be nice if such information were included, and that as you see it the required effort would be small?
I don't see how they differ. It's definitely true that packagers using the same tool might want to produce different package layouts and no doubt other differences. I don't see why it wouldn't be easy to have these differences driven by different policies acting on the same metadata, or small amounts of custom (per-packager) metadata.
Eric Smith writes:
I was just pointing out that bdist_rpm has users, and it's not likely to be abandoned.
OK, I see. I don't think there's a reason to remove useful functionality from the stdlib, unless it's clearly superseded by a similar module.
I don't see how they differ. It's definitely true that packagers using the same tool might want to produce different package layouts and no doubt other differences. I don't see why it wouldn't be easy to have these differences driven by different policies acting on the same metadata, or small amounts of custom (per-packager) metadata.
My experience in XEmacs has been that Debian, Fedora Core, Gentoo, SuSE, and Mandriva have rather different expressions of things like dependencies, and they tend to have different ideas of how forceful they should be with any given supporting package (when the package system supports different strengths of dependency). Debian, for example, supports "predepends" (you can't even install the dependent unless the prerequisite is already installed), "depends" (installing the dependent will also install the prerequisite unless the admin is quite forceful about saying no), "recommends" (it's easy to say no), and "suggests" (you only get a message saying "Things go better with Coke" and a suggestion that you may want to install Coke because you installed Things). In other systems there's either a dependency, or there isn't. Gentoo has "use flags" which are about as flexible as Debian dependencies, but their taste in recommendations is quite different. I really don't see how that kind of thing can be easily supported by a Python module maintainer, unless they're also the downstream packager.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 28, 2009, at 10:10 AM, Stephen J. Turnbull wrote:
I really don't see how that kind of thing can be easily supported by a Python module maintainer, unless they're also the downstream packager.
They simply can't. As a package developer, I'd be sort of okay with integrating patches that help downstreams, but I wouldn't be happy about it. I can't test it, and there might be conflicts between the demands of various downstreams. Much more appealing is for me to describe what's in my package with rich enough metadata that downstreams don't need anything else to overlay their build rules according to their own needs. Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSc5Lf3EjvBPtnXfVAQJoJQQAjJ4BeTk/ovhPvfJLMOSM+mcB7wz4XaX8 X9YlCnmQzPyNtbPdaaaLtkE86Sk6AC3fGO5hVjrSTANmzI02ztvNw4Lm1+HurTC5 6lghbQ1yEudZ3EgVdFu91jYBHrNPkgQtQA/oaB2/paER/8LeGqBppZiitm9plfHB 0LLHkLtylJ8= =qT/n -----END PGP SIGNATURE-----
2009/3/29 Stephen J. Turnbull <stephen@xemacs.org>:
I really don't see how that kind of thing can be easily supported by a Python module maintainer, unless they're also the downstream packager.
Almost none. But in my understanding, that's not what most linux packagers vendors ask about - they will handle the dependencies themselves anyway, because naming conventions and the like are different. What is a pain right now with distutils for packagers is: - how to control which files are installed where - how to control the build (compilation flags, etc...). Packagers generally "like" autotools packages because they can be customized along each distribution convention. Autotools do not really handle dependencies either, but they can be customized for vastly different kind of deployement (one dir for everything ala gobolinux, along the FHS for most major distributions, etc...) - and the upstream developer doesn't need to care much about it. cheers, David
On Mar 28, 2009, at 9:33 AM, Eric Smith wrote:
To be concrete, I think distutils should support (among other things): - entry points for plugins - entry points as used for producing console and windowed "scripts"
This strikes me as a nice-to-have, but: 1. I don't think they're two distinct features. 2. I'm not convinced these should go in distutils. I'd rather see an API to get resources from a package, and conventions can be developed among tool developers on how to store that information in a named resource.
- dependency declarations for other python packages
This is the most important requirement here, I think. As part of this, I'd want to be able to say things like "PIL, with JPEG and PNG support compiled in."
- dependency declarations for non-python packages
This would be very nice to have, but I suspect this is actually much more difficult than Python package dependencies, especially with any pretense at cross-platform expressions of dependencies.
PJE pointed out "A library that targets Python 2.4 and 2.5 and uses wsgiref, sqlite, ctypes, or ElementTree, for example, may have different dependencies depending on the version it is being installed in." Is that something we want to support?
Even simple cases present issues with regard to this. For example, I work on a project that relies on the uuid module, so we declare a dependency on Ka-Ping Ye's uuid module (since we're using Python 2.4). How should we write that in a version-agnostic way if we want to use the standard library version of that module with newer Pythons? -Fred -- Fred Drake <fdrake at acm.org>
Fred Drake wrote:
Even simple cases present issues with regard to this. For example, I work on a project that relies on the uuid module, so we declare a dependency on Ka-Ping Ye's uuid module (since we're using Python 2.4). How should we write that in a version-agnostic way if we want to use the standard library version of that module with newer Pythons?
Well, that could be done be getting standard library modules to: - declare what version they are - be overridable why installed packages That way, the fact that the standard library's development moves at the speed of frozen tar wouldn't stop packages in it being developed and released seperately for people who want to use newer versions of them and aren't in a situation where they need "batteries included"... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On 28/03/2009 7:49 AM, glyph@divmod.com wrote:
Perhaps bdist_wininst/_msi could be donated to the py2exe project if they would be willing to maintain it, and the new project for _deb and _rpm could be called "py2packman" or something.
As mentioned, it isn't really a natural fit - but regardless, py2exe is struggling to maintain itself at the moment... Cheers, Mark
Mark Hammond <skippy.hammond <at> gmail.com> writes:
As mentioned, it isn't really a natural fit - but regardless, py2exe is struggling to maintain itself at the moment...
It is the first auto-maintained package I have ever heard of :-) Regards Antoine.
participants (12)
-
"Martin v. Löwis" -
Antoine Pitrou -
Barry Warsaw -
Chris Withers -
David Cournapeau -
Eric Smith -
Fred Drake -
glyph＠divmod.com -
M.-A. Lemburg -
Mark Hammond -
Stephen J. Turnbull -
Thomas Heller