An observation on how system packagers and developers can be friends

Periodically, in various venues, we discuss the challenges of deploying applications with or in spite of system packaging of Python and system packaging philosophies. (Note that I'm mainly talking g about deploying applications, as opposed to individual Python packages.) In my experience, the conversation usually unfolds with developers (like me:) snarkely saying how annoyed we are and systems packagers (and administrators) explaining all of the rules they have to follow and that we should follow too. In practice, I think the result is usually that developers make an end-run around system packaging system, either building in production, or by building system packages that bundle dependencies to get needed control. The result is that at least some of the system packagers goals are subverted. In my experience, both sides have legitimate goals, but the problem is rarely, if ever, approached from the point of view of recognizing both sets of goals coming up with a (probably new) solution that addresses both sets of goals. (Note that "doing things the way I've always done them" is *not* a valid goal. :) I believe that solutions that address both sets of goals are possible and even practical, but it has to start with a consideration of the underlying goals. Jim -- Jim Fulton

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/28/2010 12:44 PM, Jim Fulton wrote:
Periodically, in various venues, we discuss the challenges of deploying applications with or in spite of system packaging of Python and system packaging philosophies.
(Note that I'm mainly talking g about deploying applications, as opposed to individual Python packages.)
In my experience, the conversation usually unfolds with developers (like me:) snarkely saying how annoyed we are and systems packagers (and administrators) explaining all of the rules they have to follow and that we should follow too.
In practice, I think the result is usually that developers make an end-run around system packaging system, either building in production, or by building system packages that bundle dependencies to get needed control. The result is that at least some of the system packagers goals are subverted.
In my experience, both sides have legitimate goals, but the problem is rarely, if ever, approached from the point of view of recognizing both sets of goals coming up with a (probably new) solution that addresses both sets of goals. (Note that "doing things the way I've always done them" is *not* a valid goal. :)
I believe that solutions that address both sets of goals are possible and even practical, but it has to start with a consideration of the underlying goals.
I like the idea in general, but worry that some conflicts may not be resolvable. For instance, I don't know what goal drives system packagers to specify UCS4 over the default UCS2, but I won't ever be happy using a Python built that way for long-running, memory-intensive applications, where I have measured the overhead of UCS4 and found it unacceptable (e.g., a server app whose steady-state process size is 800Mb under UCS4, compared to 600Mb under UCS2). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzJusQACgkQ+gerLs4ltQ6GcQCgkLKlUnbb85KFaYVMlJpBBE5F B1MAn0hDSjU7GJ24/zHExDFqg9kkmTXz =Lztf -----END PGP SIGNATURE-----

I like the idea in general, but worry that some conflicts may not be resolvable. For instance, I don't know what goal drives system packagers to specify UCS4 over the default UCS2
That is easy to say: feature support. You can't really call it "Unicode" if you can't index it by code point. Some parts of the API (e.g. the Unicode database) are incomplete in UCS-2 mode. A minor objective is C library compliance: wchar_t has also four bytes per character.
but I won't ever be happy using a Python built that way for long-running, memory-intensive applications, where I have measured the overhead of UCS4 and found it unacceptable (e.g., a server app whose steady-state process size is 800Mb under UCS4, compared to 600Mb under UCS2).
I don't think this is really in the field of issues that Jim was talking about. In any case, if you have long-running memory-intensive applications, recompiling Python might be just the right thing to do. In the same direction, I wish people would understand that 64-bit Python builds just waste memory, and that they were better off with 32-bit implementations.
From the developer point of view, I'd wonder whether there is potential for some even more dramatic savings than 25%.
Regards, Martin

In the same direction, I wish people would understand that 64-bit Python builds just waste memory, and that they were better off with 32-bit implementations.
There are certain irrational reasons to believe that "64 bits is better than 32", but there is also some logic to it. I support an application that can't run in a 32 bit python because there literally is not enough memory available. It needs a larger address space. If the system python is only 32 bits, the users can't run the application. Most of them are not qualified to try to build their own 64 bit python. Most of them are not even qualified to try to maintain two copies of python, install certain software in each, and then choose whether they need 32 or 64 bits before they start their data analysis. Since some applications need the larger address space, it is better for python to just be 64 bits all the time. It makes the computer do a little extra work for our convenience, but that is exactly the point of using computers. Mark S.

On Oct 28, 2010, at 02:02 PM, Tres Seaver wrote:
I like the idea in general, but worry that some conflicts may not be resolvable. For instance, I don't know what goal drives system packagers to specify UCS4 over the default UCS2, but I won't ever be happy using a Python built that way for long-running, memory-intensive applications, where I have measured the overhead of UCS4 and found it unacceptable (e.g., a server app whose steady-state process size is 800Mb under UCS4, compared to 600Mb under UCS2).
We're getting closer to being able to support this more easily upstream with all the work for exposing abi build flags in shared library and executable names. But that won't help you much because it's Python 3. ;) But I think to the extent that something like this could be backported into Python 2, it's probably the general approach to take. All that aside, understanding specifically what a "clean Python" means is a great first step. ;) -Barry

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/28/2010 04:50 PM, Barry Warsaw wrote:
On Oct 28, 2010, at 02:02 PM, Tres Seaver wrote:
I like the idea in general, but worry that some conflicts may not be resolvable. For instance, I don't know what goal drives system packagers to specify UCS4 over the default UCS2, but I won't ever be happy using a Python built that way for long-running, memory-intensive applications, where I have measured the overhead of UCS4 and found it unacceptable (e.g., a server app whose steady-state process size is 800Mb under UCS4, compared to 600Mb under UCS2).
We're getting closer to being able to support this more easily upstream with all the work for exposing abi build flags in shared library and executable names. But that won't help you much because it's Python 3. ;)
Doesn't matter to me anyway, as I don't have the issue about trying to share hierarchies between multiple slightly-differing builds. ;)
But I think to the extent that something like this could be backported into Python 2, it's probably the general approach to take.
All that aside, understanding specifically what a "clean Python" means is a great first step. ;)
Heh, "untar + CMMI into a non-system prefix" works for me. ;) Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzJ7q0ACgkQ+gerLs4ltQ4T3QCggKh9FSRDXdVqP3l6xmVthLaJ YOwAn3CSP3oF0tL3c7oweK3Ecz8QDg8j =ZNIn -----END PGP SIGNATURE-----

On Thu, Oct 28, 2010 at 5:44 PM, Tres Seaver <tseaver@palladion.com> wrote:
Heh, "untar + CMMI into a non-system prefix" works for me. ;)
+1 with the small addition of "after making sure the dev dependencies Python sniffs out to build modules for (zlib, crypto bits, etc.) are available." -- Benji York

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/28/2010 05:54 PM, Benji York wrote:
On Thu, Oct 28, 2010 at 5:44 PM, Tres Seaver <tseaver@palladion.com> wrote:
Heh, "untar + CMMI into a non-system prefix" works for me. ;)
+1 with the small addition of "after making sure the dev dependencies Python sniffs out to build modules for (zlib, crypto bits, etc.) are available."
Heh, agreed. That bites me on about every third machine I set up for the first time. The Usual Suspects (TM) are whatever the local packaging system calls the following (and their -dev or -devel packages, if split out): - - zlib - - ncurses - - bz2 - - readline - - openssl Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzJ9HQACgkQ+gerLs4ltQ4BawCfWxmQP+HB9atlynYrnMCTt2ka oJQAn3rk7OskTRP6vzo7kI4KjoGMhYp3 =R7tK -----END PGP SIGNATURE-----

On Oct 28, 2010, at 6:08 PM, Tres Seaver wrote:
Heh, agreed. That bites me on about every third machine I set up for the first time. The Usual Suspects (TM) are whatever the local packaging system calls the following (and their -dev or -devel packages, if split out):
- - zlib - - ncurses - - bz2 - - readline - - openssl
On Debian, there's a handy shortcut: 'apt-get build-dep', which will install the build dependencies for any given source package. So 'apt-get build-dep python' will get you all set to build Python.

Op 29-10-10 00:49, Glyph Lefkowitz schreef:
On Oct 28, 2010, at 6:08 PM, Tres Seaver wrote:
Heh, agreed. That bites me on about every third machine I set up for the first time. The Usual Suspects (TM) are whatever the local packaging system calls the following (and their -dev or -devel packages, if split out):
- - zlib - - ncurses - - bz2 - - readline - - openssl
On Debian, there's a handy shortcut: 'apt-get build-dep', which will install the build dependencies for any given source package. So 'apt-get build-dep python' will get you all set to build Python.
I did not know that one yet, thanks. I tried that on Ubuntu Jaunty. It returns rather a lot: $ sudo apt-get build-dep --dry-run python Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: debhelper debiandoc-sgml doc-base dvipdfmx ed gettext ghostscript gsfonts html2text intltool-debian lacheck latex-beamer latex-xcolor libcompress-raw-zlib-perl libcompress-zlib-perl libcroco3 libcups2 libcupsimage2 libfreezethaw-perl libgs8 libhtml-parser-perl libhtml-tagset-perl libhtml-tree-perl libice6 libio-compress-base-perl libio-compress-zlib-perl libkpathsea4 libmail-sendmail-perl libmldbm-perl libpaper-utils libpaper1 libpoppler4 libroman-perl libsgmls-perl libsm6 libsp1c2 libsys-hostname-long-perl libtext-format-perl libtiff4 libuuid-perl libxaw7 libxext6 libxmu6 libxt6 lmodern pgf po-debconf preview-latex-style prosper ps2eps psfontmgr python-docutils python-roman sgml-base sgml-data sgmlspl sp tex-common texinfo texlive-base texlive-base-bin texlive-base-bin-doc texlive-common texlive-doc-base texlive-extra-utils texlive-fonts-recommended texlive-fonts-recommended-doc texlive-generic-extra texlive-generic-recommended texlive-humanities texlive-humanities-doc texlive-latex-base texlive-latex-base-doc texlive-latex-extra texlive-latex-extra-doc texlive-latex-recommended texlive-latex-recommended-doc texlive-pictures texlive-pictures-doc texlive-pstricks texlive-pstricks-doc tipa xml-core 0 upgraded, 83 newly installed, 0 to remove and 3 not upgraded. Somehow I doubt that I need TeX to build python. (La)TeX is wonderful but there are limits to what it can do. :-) I can imagine it might be needed for building a .deb package though, so it probably makes sense. -- Maurits van Rees Programmer, Zest Software

On Sat, Oct 30, 2010 at 02:43:38AM +0200, Maurits van Rees wrote:
Op 29-10-10 00:49, Glyph Lefkowitz schreef:
On Debian, there's a handy shortcut: 'apt-get build-dep', which will install the build dependencies for any given source package. So 'apt-get build-dep python' will get you all set to build Python.
I did not know that one yet, thanks. I tried that on Ubuntu Jaunty.
Jaunty went end-of-life a month ago, time to upgrade?
It returns rather a lot:
$ sudo apt-get build-dep --dry-run python Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: ... 0 upgraded, 83 newly installed, 0 to remove and 3 not upgraded.
Somehow I doubt that I need TeX to build python.
Why is it a build-dep, anyway? Python's documentation sources are now ReStructuredText, not LaTeX -- since 2.6, I think. I don't believe there are PDFs shipped in python2.x-doc packages. Is it an obsolete build dependency from earlier times? Marius Gedminas -- Favorite MAC error message: "Not enough memory to eject disk!"

On Oct 30, 2010, at 04:59 PM, Marius Gedminas wrote:
Why is it a build-dep, anyway? Python's documentation sources are now ReStructuredText, not LaTeX -- since 2.6, I think. I don't believe there are PDFs shipped in python2.x-doc packages. Is it an obsolete build dependency from earlier times?
Apparently so. I just took a look at the python2.6 source package for Ubuntu (likely the same for Debian), and there is no build-dep on latex. -Barry
participants (9)
-
"Martin v. Löwis"
-
Barry Warsaw
-
Benji York
-
Glyph Lefkowitz
-
Jim Fulton
-
Marius Gedminas
-
Mark Sienkiewicz
-
Maurits van Rees
-
Tres Seaver