Re: [Distutils] shebang line modified by setuptools
At 09:07 PM 4/11/2008 -0700, Cliff Wells wrote:
Hi Phillip,
I recently upgraded a hosting server from Python 2.4 to Python 2.5. Many of the sites on this server couldn't easily be upgraded to 2.5 (at least not in a timely fashion), so my interim solution was to simply symlink ~/bin/python -> /usr/bin/python2.4 (~/bin is first in the user's PATH). This worked swimmingly except for applications which relied on Paste (I should note I use the method outlined here: http://peak.telecommunity.com/DevCenter/EasyInstall#administrator-installati...).
The reason the Paste apps had an issue is because apparently setuptools modifies the shebang line of paster to be "#!/usr/bin/python". This hardcoded path overrides any PATH settings. I understand that this is done at the time Paste is installed to tie it to the installed version of Python. This makes sense, except that it doesn't do what it's intended to when Python is upgraded (/usr/bin/python is no longer the version it was installed with).
It seems the correct solution to this is to use "#!/usr/bin/env python" (or rather, evaluate `which env` to account for some systems which have /bin/env rather that /usr/bin/env) which allows the sys admin to override the Python binary much more easily.
No, it isn't the correct solution. The correct solution is to re-run easy_install on the target -- it will leave the existing installed code in place, but regenerate the scripts with headers pointing to the currently-in-use Python. Even that is often not as good an idea as reinstalling for a new version of Python; some packages (notably, setuptools itself, but I have others) include different code and data depending on the Python version they are built for. For example, a package built for Python 2.5 might not include dependency specs for ctypes, sqlite, or wsgiref, but the same package might need to specify those dependencies on earlier versions of Python. By the way, please don't send me unsolicited private emails about setuptools unless you're a client or potential client. There's a reason that every single official web page about setuptools directs you to email the distutils-sig. I don't do private email for ANY of my open source projects, unless you're a client or looking to be one. Mailing lists have searchable archives, meaning that my time answering a question is being offset by its potential utility to a much larger audience. Thanks.
On Sat, 2008-04-12 at 12:19 -0400, Phillip J. Eby wrote:
At 09:07 PM 4/11/2008 -0700, Cliff Wells wrote:
It seems the correct solution to this is to use "#!/usr/bin/env python" (or rather, evaluate `which env` to account for some systems which have /bin/env rather that /usr/bin/env) which allows the sys admin to override the Python binary much more easily.
No, it isn't the correct solution. The correct solution is to re-run easy_install on the target -- it will leave the existing installed code in place, but regenerate the scripts with headers pointing to the currently-in-use Python.
This is good to know, but still not as good as using env and the reason is simple: every Unix user on the planet knows the env solution and what it means if they see it in a script (they can override using PATH). The same cannot be said for the above. Unless I'm missing something, I can't see the advantage your way has over the traditional way. I'll be glad to hear how I'm wrong.
Even that is often not as good an idea as reinstalling for a new version of Python; some packages (notably, setuptools itself, but I have others) include different code and data depending on the Python version they are built for. For example, a package built for Python 2.5 might not include dependency specs for ctypes, sqlite, or wsgiref, but the same package might need to specify those dependencies on earlier versions of Python.
Absolutely. Except I was specifically *not* upgrading many applications. Rather I wanted them to remain at 2.4. These are the ones I had to change the shebang line for. Now I know I could have run easy_install instead, but that information wasn't easily located at the time.
By the way, please don't send me unsolicited private emails about setuptools unless you're a client or potential client. There's a reason that every single official web page about setuptools directs you to email the distutils-sig. I don't do private email for ANY of my open source projects, unless you're a client or looking to be one. Mailing lists have searchable archives, meaning that my time answering a question is being offset by its potential utility to a much larger audience. Thanks.
My apologies. The link for the mailing list on the page that covers this on the peak site is quite small and follows a lot of text. Regards, Cliff Wells
On Sat, Apr 12, 2008 at 09:58:28AM -0700, Cliff Wells wrote:
On Sat, 2008-04-12 at 12:19 -0400, Phillip J. Eby wrote:
At 09:07 PM 4/11/2008 -0700, Cliff Wells wrote:
It seems the correct solution to this is to use "#!/usr/bin/env python" (or rather, evaluate `which env` to account for some systems which have /bin/env rather that /usr/bin/env) which allows the sys admin to override the Python binary much more easily.
No, it isn't the correct solution. The correct solution is to re-run easy_install on the target -- it will leave the existing installed code in place, but regenerate the scripts with headers pointing to the currently-in-use Python.
This is good to know, but still not as good as using env and the reason is simple: every Unix user on the planet knows the env solution and what it means if they see it in a script (they can override using PATH). The same cannot be said for the above.
+1. Gaël
On Sat, 2008-04-12 at 13:49 -0400, Phillip J. Eby wrote:
At 09:58 AM 4/12/2008 -0700, Cliff Wells wrote:
Unless I'm missing something, I can't see the advantage your way has over the traditional way.
Well for one, it isn't affected by changes in PATH.
I'd call this a distinct disadvantage: the shebang line is a Unixism, not a Pythonism. Python applications shouldn't be an aberation on a Unix platform. PATH is *supposed* to affect applications. Trying to break this isn't helping anyone. It's also worth noting that this a deployment-specific consideration. Many deployers are not Python experts (or at least, we shouldn't expect them to be), but they ought to be fairly expert in their deployment platform. I would not expect a sysadmin to know where to start looking to know that they should re-run easy_install, but I would certainly expect them to know how to fix a PATH variable for an account. Regards, Cliff Wells
On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote:
At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote:
PATH is *supposed* to affect applications.
It affects which application you should run, not which interpreter you run the application with.
I think that's splitting hairs and pretty much just made up to defend your position. Regardless, ultimately it's dead wrong. Of *course* it should affect which interpreter you run. If I install a patched version of Python (or Perl, or PHP, or whatever), into /usr/local/bin and put that before /usr/bin in the PATH, I most absolutely expect that to be the interpreter that is used, regardless of what was used at the time the app was installed. If this breaks the application, it's fairly straightforward to adjust the PATH to fix it. Most importantly, it's something that a Unix admin (vs a Python programmer) can be expected to know. Anyway, regardless of who's correct concerning this issue (we can agree to disagree), I'm cannot understand why you'd want Python to behave differently (from a deployment standpoint) than other languages. If you want sysadmins to hate deploying Python apps, you are on the right road (make it unusual and require special Python knowledge to do "right"). In fact, "rightness" has little to do with it. Many conventions and standards aren't "right", but they continue to exist because that's what is understood. Being "right" at the expense of being non-standard isn't necessarily a good choice. Regards, Cliff
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 12, 2008, at 6:26 PM, Cliff Wells wrote:
On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote:
At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote:
PATH is *supposed* to affect applications.
It affects which application you should run, not which interpreter you run the application with.
I think that's splitting hairs and pretty much just made up to defend your position. Regardless, ultimately it's dead wrong. Of *course* it should affect which interpreter you run. If I install a patched version of Python (or Perl, or PHP, or whatever), into /usr/local/bin and put that before /usr/bin in the PATH, I most absolutely expect that to be the interpreter that is used, regardless of what was used at the time the app was installed. If this breaks the application, it's fairly straightforward to adjust the PATH to fix it. Most importantly, it's something that a Unix admin (vs a Python programmer) can be expected to know.
Many operating systems and distributions now use Python extensively. Those should absolutely hardcode the #! line to use the system Python and should not in any way be affected by my $PATH. I've seen too many situations where changing $PATH breaks some aspect of the OS. I happen to think that setuptools is doing the right thing here, but it would be nice to have an override. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAE6AXEjvBPtnXfVAQJKuQP+JfqPDweFzQ1hN50XFQbi6InQEqL+b+dY M4UIdl5aS48ozoYIKGgM1aItgt/dfe0imCQkl2/cY9vzjmRRVVzK0RurPHjxyAWT CSFi+FuG54rJV4M0xwnoAZaEHQqo65g35iGz+mMe68ytKaSEt1TgukRzyF6RudBf 3Up+IgC8Kqc= =7t0k -----END PGP SIGNATURE-----
On Sat, Apr 12, 2008 at 03:26:52PM -0700, Cliff Wells wrote:
Anyway, regardless of who's correct concerning this issue (we can agree to disagree), I'm cannot understand why you'd want Python to behave differently (from a deployment standpoint) than other languages.
Numbers ! Only numbers can speak. On my Ubuntu box (standard, nothing special done to it): varoquau@epsilon:~$ grep '#!/usr/bin/env python' /usr/bin/* 2>/dev/null wc -l 33 varoquau@epsilon:~$ grep '#!/usr/bin/python' /usr/bin/* 2>/dev/null | wc -l 113 And if we don't look at the language type: varoquau@epsilon:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null | wc -l 39 (the rest is pretty much perl, by the way). On a server I have an account on (Debian, I don't administer it): varoquau@quatramaran:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null | wc -l 52 (this one has a much higher ratio of perl to python). On my mailhost: varoquau@phare:~$ grep '#!/usr/bin/python' /usr/bin/* 2>/dev/null | wc -l 10 varoquau@phare:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null /usr/bin/raggle:#!/usr/bin/env ruby /usr/bin/web2png:#!/usr/bin/env python On my hosted and managed webserver (these guys seem to like ruby): (uiserver):u39018796:~ > grep '#!/usr/bin/python' /usr/bin/* 2>/dev/null | wc -l 0 (uiserver):u39018796:~ > grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null /usr/bin/gem:#!/usr/bin/env ruby /usr/bin/gem_mirror:#!/usr/bin/env ruby /usr/bin/gem_server:#!/usr/bin/env ruby /usr/bin/gemlock:#!/usr/bin/env ruby /usr/bin/gemri:#!/usr/bin/env ruby /usr/bin/gemwhich:#!/usr/bin/env ruby /usr/bin/index_gem_repository:#!/usr/bin/env ruby (uiserver):u39018796:~ > grep '#!/usr/bin/ruby' /usr/bin/* 2>/dev/null /usr/bin/erb1.8:#!/usr/bin/ruby1.8 /usr/bin/irb1.8:#!/usr/bin/ruby1.8 /usr/bin/rdoc1.8:#!/usr/bin/ruby1.8 /usr/bin/testrb1.8:#!/usr/bin/ruby1.8 (uiserver):u39018796:~ > In conclusion, I think there is no real rule. It is more on a case by case basis. I prefer the "env". I could yell to the top of my voice on this mailing list to back my opinion. But the numbers are there, and they don't help me make my point :-). Gaël
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 12, 2008, at 6:50 PM, Gael Varoquaux wrote:
On Sat, Apr 12, 2008 at 03:26:52PM -0700, Cliff Wells wrote:
Anyway, regardless of who's correct concerning this issue (we can agree to disagree), I'm cannot understand why you'd want Python to behave differently (from a deployment standpoint) than other languages.
Numbers ! Only numbers can speak. On my Ubuntu box (standard, nothing special done to it):
varoquau@epsilon:~$ grep '#!/usr/bin/env python' /usr/bin/* 2>/dev/ null wc -l 33
And if we don't look at the language type:
varoquau@epsilon:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null | wc -l 39
These are all broken and you should report bugs on them. I have reported many for Ubuntu. A system application should only ever depend on the system Python (or interpreter), never on the whims of your $PATH. If you want an application to depend on $PATH, grab a development release and install that in your favorite location. That's when /usr/bin/env is appropriate. Cheers, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAFMoXEjvBPtnXfVAQKC2gQAgiFFH9KGinw3yy9wyAI6p1Hn2Sy+Sb6e x5z3B9kkT99zOl5AxdsaCEPWEIB+/Q2tRJypZXPVe9o7GPMIZETF9RzGIFV/tsmh LQF+lIPbdvXHsK1f1CibuRt3P+ij+EuSWH81FI7uE3SLXine51WRW+Pp+m3+74Xp kEP1dWn1GvE= =NFBv -----END PGP SIGNATURE-----
Barry Warsaw wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 12, 2008, at 6:50 PM, Gael Varoquaux wrote:
On Sat, Apr 12, 2008 at 03:26:52PM -0700, Cliff Wells wrote:
Anyway, regardless of who's correct concerning this issue (we can agree to disagree), I'm cannot understand why you'd want Python to behave differently (from a deployment standpoint) than other languages. Numbers ! Only numbers can speak. On my Ubuntu box (standard, nothing special done to it):
varoquau@epsilon:~$ grep '#!/usr/bin/env python' /usr/bin/* 2>/dev/ null wc -l 33
And if we don't look at the language type:
varoquau@epsilon:~$ grep '#!/usr/bin/env' /usr/bin/* 2>/dev/null | wc -l 39
These are all broken and you should report bugs on them. I have reported many for Ubuntu. A system application should only ever depend on the system Python (or interpreter), never on the whims of your $PATH.
I agree with Barry here -- I would even go further and say that the Python used by system apps (i.e., apps that the OS depends on) should be separate and should not be on the default PATH (so the user can't use it accidentally), and no non-system apps should use the system Python unless specifically installed by the user or sysadmin into the system Python's area. I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;) Steve
On Sat, Apr 12, 2008 at 09:41:08PM -0400, Stephen Waterbury wrote:
These are all broken and you should report bugs on them. I have reported many for Ubuntu. A system application should only ever depend on the system Python (or interpreter), never on the whims of your $PATH.
I agree with Barry here -- I would even go further and say that the Python used by system apps (i.e., apps that the OS depends on) should be separate and should not be on the default PATH (so the user can't use it accidentally), and no non-system apps should use the system Python unless specifically installed by the user or sysadmin into the system Python's area.
I agree with you, and not with Barry that all these are broken. If you look at the apps that use "#!/usr/bin/env", they are all pretty non-system apps, like for instance "f2py". I think it is good that these apps use the default python: I don't want to have f2py to be running for another python than my default one. I am actually quite surprised to see that ipython is installed with a hardcoded path to /usr/bin/python. Anyway, I am not going to bug report anything, as my system works just fine, thank you. Gaël
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 13, 2008, at 5:50 AM, Gael Varoquaux wrote:
On Sat, Apr 12, 2008 at 09:41:08PM -0400, Stephen Waterbury wrote:
These are all broken and you should report bugs on them. I have reported many for Ubuntu. A system application should only ever depend on the system Python (or interpreter), never on the whims of your $PATH.
I agree with Barry here -- I would even go further and say that the Python used by system apps (i.e., apps that the OS depends on) should be separate and should not be on the default PATH (so the user can't use it accidentally), and no non-system apps should use the system Python unless specifically installed by the user or sysadmin into the system Python's area.
I agree with you, and not with Barry that all these are broken. If you look at the apps that use "#!/usr/bin/env", they are all pretty non-system apps, like for instance "f2py". I think it is good that these apps use the default python: I don't want to have f2py to be running for another python than my default one. I am actually quite surprised to see that ipython is installed with a hardcoded path to /usr/bin/python.
Anyway, I am not going to bug report anything, as my system works just fine, thank you.
I'm not sure how you can disagree with me and not with Steve since he and I agree the system apps are broken. ;) Oh well, doesn't matter. Steve has a very interesting idea though, and it makes sense to me. I'm not sure where in the FHS you'd put that, but it makes sense for Python-written system apps to use the hidden Python and for /usr/bin/ python to be one that users can play with. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAH3jHEjvBPtnXfVAQKIJAP/WWE6i1pjpFJU3Z0mT3AiYeIKZjTfVXjV jC0DAgGj9AWS9TS1eGYWntb65C3ibZK2vsFjkr4Zg32yx1Dl2qvOfQ3yPD5+z7k5 pLP+LfP1U80HDl2poPRJCetnwV/F29IJpEHgNnWZboqrRrPxJlD3WI8L9D4gECP5 I19vl6e/IVo= =3/Pm -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote:
I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;)
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature. I've reported bugs on this and had discussions with some of the Debian Python packaging folks. I'm hoping that we'll find a solution that doesn't collide with a from-source default Python build. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAH1mnEjvBPtnXfVAQJAvAP/S1p03Jlq73fZB4H6Qt7cH/qXpInQ4G7i VDMdH0yxZ3Fih8LG/eFcvpu0YCI7iHWtct+h2GXA/gUVPlgWxwJDE2Ahixg9MNS3 faETU1DNqJb4oUbwjZD/FzwLCGRXp/17s4+Q2JX4YaOvShsVeS+F3Kr+3zwfDL0m hrMRfaJTYSE= =a8MZ -----END PGP SIGNATURE-----
On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote:
On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote:
I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;)
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature.
I want it like that. You are confusing /opt and /usr/local. This is the way I expect things to work. I do not want to install my own packages in "/usr/lib/pythonX.Y/site-packages". This is for apt to deal with, not for me. Gaël
Gael Varoquaux wrote:
On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote:
On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote:
I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;)
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature.
I want it like that. You are confusing /opt and /usr/local.
Heh ... that shows my Berkeley roots -- I'm not confused, I'm just an old SunOS guy still in denial about that damn SysV stuff that Solaris brought in, like using /opt instead of /usr/local -- bah, humbug! The OS has /usr; /usr/local is *mine* dammit! ;) Steve
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 13, 2008, at 11:03 AM, Stephen Waterbury wrote:
Gael Varoquaux wrote:
On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote:
On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote:
I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;)
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature.
I want it like that. You are confusing /opt and /usr/local.
Heh ... that shows my Berkeley roots -- I'm not confused, I'm just an old SunOS guy still in denial about that damn SysV stuff that Solaris brought in, like using /opt instead of /usr/local -- bah, humbug! The OS has /usr; /usr/local is *mine* dammit! ;)
You and me both Steve! Of course, we're old farts so when we retire <wink> the kids can have their /usr/local. Until then, they'll have to pry it out of hands. :) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAI5EXEjvBPtnXfVAQKU3wP+LnbcfTiE9AwDmIdNGrbnijGxWM5xGNgF Zg42Zs9RWWjjCABoOkZyHw1k3f0mp5DZtgA/XR8joPiVwyq8VV2diIQI4pw/c2No AGP0XpTOeqjHSm84jEDIT1YaFfp5NTuDzN360zxM4neLi9nG7pVD+OGebL4n/OPh aSB08rP/vKM= =cqEh -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 13, 2008, at 8:04 AM, Gael Varoquaux wrote:
On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote:
On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote:
I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;)
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature.
I want it like that. You are confusing /opt and /usr/local.
This is the way I expect things to work. I do not want to install my own packages in "/usr/lib/pythonX.Y/site-packages". This is for apt to deal with, not for me.
Actually, the solution that I believe is a good compromise is to find some other path in /usr/local to augment the system Python's sys.path with. All I'm saying is that Python itself uses /usr/local/lib/ pythonX.Y/site-packages as the default from-source path, so if Debian wants a /usr/local path, it can pick some other subdirectory path and still meet its goals. BTW, I believe Debian is unique here. I talked to the Fedora guys and I've talked to a few Gentoo guys and both seemed surprised by Debian's policy here. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAI4A3EjvBPtnXfVAQLeagP/aCKr6WL+qvvM8vhABFxJDBbOJw1hQUQT fCUwL3bZ21a6iKiHLv5mdVBfaFjcqEmnbGqizUmTgnTuG+SaUXHNqBBG2lDCTZUF fNwWThN/TRtqO5OdknqFcfTQARtBE+YN52CpugjSmRrudZtAAdslWds8HnUP3nlG kN9J0TsEo1E= =DDbJ -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Barry Warsaw napsal(a): | On Apr 13, 2008, at 8:04 AM, Gael Varoquaux wrote: |> On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: |>> On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: |>>> I used to always set up my own Python[s] in /usr/local |>>> and put that first in my PATH, but I have gotten lazy lately, and |>>> sometimes it will bite me. ;) |>> On Debian and derivatives (e.g. Ubuntu) you might have even more fun. |>> They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of |>> the system python*! This means that you can break your system Python |>> by installing a version of Python from source and then distutil'ing |>> things into there. Astoundingly, this is promoted as a feature. |> I want it like that. You are confusing /opt and /usr/local. | |> This is the way I expect things to work. I do not want to install my |> own |> packages in "/usr/lib/pythonX.Y/site-packages". This is for apt to |> deal |> with, not for me. | | Actually, the solution that I believe is a good compromise is to find | some other path in /usr/local to augment the system Python's sys.path | with. All I'm saying is that Python itself uses /usr/local/lib/ | pythonX.Y/site-packages as the default from-source path, so if Debian | wants a /usr/local path, it can pick some other subdirectory path and | still meet its goals. BTW, I believe Debian is unique here. I talked | to the Fedora guys and I've talked to a few Gentoo guys and both | seemed surprised by Debian's policy here. Actually, openSUSE does this too, and i've even gone as far as making distutils install into /usr/local by default. Several reasons to do this: a) FHS says, "user installed apps go into /usr/local, period" b) the user should have a clean site-packages directory for his needs c) we don't want the user to mess with package-managed modules in /usr and finally, the reason i'm not very happy to "pick some other subdirectory path": d) it is simple to tell distutils that default prefix is /usr/local. It is complicated and ugly to tell it that by default it should go to /usr/local/some-directory-of-my-own, and still be able to revert to system-ish installation by supplying --prefix=/usr (following the same semantics that autoconf/automake is using). Of course we could introduce SUSE-specific --install-like-system-does switch, but if the user wants, for whatever reason, to install to a system-ish location, he would need to find out about this. And furthermore, it would make cross-distro packaging even harder (openSUSE Buildservice anyone?) Plus, /usr/local/lib/pythonX.Y/site-packages still looks like a reasonably standard path, whereas /usr/local/weird-place/nobody-ever-heard-of is just your garden variety nonsensical and unsupported distro-specific customization. I never realized that this would be a bad thing to do, and i still dot think so - /usr/local is at the end of sys.path anyway, so user can override system-installed packages and break his python only if he specifically tries to do that. I will be happy to abandon this practice as soon as there is some kind of official policy that solves all the mentioned points. (Of course, things would be much simpler if python, in perl's fashion, by default supported site-packages and vendor-packages directories specifically for this purpose - still violating FHS, but in a way nobody really cares about anymore. Nudge nudge, wink wink ;) ) regards jan matejek | | -Barry | _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFIA6mUjBrWA+AvBr8RAsK6AJ9lo7foixt2/LmV2RDb3TCM6e46igCeKB4f R2mWEX8hQ4qG2UaeescJS24= =u88R -----END PGP SIGNATURE-----
On Mon, Apr 14, 2008 at 08:59:32PM +0200, Jan Matejek wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Barry Warsaw napsal(a): | On Apr 13, 2008, at 8:04 AM, Gael Varoquaux wrote: |> On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote: |>> On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote: |>>> I used to always set up my own Python[s] in /usr/local |>>> and put that first in my PATH, but I have gotten lazy lately, and |>>> sometimes it will bite me. ;) |>> On Debian and derivatives (e.g. Ubuntu) you might have even more fun. |>> They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of |>> the system python*! This means that you can break your system Python |>> by installing a version of Python from source and then distutil'ing |>> things into there. Astoundingly, this is promoted as a feature. |> I want it like that. You are confusing /opt and /usr/local. | |> This is the way I expect things to work. I do not want to install my |> own |> packages in "/usr/lib/pythonX.Y/site-packages". This is for apt to |> deal |> with, not for me. | | Actually, the solution that I believe is a good compromise is to find | some other path in /usr/local to augment the system Python's sys.path | with. All I'm saying is that Python itself uses /usr/local/lib/ | pythonX.Y/site-packages as the default from-source path, so if Debian | wants a /usr/local path, it can pick some other subdirectory path and | still meet its goals. BTW, I believe Debian is unique here. I talked | to the Fedora guys and I've talked to a few Gentoo guys and both | seemed surprised by Debian's policy here.
Actually, openSUSE does this too, and i've even gone as far as making distutils install into /usr/local by default. Several reasons to do this: a) FHS says, "user installed apps go into /usr/local, period" b) the user should have a clean site-packages directory for his needs c) we don't want the user to mess with package-managed modules in /usr
and finally, the reason i'm not very happy to "pick some other subdirectory path": d) it is simple to tell distutils that default prefix is /usr/local. It is complicated and ugly to tell it that by default it should go to /usr/local/some-directory-of-my-own, and still be able to revert to system-ish installation by supplying --prefix=/usr (following the same semantics that autoconf/automake is using).
I entirely agree with this (from Debian's POV). This is the way many things in the system works, you install shared libraries to /usr/local/lib etc. I think Python needs 3 clear site-packges directories: 1) system 2) local-admin 3) user. Personally I'd like to see PEP 370, which is trying to introduce 3) above, move in that direction. No idea how I go about that though. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Barry Warsaw wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote:
I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;)
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature.
That *is* astounding. I've been using Ubuntu for several years now, and I blush to admit I never noticed that until just the other day when I tried installing Python 2.6 from source (into the default /usr/local location, of course) and it broke all my running web stuff, so I hastily removed it -- sheesh! As I mentioned, I've been "lazy" about using the system Python for my own apps and development, but I would feel much more comfortable if the system Python had its own space.
I've reported bugs on this and had discussions with some of the Debian Python packaging folks. I'm hoping that we'll find a solution that doesn't collide with a from-source default Python build.
Thanks for fighting the good fight, Barry -- hang in there! Maybe you can push for my "radical" solution. :) Steve
On Sun, Apr 13, 2008 at 11:16:57AM -0400, Stephen Waterbury wrote:
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature.
That *is* astounding. I've been using Ubuntu for several years now, and I blush to admit I never noticed that until just the other day when I tried installing Python 2.6 from source (into the default /usr/local location, of course) and it broke all my running web stuff, so I hastily removed it -- sheesh!
As I mentioned, I've been "lazy" about using the system Python for my own apps and development, but I would feel much more comfortable if the system Python had its own space.
I've reported bugs on this and had discussions with some of the Debian Python packaging folks. I'm hoping that we'll find a solution that doesn't collide with a from-source default Python build.
Well, I am not too happy about what I hear here. It seems to me as if Ubuntu is headed the macOS X way, where (it seems) that a second Python needs to be installed on top of the system Python to add modules to it. I know it is not exactly the case, but this change is _really_ going to confuse people and I can this the mailing lists of module developper pilling up with requests for help. There needs to be a way for the user to install his own modules and keep using the system Python. I think it is a very bad idea to do it in /usr. Yes, you can teach your users to add things to their PYTHONPATH, but this put one additional thing to learn for people who just want to get things done. I think users are going to hate this. Maybe something less drastic can be done, by making sure that the /usr/local is far in the PYTHONPATH, that way packages installed in /usr/local do not override the system packages. Of course setuptools will break this, because it adds itsef to front of the PYTHONPATH, and totally break the PYTHONPATH semantics (Grrr). I still think you guys have it wrong, and if you want a new Python install, you do it in /opt. For me /usr/local is for user programs that cooperate for system programs. My 2 cents, Gaël
Gael Varoquaux wrote:
On Sun, Apr 13, 2008 at 11:16:57AM -0400, Stephen Waterbury wrote:
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature.
That *is* astounding. I've been using Ubuntu for several years now, and I blush to admit I never noticed that until just the other day when I tried installing Python 2.6 from source (into the default /usr/local location, of course) and it broke all my running web stuff, so I hastily removed it -- sheesh!
As I mentioned, I've been "lazy" about using the system Python for my own apps and development, but I would feel much more comfortable if the system Python had its own space.
I've reported bugs on this and had discussions with some of the Debian Python packaging folks. I'm hoping that we'll find a solution that doesn't collide with a from-source default Python build.
Well, I am not too happy about what I hear here. It seems to me as if Ubuntu is headed the macOS X way, where (it seems) that a second Python needs to be installed on top of the system Python to add modules to it. I know it is not exactly the case, but this change is _really_ going to confuse people and I can this the mailing lists of module developper pilling up with requests for help. There needs to be a way for the user to install his own modules and keep using the system Python.
I think you misunderstand my proposal, Gael ... You and I are using "system Python" to mean different things. In my proposal, *the* "system Python" and a packaged Python installed using the system's package manager would be separate things (and may or may not be the same version of Python)! What I am proposing: 1) the OS comes with its own "system Python", which is installed not as the "python" package, but as some OS-required package (maybe call it "system-python" or something) and it goes into /usr/system/bin/python or whatever -- it doesn't matter what the path is as long as it's not /usr/bin or anything on the default path. And system utilities that are python scripts should have their own system-specific, hard-coded shebang line. 2) separately from the "system Python", the available packages shown by the system's package manager include one or more "python.x" packages which are python interpreters that the user or sysadmin can optionally install, and which go into /usr. And the system package manager -- e.g., apt on Debian/Ubuntu systems) would have all its usual nicely-packaged python apps (python-this, python-that, ...) that would also install into /usr and use the nicely-packaged python (not to be confused with the "system Python" of 1). Steve
Stephen Waterbury wrote: Oops, my email client doesn't detect unbalanced parens -- typo here (what I get for too-complicated paren/double-dash-mixing ;):
optionally install, and which go into /usr. And the system package manager -- e.g., apt on Debian/Ubuntu systems) would have all its usual ^ should be "--" nicely-packaged python apps (python-this, python-that, ...) that would also install into /usr and use the nicely-packaged python (not to be confused with the "system Python" of 1).
Steve
On Sun, Apr 13, 2008 at 12:31:08PM -0400, Stephen Waterbury wrote:
What I am proposing:
1) the OS comes with its own "system Python", which is installed not as the "python" package, but as some OS-required package (maybe call it "system-python" or something) and it goes into /usr/system/bin/python or whatever -- it doesn't matter what the path is as long as it's not /usr/bin or anything on the default path. And system utilities that are python scripts should have their own system-specific, hard-coded shebang line.
2) separately from the "system Python", the available packages shown by the system's package manager include one or more "python.x" packages which are python interpreters that the user or sysadmin can optionally install, and which go into /usr. And the system package manager -- e.g., apt on Debian/Ubuntu systems) would have all its usual nicely-packaged python apps (python-this, python-that, ...) that would also install into /usr and use the nicely-packaged python (not to be confused with the "system Python" of 1).
OK. I am starting to see what you mean. I agree it does make sense. It seems to me that you are bringing in a distinction between "system Python scripts" and user Python script. For me the system Python scripts should live in "/bin" and use the system Python, and the users should live in "/usr/bin" and use "/usr/bin/env python". But that's just me. Inspecting my boxes did show that this is quite close to the way it is already on Debian systems. I don't have accounts on other kind of Unix, so I can't see how it is done elsewhere. Cheers, Gaël
At 05:29 PM 4/13/2008 +0200, Gael Varoquaux wrote:
Of course setuptools will break this, because it adds itsef to front of the PYTHONPATH, and totally break the PYTHONPATH semantics (Grrr).
The entire point of PYTHONPATH is to be able to put things at the front of it, so I don't see how that's breaking anything. When you ask easy_install to install something as the default importable version of that package, the only way to ensure that is by placing that thing at the front of sys.path, to override any other version of it that might be installed elsewhere on sys.path. Well, the only way short of going through all of sys.path and deleting anything that might potentially be in the way. That approach was tried in I think the 0.4 line of setuptools, however, and it wasn't very popular. :)
On Sun, Apr 13, 2008 at 02:04:56PM -0400, Phillip J. Eby wrote:
At 05:29 PM 4/13/2008 +0200, Gael Varoquaux wrote:
Of course setuptools will break this, because it adds itsef to front of the PYTHONPATH, and totally break the PYTHONPATH semantics (Grrr).
The entire point of PYTHONPATH is to be able to put things at the front of it, so I don't see how that's breaking anything.
Well actually, my phrasing in the above was catastrophic. I have mixed PYTHONPATH and sys.path. Anyhow, I have a problem with how setuptools deals with both: * sys.path
When you ask easy_install to install something as the default importable version of that package, the only way to ensure that is by placing that thing at the front of sys.path, to override any other version of it that might be installed elsewhere on sys.path.
What I would want is setuptools to respect the priorities for packages depending on where they are installed. In other words, if I install a package in /usr/local/lib and this is further down the priorities than /usr/lib, then I don't expect this package to override module imports happening from /usr/lib. You can justify what setuptools is doing from an implementation point of view, but it doesn't change the fact that it is doing the wrong thing. Wrong because it goes against what Python does. Thus I have to learn some completely new import semantics to deal with the fact that some of the packages on my system where installed by setuptools. In addition there is a good reason behind the standard Python behavior. * PYTHONPATH With standard Python, I can override the import mechanism using the PYTHONPATH. This way I can make sure that the modules I want get imported. And pretty much all the programs I can think of with a path environment variable behave like this. Now if I have a module installed with setuptools, I no longer have the possibility to do this, because setuptools inserts it self before the PYTHONPATH. First of all this really breaks the users expectancies. Second it puts people in a dead end. Actually it forces them to use setuptools, or to monkey-patch sys.path themselves, to override the import of a module. Once again you are expecting your users to know way more than a beginner's knowledge of Python. Python is simple and does what you expect... no longer. This is way worse than not respecting sys.path priorities. I have lost hours on that, and when I found out why, ... well I won't copy my remarks about setuptools on this mailing list. And what has happened to me has probably happened to many. This second problem could probably easily be fixed, though, by some more clever monkey patching of the sys.path. Actually both problems are solvable, if you care about not changing the way Python works. If you don't care, than make it obvious everywhere. But even if you do, I think it is a very bad thing, because people will still fall for the trap (eg I copy my scripts to a new server, which uses setuptools, although I don't know it, and I actually don't know what setuptools are). Don't call it Python, if you are going to change fundamentally the way it works. My 2 cents, Gaël
At 08:26 PM 4/13/2008 +0200, Gael Varoquaux wrote:
On Sun, Apr 13, 2008 at 02:04:56PM -0400, Phillip J. Eby wrote:
At 05:29 PM 4/13/2008 +0200, Gael Varoquaux wrote:
Of course setuptools will break this, because it adds itsef to front of the PYTHONPATH, and totally break the PYTHONPATH semantics (Grrr).
The entire point of PYTHONPATH is to be able to put things at the front of it, so I don't see how that's breaking anything.
Well actually, my phrasing in the above was catastrophic. I have mixed PYTHONPATH and sys.path. Anyhow, I have a problem with how setuptools deals with both:
* sys.path
When you ask easy_install to install something as the default importable version of that package, the only way to ensure that is by placing that thing at the front of sys.path, to override any other version of it that might be installed elsewhere on sys.path.
What I would want is setuptools to respect the priorities for packages depending on where they are installed. In other words, if I install a package in /usr/local/lib and this is further down the priorities than /usr/lib, then I don't expect this package to override module imports happening from /usr/lib.
And you're saying that setuptools does this? Since when? [pause to investigate] Hm... that's a bug, actually. If there are multiple 'site' directories, then what you describe will indeed happen; eggs in the later 'site' directory will indeed override those in the former, which should not happen. The issue is that the postamble of easy-install.pth is simply inserting the eggs at the head of sys.path, even in cases where they could simply be inserted immediately before the directory the .pth file is in. One minor issue, however, is that eggs in the systemwide site-packages must also be able to override the stdlib - so simply locating the correct insertion point isn't going to fix the whole problem. However, now that it's a known bug, it's merely an engineering problem to fix it. :)
With standard Python, I can override the import mechanism using the PYTHONPATH. This way I can make sure that the modules I want get imported. And pretty much all the programs I can think of with a path environment variable behave like this. Now if I have a module installed with setuptools, I no longer have the possibility to do this, because setuptools inserts it self before the PYTHONPATH.
I'm not following you here. Are you trying to specify the use of a different version of setuptools? What do you mean by "setuptools inserts itself before the PYTHONPATH"?
This second problem could probably easily be fixed, though, by some more clever monkey patching of the sys.path.
In general, setuptools' path management could be improved by tweaking its path hacks so that an egg is positioned only in front of the directory that contains it (except for lib/site-packages eggs, which need to also override lib). Patches welcome.
On Sun, Apr 13, 2008 at 05:12:48PM -0400, Phillip J. Eby wrote:
However, now that it's a known bug, it's merely an engineering problem to fix it. :)
Fantastic! I thought it was a feature, so I was unhappy, but if it's only a bug, well, I know too well that these things happen, and if nobody reports them, they don't get fixed.
With standard Python, I can override the import mechanism using the PYTHONPATH. This way I can make sure that the modules I want get imported. And pretty much all the programs I can think of with a path environment variable behave like this. Now if I have a module installed with setuptools, I no longer have the possibility to do this, because setuptools inserts it self before the PYTHONPATH.
I'm not following you here. Are you trying to specify the use of a different version of setuptools? What do you mean by "setuptools inserts itself before the PYTHONPATH"?
Here is how things are happening (I simplified a bit the example, because I have a lot of things in my sys.path): varoquau@epsilon:~$ python -c "import sys; print sys.path" ['', '/usr/local/lib/python2.5/site-packages/configobj-4.4.0-py2.5.egg', '/usr/lib/python2.5/site-packages', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode'] varoquau@epsilon:~$ export PYTHONPATH="/home/varoquau/python:" varoquau@epsilon:~$ python -c "import sys; print sys.path" ['', '/usr/local/lib/python2.5/site-packages/configobj-4.4.0-py2.5.egg', '/usr/lib/python2.5/site-packages', '/home/varoquau/python', '/home/varoquau', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode'] Notice how what I added in my PYTHONPATH (the environment variable) is behind the egg. Well, what if I wanted to override configobj by my own version, in "/home/varoquau/python"? I am not able to do this without writing some Python code to monkey patch sys.path. The "$PYTHONPATH" should always come in front of everything. Are my explanations clear this time? Maybe this was not a feature, and another bug. In which case, please excuse the tone of my previous message. I should have reported the problem. I thought it was a know one, because it seems so natural to do the following in a Unix world. Cheers, Gaël
At 11:24 PM 4/13/2008 +0200, Gael Varoquaux wrote:
Here is how things are happening (I simplified a bit the example, because I have a lot of things in my sys.path):
varoquau@epsilon:~$ python -c "import sys; print sys.path" ['', '/usr/local/lib/python2.5/site-packages/configobj-4.4.0-py2.5.egg', '/usr/lib/python2.5/site-packages', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode'] varoquau@epsilon:~$ export PYTHONPATH="/home/varoquau/python:" varoquau@epsilon:~$ python -c "import sys; print sys.path" ['', '/usr/local/lib/python2.5/site-packages/configobj-4.4.0-py2.5.egg', '/usr/lib/python2.5/site-packages', '/home/varoquau/python', '/home/varoquau', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode']
Notice how what I added in my PYTHONPATH (the environment variable) is behind the egg. Well, what if I wanted to override configobj by my own version, in "/home/varoquau/python"? I am not able to do this without writing some Python code to monkey patch sys.path. The "$PYTHONPATH" should always come in front of everything. Are my explanations clear this time?
Maybe this was not a feature, and another bug.
It's the same bug, actually. Ideally, the configobj.egg should be in front of the /usr/lib/python25.zip, but not any further forward than that. The last line of easy-install.pth is essentially too crude in its insertion method: it defaults to inserting at position zero, when it should default to a location that is either directly before the directory containing the .pth, or if that directory is the primary site-packages directory, then it should insert in a position that is after PYTHONPATH and the script directory, but before the stdlib. Now, if someone will contribute an alternative postamble for easy-install.pth that can figure all that out in a cross-platform/cross-python-version way -- and within the constraints of the limited code you can jam into a .pth file -- we can get it fixed. :) (The cross-python-version comment is a reference to the fact that the 'site' module was heavily rewritten in either 2.4 or 2.5; I forget which. So the tools available from inside a .pth file vary somewhat between Python versions.)
Gael Varoquaux wrote:
a second Python needs to be installed on top of the system Python to add modules to it.
Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood. -- Greg
Greg Ewing wrote:
Gael Varoquaux wrote:
a second Python needs to be installed on top of the system Python to add modules to it.
Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood.
That was exactly my proposal -- see my message in this thread at 12:31 PM today. Steve
Greg Ewing <greg.ewing@canterbury.ac.nz> writes:
Gael Varoquaux wrote:
a second Python needs to be installed on top of the system Python to add modules to it.
Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood.
I can't see how this is a good idea. I've seen it mentioned multiple times in this thread, but without justification. It's my position that the Python instance one uses for development should diverge as little as possible from the default system instance. Otherwise one is actively pursuing a recipe for dependency failures when one eventually deploys the result. Yes, there should be a way to deploy the code one is actively developing and testing to some location that will not affect the rest of the operation of the system. That is a far cry from asking that there should be multiple, divergent Python instances on the system. The further step of saying that the operating system should not use the same Python instance as a "user" on the system (and note that the line between those is far from clear), but instead should use some Python instance that behaves differently from every other use of Python on the system, seems like an even worse proposition, for the above reasons. Help me understand, please. -- \ "Most people don't realize that large pieces of coral, which | `\ have been painted brown and attached to the skull by common | _o__) wood screws, can make a child look like a deer." -- Jack Handey | Ben Finney
Ben Finney wrote:
Greg Ewing <greg.ewing@canterbury.ac.nz> writes:
Gael Varoquaux wrote:
a second Python needs to be installed on top of the system Python to add modules to it. Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood.
I can't see how this is a good idea. I've seen it mentioned multiple times in this thread, but without justification.
It's my position that the Python instance one uses for development should diverge as little as possible from the default system instance. Otherwise one is actively pursuing a recipe for dependency failures when one eventually deploys the result.
Yes, there should be a way to deploy the code one is actively developing and testing to some location that will not affect the rest of the operation of the system. That is a far cry from asking that there should be multiple, divergent Python instances on the system.
The further step of saying that the operating system should not use the same Python instance as a "user" on the system (and note that the line between those is far from clear), but instead should use some Python instance that behaves differently from every other use of Python on the system, seems like an even worse proposition, for the above reasons.
Help me understand, please.
Have you read my proposal? It's in the message posted by me to this thread at 12:31 PM today. My main motivation in proposing it is to give the user complete control over the Python environment that is in their path -- which is not easy to do on Debian/Ubuntu at the moment -- by giving the Python utilities and packages that OS-related functions depend on a completely insular Python environment that the OS can manage however it sees fit. In my view, both as a developer and a user, but especially as a developer, it would make life easier. Steve
Stephen Waterbury wrote:
Have you read my proposal? It's in the message posted by me to this thread at 12:31 PM today.
That's not a good way to reference it, of course, so here it is: Definitions: "system Python" -- the Python (and its site-pkgs, etc.) that any Python scripts used by the OS depend on, which would of course by solely determined by the OS. This would be distinct, in my proposal, from any packages named "python" or "python.x" in the package namespace of the system package manager (e.g. apt). What I am proposing: 1) that the OS comes with its own "system Python", which is installed not as the "python" package, but as some OS-required package (maybe call it "system-python" or something) and it goes into /usr/system/bin/python or whatever -- it doesn't matter what the path is as long as it's not /usr/bin or anything on the default path. And system utilities that are python scripts should have their own system-specific, hard-coded shebang line. 2) that separately from the "system Python", the available packages managed by the system's package manager include one or more "python.x" packages which are python interpreters that the user or sysadmin can optionally install, and which go into /usr, and which would share *nothing* with the "system Python". And the system package manager -- e.g., apt on Debian/Ubuntu systems) would have all its usual nicely-packaged python apps (python-this, python-that, ...) that would also install into /usr and use the nicely-packaged python (not to be confused with the "system Python" of 1). Fire away! :) Steve
On Apr 14, 2008, at 12:59 AM, Stephen Waterbury wrote:
Ben Finney wrote:
Greg Ewing <greg.ewing@canterbury.ac.nz> writes:
Gael Varoquaux wrote:
a second Python needs to be installed on top of the system Python to add modules to it. Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood.
I can't see how this is a good idea. I've seen it mentioned multiple times in this thread, but without justification.
It's my position that the Python instance one uses for development should diverge as little as possible from the default system instance. Otherwise one is actively pursuing a recipe for dependency failures when one eventually deploys the result.
Yes, there should be a way to deploy the code one is actively developing and testing to some location that will not affect the rest of the operation of the system. That is a far cry from asking that there should be multiple, divergent Python instances on the system.
The further step of saying that the operating system should not use the same Python instance as a "user" on the system (and note that the line between those is far from clear), but instead should use some Python instance that behaves differently from every other use of Python on the system, seems like an even worse proposition, for the above reasons.
Help me understand, please.
Have you read my proposal? It's in the message posted by me to this thread at 12:31 PM today.
My main motivation in proposing it is to give the user complete control over the Python environment that is in their path -- which is not easy to do on Debian/Ubuntu at the moment -- by giving the Python utilities and packages that OS-related functions depend on a completely insular Python environment that the OS can manage however it sees fit. In my view, both as a developer and a user, but especially as a developer, it would make life easier.
This seems related to the use case I was going after in 'autoinstall'. Which is to move the control of the Python environment to the application. This way each application can have its own insular Python environment. A first baby step in that direction: http://pypi.python.org/pypi/autoinstall/0.1a2 Currently it only supports binding dotted packages names to URLs (relative or absolute) that point to zip importable versions of the package. But I've been using it some and it's been making my life easier indeed. Curious what people think of a per application approach? The use case seems to be mentioned a fair bit in the various threads.
Stephen Waterbury <waterbug@pangalactic.us> writes:
Ben Finney wrote:
It's my position that the Python instance one uses for development should diverge as little as possible from the default system instance. Otherwise one is actively pursuing a recipe for dependency failures when one eventually deploys the result.
[...] Help me understand, please.
Have you read my proposal?
I'd read it, yes. Thanks for bringing it explicitly back into the thread.
My main motivation in proposing it is to give the user complete control over the Python environment that is in their path -- which is not easy to do on Debian/Ubuntu at the moment
This is a laudable goal. It is met, for just about every other user-configurable application, by allowing user-specific *overrides* to the defaults to be set in configuration files.
by giving the Python utilities and packages that OS-related functions depend on a completely insular Python environment that the OS can manage however it sees fit. In my view, both as a developer and a user, but especially as a developer, it would make life easier.
I can't see how you come to this conclusion. Surely one wants the possibility of divergence from the default to be explicit and easily discoverable? This is entirely lost if the only assumption that can be made is "the entire Python environment has been replicated and utterly isolated from the system default Python". Stephen Waterbury <waterbug@pangalactic.us> writes:
"system Python" -- the Python (and its site-pkgs, etc.) that any Python scripts used by the OS depend on, which would of course by solely determined by the OS.
This would be distinct, in my proposal, from any packages named "python" or "python.x" in the package namespace of the system package manager (e.g. apt).
It seems clear to me that divergence between these two should be kept to a minimum; and that such divergence is best minimalised if they are *the same instance*, merely operating under explicit customisation for each use that needs it. -- \ "Pinky, are you pondering what I'm pondering?" "I think so, | `\ Brain, but isn't that why they invented tube socks?" -- _Pinky | _o__) and The Brain_ | Ben Finney
Ben Finney wrote:
Stephen Waterbury <waterbug@pangalactic.us> writes:
"system Python" -- the Python (and its site-pkgs, etc.) that any Python scripts used by the OS depend on, which would of course be solely determined by the OS.
This would be distinct, in my proposal, from any packages named "python" or "python.x" in the package namespace of the system package manager (e.g. apt).
It seems clear to me that divergence between these two should be kept to a minimum ...
Why? Steve
Ben Finney <bignose+hates-spam@benfinney.id.au> writes:
Stephen Waterbury <waterbug@pangalactic.us> writes:
My main motivation in proposing it is to give the user complete control over the Python environment that is in their path -- which is not easy to do on Debian/Ubuntu at the moment
This is a laudable goal. It is met, for just about every other user-configurable application, by allowing user-specific *overrides* to the defaults to be set in configuration files.
I'm glad to see PEP 370 <URL:http://www.python.org/dev/peps/pep-0370/> has been approved (2008-05-06) for Python 2.6 and Python 3.0. Abstract This PEP proposes a new a per user site-packages directory to allow users the local installation of Python packages in their home directory. This addresses the issue following exactly the principle I described above, so I heartily approve. -- \ "Experience is that marvelous thing that enables you to | `\ recognize a mistake when you make it again." -- Franklin P. | _o__) Jones | Ben Finney
On Mon, Apr 14, 2008 at 01:30:14PM +1200, Greg Ewing wrote:
Gael Varoquaux wrote:
a second Python needs to be installed on top of the system Python to add modules to it.
Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood.
I am quite wary about these proposals, as well as the one environment per application ones. What you propose resembles very much to what MacOSX does, and MacOSX seems just so broken for Python. I don't use it, but I see on the different scientific-Python-related mailing lists how users have difficulties with MacOSX, and I have heard many people complain about this "feature". As a per-application environment, I think it is bad, because it limits reuse. As I see things, applications should be able to have access to all the Python modules installed, to be able to use them, if they need. I get definitely see applications having more feature if some modules are installed (eg. Sphinx, which does syntax highlighting if pygments is installed). I think this discussion is really going on because Python does not have good library-versioning support. What it needs is to get this, and not to get complete isolation, apart for some system-critical tasks, and only for these. AFAIK this is one of the problems setuptools is trying to solve. Not everybody is happy with the technical solution setuptools has come up, this only means it has to be improved. I think this library-versioning issue is very important, and needs to get support from the core developers, but I know this is not everybody's opinion :->. My 2 cents, Gaël
Gael Varoquaux wrote:
I think this discussion is really going on because Python does not have good library-versioning support.
I personally think that the very idea to have several side by side versions of the same package is doomed to failure: what is needed is a stable API for the used packages. Library versioning is really only (a small) part of the problem, and supporting it won't magically solve the issue of complex deployments. cheers, David
David Cournapeau wrote:
what is needed is a stable API for the used packages.
That's a nice ideal to aim for, but it's only achievable for old and mature packages. One could change the package name every time the API changes, but then *any* change to the API would make the package unusable by apps expecting an earlier version, even if the new API is backwards compatible. Having said that, I just realised you can address that by putting in symlinks for the previous versions. So, I hereby propose the following convention for naming of Python packages: packagename_<version> where <version> is a number or other identifier that is changed whenever the API changes in any way. If we all follow this convention, we will be able to install and use multiple versions of a package side-by-side just as easily as we can Python versions, with no need for setuptools or any other fancy technology -- just plain standard Python the way it currently is! -- Greg
Greg Ewing wrote:
David Cournapeau wrote:
what is needed is a stable API for the used packages.
That's a nice ideal to aim for, but it's only achievable for old and mature packages.
Yes, many packages can't have a stable API, at least for some time. But then, should you rely on them for deployment ? I am really not convinced that you can have both stable systems and packages which keep changing whenever they want. I don't believe that library versioning will give you that, that's just a really minor part of it. When you think about it for a second, what is library versioning ? A convention to track API changes. Library versioning without API stability just does not make sense to me. What would happen if with apt-get you could install gtk 2.0, 2.2, etc... up to gtk-2.12, and gtk broke some API between minor versions ? Nothing would work, because you would quickly have one package B which work with gtk 2.8 but not above, package C which needs gtk 2.10, and package D which needs B and C. That's why you have gtk-2.n, and debian project put huge effort to be sure that with some n, many projects will work on it. If gtk is not stable, this cannot happen. I am not aware of a single big open source project which have many dependencies without strong API versioning policies (which means not breaking between minor changes). For small projects, it is no problem. For big ones, you need either to rely on strongly versioned api, or to cut dependencies (by copying a private copy: see mozilla/firefox which embed their own gecko, ardour which embeds its own gtk, sndfile which embeds its own copy of ogg, etc...)
If we all follow this convention, we will be able to install and use multiple versions of a package side-by-side just as easily as we can Python versions, with no need for setuptools or any other fancy technology -- just plain standard Python the way it currently is!
I don't see how this could work. How do you do if you have a package D which depends on both C and B, and C needs one version of one package A, and B another one ? If you cannot use a common A, this cannot work (how one python instance would use different versions of the same package ?) cheers, David
David Cournapeau wrote:
Library versioning without API stability just does not make sense to me.
Yes, obviously if you have library versioning you need to use it properly in order for it to be any use.
How do you do if you have a package D which depends on both C and B, and C needs one version of one package A, and B another one?
If they really do need different versions, this is insoluble. If they can actually use the same version because it's compatible enough, it may be necessary to make Python smart enough to notice that they're symlinks to the same file and only import it once. Or maybe something does need to be added to the language that understands version numbering, compatibility rules, etc. as I suggested in another post. -- Greg
Greg Ewing wrote:
If they really do need different versions, this is insoluble.
But that's by far the most significant problem of packages with a lot of dependencies ! And by enabling multiple versions, side by side, you will make this problem more pervasive. I don't know if you have experience with packaging on linux systems, but if it was allowed multiple versions of the same library (say gtk 2.4, gtk 2.6, etc...) with incompatible changes of API in between, it would be a nightmare. What will happen if multiple versions of the same package are allowed in python ? Everyone will start to say :I need version == N, and quickly, it will be unmanageable, because the above diamond diagram A <- (B <- D1, C <- D2) will happen all the time. You really need to be able to say I need version >= N, which means maintaining API. It is ok to have a few major, side by side versions for fundamental packages (on my ubuntu system, I do have system installed qt 3 and qt 4, for example). This would be good to have in python, I agree. So my point is that package versioning only work if you can depends on one version and all revision after, which means side by side installation of several minor revisions does not help. Having 2,3 major different versions of the same package is OK, if the package changes say every two years (like python). But it won't work if it changes every month (imagine what would happen if python 2.3.1 was not compatible with 2.3.2 ?) IOW, enabling version requirement without strong API compatibility is the path to dependency hell. If you enable side by side installation of many versions of the same package, you will make the problem worse, not better. If you need to use a package without strong API compatibility commitment, then the only real solution is to make your own copy. cheers, David
David Cournapeau wrote:
Greg Ewing wrote:
If they really do need different versions, this is insoluble.
But that's by far the most significant problem of packages with a lot of dependencies !
But if your application really does depend on two libraries that have conflicting requirements like this, the application itself is screwed to begin with. There's *no* way of making it work on *any* system, whether that system has library versioning or not. Consequently, the developer will be unable to make it work on his own machine, and will therefore never attempt to deploy it!
It is ok to have a few major, side by side versions for fundamental packages (on my ubuntu system, I do have system installed qt 3 and qt 4, for example). This would be good to have in python, I agree.
That's what I'm talking about! As long as the API remains backward compatible, there is no need to have anything but the latest version installed -- and in fact, in the kind of system I have in mind, the earlier versions would *never be used* even if they were installed. That's because the program would say something like "Please give me gtk-2.3", and the system would know that anything called gtk-2.x with x >= 3 is compatible with that, and would pick the one with the greatest x out of those available. However, if a gtk-3.0 ever comes out, it implies that the API is *not* backward compatible. So then it makes sense to have both a gtk-2.x and a gtk-3.y installed, and the versioning system will keep them from getting confused. This relies on a couple of things: (a) library developers sticking rigidly to the version numbering semantics; (b) applications always requesting the version they actually require.
IOW, enabling version requirement without strong API compatibility is the path to dependency hell.
I fully agree. However, this is a social issue, not a technical one. Everyone using the versioning system would need to understand the rules and be willing to follow them. -- Greg
Greg Ewing wrote:
But if your application really does depend on two libraries that have conflicting requirements like this, the application itself is screwed to begin with. There's *no* way of making it work on *any* system, whether that system has library versioning or not.
Of course. I was just stressing that versioning does not solve the deployment problem, but merely is a small part of the solution.
Consequently, the developer will be unable to make it work on his own machine, and will therefore never attempt to deploy it!
It can happen easily once you have different systems: maybe on my os it is not a problem, but maybe it is on yours. Python is not exactly the same everywhere, the version of python is not the same everywhere, etc...
That's what I'm talking about! As long as the API remains backward compatible, there is no need to have anything but the latest version installed -- and in fact, in the kind of system I have in mind, the earlier versions would *never be used* even if they were installed.
What's the point of having it installed, then ? I am confused.
IOW, enabling version requirement without strong API compatibility is the path to dependency hell.
I fully agree. However, this is a social issue, not a technical one.
Yes, that's mainly a social issue, which is why I wanted to stress that versioning won't magically solve it (just to be clear, I am not saying that that's what you said, but I felt like one general idea in this thread was that once you have versioning, you solve deployment issues). All I am saying is: - If you use a package without a strong API commitment, and if you use versioning to use it, you will make problems worse, not better. - If you use versioning for packages with strong API commitment, then it is definitely useful. cheers, David
David Cournapeau wrote:
Greg Ewing wrote:
the earlier versions would *never be used* even if they were installed.
What's the point of having it installed, then ? I am confused.
I never said there was a point. When I mentioned multiple installed versions, I was talking about multiple *incompatible* versions. That part can be handled purely by convention -- just be disciplined enough to change the package name whenever an incompatible API change occurs. But the other part is what happens when there is a backward-compatible change -- apps need some way to say what minimum version they need, without having to exactly match the version installed. I've already mentioned one way of dealing with that -- install symlinks or .pth files or some such thing pointing from all the supported minor versions to the one that's currently installed. But that would cause difficulties due to different paths to the same module not being recognised as such. I think that Python would benefit from having some standard mechanism added to deal with this. One possibility I thought of is to extend the import statement so you can say things like import gtk[2,3] which means "give me gtk with major version == 2 and minor version >= 3". Any thoughts on that idea? -- Greg
On Apr 15, 2008, at 9:09 PM, Greg Ewing wrote:
David Cournapeau wrote:
Greg Ewing wrote:
If they really do need different versions, this is insoluble.
But that's by far the most significant problem of packages with a lot of dependencies !
But if your application really does depend on two libraries that have conflicting requirements like this, the application itself is screwed to begin with. There's *no* way of making it work on *any* system, whether that system has library versioning or not.
Consequently, the developer will be unable to make it work on his own machine, and will therefore never attempt to deploy it!
That's good and fine for the situation where one application is being deployed on a Python installation, but that isn't a very realistic situation. There is usually more than one application/script/whatever running off of a given Python installation. Here's an example of how you can get into an unworkable situation: Application A depends on: library B v 1.1 library C v 2.3 Library B v 1.1 has no dependencies. Now application D comes along. It depends upon: library B v 1.4 But library B v 1.4 depends upon library C v 2.6. Application A *breaks* with library C v 2.6. Which does the user give up? Application A (their storage management client) or application D (their monitoring agent)? This sort of stuff happens all the time. This is what a Versioning (with a capital V attempts to solve.)
That's because the program would say something like "Please give me gtk-2.3", and the system would know that anything called gtk-2.x with x >= 3 is compatible with that, and would pick the one with the greatest x out of those available.
Thats a great theory, but that's not how the real world works. Python packages are an ecology where there will be inconsistencies between different minor versions of the same package. A legitimate bug fix may break behavior that other packages depend upon while correcting the behavior for others. That's just the way stuff happens. We can encourage 10,000 disparate developers to adhere to best practices, but it's fantasy to imagine that they will. Most of the people putting together packages are donating their 80% solutions to the rest of the world. They will develop these packages over time. The packages will be changed to suit their *own* needs as time goes on. Those changes will be released back to the public eventually (we hope). Some of those changes are going to break compatibility with the old version for some reason or another. Perhaps 100% backwards compatibility either wasn't important to the author. Perhaps because they didn't have the time to do exhaustive backwards compatibility testing. Why didn't they expend the effort? Because she's working on her thesis or maybe because his wife just had triplets. Perhaps people came to depend on functionality that the author didn't ever expect them to use, or never even anticipated, and has never even heard' of. What we have to figure out his how to allow *those* packages to work with *these* applications no matter what.
IOW, enabling version requirement without strong API compatibility is the path to dependency hell.
I fully agree. However, this is a social issue, not a technical one. Everyone using the versioning system would need to understand the rules and be willing to follow them.
What is needed is a Real Versioning System. A system in which a library or application consists of: 1) files 2) a dependency link map When the system loads the package it links in the dependent modules. Then you can get a system where: Application A imports B v 1.1 and C v 2.3 and Application D imports B v 1.4 and C v 2.6. At least that's the holy grail as I see it. -jeff
Jeff Younker wrote:
Thats a great theory, but that's not how the real world works. Python packages are an ecology where there will be inconsistencies between different minor versions of the same package.
I'm not sure what you're arguing here. If you're saying that having a version management system won't make these sorts of problems go away, then I fully agree. But that's no reason not to have a verion management system. If you have one, then at least it provides a clear set of rules for people to try to adhere to, and helps show up when they aren't being adhered to, so that the problem can be fixed.
A legitimate bug fix may break behavior that other packages depend upon while correcting the behavior for others.
If it does that, then it will break other packages anyway, whether a versioning system is being used or not. So I would say it's introduced another bug which in turn needs to be fixed. The versioning system is not the cause of this and can't be blamed for it.
What is needed is a Real Versioning System. A system in which a library or application consists of:
1) files 2) a dependency link map
That sound like a dependency management system to me, not a version management system. They're different things, although dependency management can benefit from having a clear way of labelling versions.
When the system loads the package it links in the dependent modules. Then you can get a system where:
Application A imports B v 1.1 and C v 2.3
and
Application D imports B v 1.4 and C v 2.6.
Ummm... how is that any different from what I suggested? If in the Python context "loads" means "uses an import statement". -- Greg
Jeff Younker wrote:
That's good and fine for the situation where one application is being deployed on a Python installation, but that isn't a very realistic situation. There is usually more than one application/script/whatever running off of a given Python installation.
Here's an example of how you can get into an unworkable situation:
Application A depends on: library B v 1.1 library C v 2.3
Library B v 1.1 has no dependencies.
Now application D comes along. It depends upon: library B v 1.4
But library B v 1.4 depends upon library C v 2.6.
Application A *breaks* with library C v 2.6.
Which does the user give up? Application A (their storage management client) or application D (their monitoring agent)?
This sort of stuff happens all the time.
This is what a Versioning (with a capital V attempts to solve.)
No, versioning does not solve this at all, this has nothing to do with versioning. Dependency management does, and the two issues are mostly orthogonal. Installing multiple versions of the same package with versioning as an attempt to solve dependencies is broken. It has been used before, without any success.
Thats a great theory, but that's not how the real world works. Python packages are an ecology where there will be inconsistencies between different minor versions of the same package. A legitimate bug fix may break behavior that other packages depend upon while correcting the behavior for others. That's just the way stuff happens.
We can encourage 10,000 disparate developers to adhere to best practices, but it's fantasy to imagine that they will. Most of the people putting together packages are donating their 80% solutions to the rest of the world.
But most packages won't depend one on each other; most of the 80 % packages you are talking about are not meant to be used by other. Thinking versioning + side by side installation will solve the problem of breaking API is totally wrong. It just does not work. Please take a look at this page I mentioned before: http://www.mono-project.com/Assemblies_and_the_GAC It explains the problem very well, much better that I will ever be able to. Look at the paragraph "What should be installer in the GAC", the GAC being the system used in the .net ecosystem to maintain multiple version side by side and provide a system to serve the right library (assembly in .net terms) at the right application. Again, having a few packages with several versions is Ok. But enabling this as the general rule is path to hell from a maintenance/support point of view. cheers, David
On Tue, 2008-04-15 at 13:01 +1200, Greg Ewing wrote:
David Cournapeau wrote:
what is needed is a stable API for the used packages.
That's a nice ideal to aim for, but it's only achievable for old and mature packages.
I don't think so. It requires vigil on the part of the maintainer, but even young packages can have a sane versioning policy.
One could change the package name every time the API changes, but then *any* change to the API would make the package unusable by apps expecting an earlier version, even if the new API is backwards compatible.
Having said that, I just realised you can address that by putting in symlinks for the previous versions.
I don't think that's supported on Windows.
So, I hereby propose the following convention for naming of Python packages:
packagename_<version>
I don't know that this is an actual replacement for setuptools (rather a way to simplify setuptools), but I like the idea in general. However what I'd rather see is: site-packages/package/major_version It would keep the site-packages directory from becoming a giant mess and be more portable.
where <version> is a number or other identifier that is changed whenever the API changes in any way.
I think the convention is major.minor where minor releases are backwards-compatible and major releases aren't expected to be (but might be). Sometimes this is confusing as projects will do a major version release due to say, major code-refactoring that doesn't necessarily affect the API, so the API "version" isn't necessarily reflected in the package version. On the other hand, not using the higher version certainly won't break existing code either, so it's rather moot whether you *could* use the higher versioned software if your concern is not breaking an existing system. Regards, Cliff
Cliff Wells wrote:
I think the convention is major.minor where minor releases are backwards-compatible and major releases aren't expected to be (but might be).
AFAIK, that's the general rule, but python itself does not respect this convention, so I don't see this happening for python packages. cheers, David
On 15/04/2008, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
That's a nice ideal to aim for, but it's only achievable for old and mature packages.
One could change the package name every time the API changes, but then *any* change to the API would make the package unusable by apps expecting an earlier version, even if the new API is backwards compatible.
Is nobody but me seeing shades of Windows "DLL hell" in all of this? Seriously, the problems being discussed here seem very similar to those encountered in the past by Windows with differing versions of DLLs. A number of solutions have been tried, with varying degrees of success. Anyone looking to address this problem for Python, really should take a look at the DLL hell history in Windows, if only to see what types of approach didn't work. Paul.
Paul Moore wrote:
Is nobody but me seeing shades of Windows "DLL hell" in all of this?
Dll hell was caused because there was no versioning, and new dll overwrote older ones, while not being compatible. If we add a versioning checking, we won't have dll hell problem, but dependency hell, actually, which is not much better. As far as I understand, with .net, MS started to support multiple version (several runtimes could be installed at the same time), but it did not really solve the deployment issues.
Seriously, the problems being discussed here seem very similar to those encountered in the past by Windows with differing versions of DLLs. A number of solutions have been tried, with varying degrees of success. Anyone looking to address this problem for Python, really should take a look at the DLL hell history in Windows, if only to see what types of approach didn't work.
One of the recent approach I am aware of is the GAC (global assembly cache) for .net support. It enable multiple versions of the same assembly (more or less equivalent to a python module) to be installed at the same time: http://www.mono-project.com/Assemblies_and_the_GAC This document is actually very interesting (I've just read it while looking for a link for the GAC). Although aimed at .net development, it is almost entirely applicable to the problems we are talking about here. Interestingly, it recommends against having multiple versions of the same assembly in the GAC without committment to stable API. cheers, David
Paul Moore wrote:
Is nobody but me seeing shades of Windows "DLL hell" in all of this?
I think DLL Hell occurred partly because people *didn't* distinguish different API versions clearly with different file names. Also because programs were allowed to put shared stuff in the system directory with no form of reference counting. -- Greg
Gael Varoquaux wrote:
On Mon, Apr 14, 2008 at 01:30:14PM +1200, Greg Ewing wrote:
Gael Varoquaux wrote:
a second Python needs to be installed on top of the system Python to add modules to it.
Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood.
I am quite wary about these proposals, as well as the one environment per application ones.
In my experience, it is a fairly standard practice across all platforms to install independent application environments for any large, complex applications that come with a vendor support agreement. Admittedly, that isn't alot of Python applications, especially of the kind you're referring to where users are expected to add, remove, and intermix modules. However, that doesn't mean there aren't any Python apps that do fit the bill -- in fact, Enthought does alot of that. We want our apps to work even if the sysadmin upgrades the system libraries, Python, or what have you, or if the user installs multiple applications that have competing library requirements.
What you propose resembles very much to what MacOSX does, and MacOSX seems just so broken for Python. I don't use it, but I see on the different scientific-Python-related mailing lists how users have difficulties with MacOSX, and I have heard many people complain about this "feature".
As a per-application environment, I think it is bad, because it limits reuse. As I see things, applications should be able to have access to all the Python modules installed, to be able to use them, if they need. I get definitely see applications having more feature if some modules are installed (eg. Sphinx, which does syntax highlighting if pygments is installed).
I think this discussion is really going on because Python does not have good library-versioning support. What it needs is to get this, and not to get complete isolation, apart for some system-critical tasks, and only for these. AFAIK this is one of the problems setuptools is trying to solve. Not everybody is happy with the technical solution setuptools has come up, this only means it has to be improved.
IMO, this sort of multi-version thing only works if *everything* in the stack participates fully. I think the reason more things don't do this is because of the maintenance load it puts on developers / release managers to bump and synchronize versions apporpriately. -- Dave
Gael Varoquaux wrote:
On Mon, Apr 14, 2008 at 01:30:14PM +1200, Greg Ewing wrote:
Maybe the system should come with two pythons installed,
What you propose resembles very much to what MacOSX does... I see on the different scientific-Python-related mailing lists how users have difficulties with MacOSX
It differs from MacOSX, because MacOSX only comes with one Python installed. A lot of the problems go away if you install another one, but things aren't set up that way by default.
I think this discussion is really going on because Python does not have good library-versioning support.
That's probably true. I was wondering recently whether there should be some way to specify which version of a module you want when you import it, e.g. import fooble[3] which means you want a fooble with API version 3 or something compatible with it. -- Greg
Gael Varoquaux <gael.varoquaux@normalesup.org> writes:
On Mon, Apr 14, 2008 at 01:30:14PM +1200, Greg Ewing wrote:
Maybe the system should come with two pythons installed, one for use by the system and the other for users to add things to. Or at least be set up so that it appears that way -- they might share files under the hood.
I am quite wary about these proposals, as well as the one environment per application ones.
What you propose resembles very much to what MacOSX does, and MacOSX seems just so broken for Python. I don't use it, but I see on the different scientific-Python-related mailing lists how users have difficulties with MacOSX, and I have heard many people complain about this "feature".
As a per-application environment, I think it is bad, because it limits reuse. As I see things, applications should be able to have access to all the Python modules installed, to be able to use them, if they need. I get definitely see applications having more feature if some modules are installed (eg. Sphinx, which does syntax highlighting if pygments is installed).
Thank you. This is a major part of my concern. The Python environment should be reliably the same when invoked on a given system configuration, with egregious application-specific differences to *the instance itself* kept to a minimum. That's not to say that applications can't *override* the default instance; but such customisation should be kept to a minimum in order that dependencies, special cases, etc. are minimised. -- \ "What's another word for Thesaurus?" -- Steven Wright | `\ | _o__) | Ben Finney
On Sun, Apr 13, 2008 at 07:59:21AM -0400, Barry Warsaw wrote:
On Apr 12, 2008, at 9:41 PM, Stephen Waterbury wrote:
I used to always set up my own Python[s] in /usr/local and put that first in my PATH, but I have gotten lazy lately, and sometimes it will bite me. ;)
On Debian and derivatives (e.g. Ubuntu) you might have even more fun. They put /usr/local/lib/pythonX.Y/site-packages on the sys.path *of the system python*! This means that you can break your system Python by installing a version of Python from source and then distutil'ing things into there. Astoundingly, this is promoted as a feature.
I've reported bugs on this and had discussions with some of the Debian Python packaging folks. I'm hoping that we'll find a solution that doesn't collide with a from-source default Python build.
I think it would be really good if we could agree on a simple solution for this problem. However I haven't really seen an agreement come out of this thread. So far what (I think) some proposed solutions are: 1 Have a different path in /usr/local (not /usr/local/lib/pythonX.Y/site-packages) to augment the system sys.path with. 2 Make sure /usr/local is near the end of the sys.path so system apps won't break due to it (currently /usr/local/lib/pythonX.Y/site-packages appears before /usr/lib/pythonX.Y/site-packages on my system, not sure if this is normal and if not who's fault that is). 3 Don't support compiling your own python in /usr/local, do that in /opt. 4 Introduce a hidden python on the system which the system uses to run system applications that use python. The user would be unable to change that environment unless they really tried. (But surely then there are still discussions about which paths end up on sys.path for the user python and which paths are for locally compiled pythons, I might have missed something in this discussion). Personally I think the solution should be as simple as possible so I'd go for 1, 2 or 3. 3 will be difficult to teach people (definitely if they disagree) so that won't work, which leaves 1 and 2. Opinions? Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Floris Bruynooghe <floris.bruynooghe@gmail.com> writes:
I think it would be really good if we could agree on a simple solution for this problem. However I haven't really seen an agreement come out of this thread. So far what (I think) some proposed solutions are:
1 Have a different path in /usr/local (not /usr/local/lib/pythonX.Y/site-packages) to augment the system sys.path with.
2 Make sure /usr/local is near the end of the sys.path so system apps won't break due to it (currently /usr/local/lib/pythonX.Y/site-packages appears before /usr/lib/pythonX.Y/site-packages on my system, not sure if this is normal and if not who's fault that is).
These both appear to be addressing the issue that customisations (of the form "install a new Python package") in '/usr/local' can disrupt the behaviour of system tools using Python. So, here's my proposed solution: Provide a simple way for a Python program or instance to specify "don't have '/usr/local' in the 'sys.path'". Recommend that the system Python tools use this feature to ignore '/usr/local'. Continue using '/usr/local' for custom site-local packages. That is, if we're talking about the system Python tools being the unusual case ("don't want site-local custom packages"), then that's where the focus should be on allowing for that special case. Yes, I'm aware of 'sys.path.remove("/usr/local")'. Is that the simplest way to ensure the above behaviour? -- \ "If you wish to strive for peace of soul, then believe; if you | `\ wish to be a devotee of truth, then inquire." -- Friedrich | _o__) Nietzsche | Ben Finney ?
On Sat, Apr 19, 2008 at 09:05:12AM +1000, Ben Finney wrote:
Floris Bruynooghe <floris.bruynooghe@gmail.com> writes:
I think it would be really good if we could agree on a simple solution for this problem. However I haven't really seen an agreement come out of this thread. So far what (I think) some proposed solutions are:
1 Have a different path in /usr/local (not /usr/local/lib/pythonX.Y/site-packages) to augment the system sys.path with.
2 Make sure /usr/local is near the end of the sys.path so system apps won't break due to it (currently /usr/local/lib/pythonX.Y/site-packages appears before /usr/lib/pythonX.Y/site-packages on my system, not sure if this is normal and if not who's fault that is).
These both appear to be addressing the issue that customisations (of the form "install a new Python package") in '/usr/local' can disrupt the behaviour of system tools using Python.
So, here's my proposed solution:
Provide a simple way for a Python program or instance to specify "don't have '/usr/local' in the 'sys.path'". Recommend that the system Python tools use this feature to ignore '/usr/local'. Continue using '/usr/local' for custom site-local packages.
Seems hard. While it might be feasable for apps (who will live in known directories) it becomes a lot harder when random applications ship python scripts in some random directory, e.g. cgi-bin or so. It basically means that a large amount of packages in a distro will always be broken to this effect until somone stumbles over this particular bug for this particular package and happens to be good enough to file that bug. It's the same problem as with requiring system packages to use a separate python installation with a more strictly pinned sys.path.
That is, if we're talking about the system Python tools being the unusual case ("don't want site-local custom packages"), then that's where the focus should be on allowing for that special case.
So most people here don't like Debian's and SUSE's behaviour, that's sad but anyway, which means 1 above --introducing a new Python endorsed site-packages dir in /usr/local-- seems unlikely. Since 2 above --making sure /usr/local is at the end of sys.path-- can be done entirely by the distributions who like /usr/local on sys.path this seems like the best option. In Debian this is currently implemented by adding /usr/local/ to the prefixes of site.py, ensuring that .pth files in /usr/local are still usable. So just to err on the side of safety: can anyone think of a reason that having /usr/local/lib/pythonX.Y/site-packages/ at the end of sys.path in this way could cause problems? Thanks Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
At 03:26 PM 4/12/2008 -0700, Cliff Wells wrote:
On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote:
At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote:
PATH is *supposed* to affect applications.
It affects which application you should run, not which interpreter you run the application with.
I think that's splitting hairs and pretty much just made up to defend your position.
Anybody who thinks they can globally change which Python interpreter they're running their scripts with is cruising for a serious bruising. Better they discover their error sooner than later. Have you ever wondered why Red Hat has that stupid 'python2' executable? It's because they used #! lines pointing to 'python' rather than referencing a specific *version* of Python... and had to pay for it for years thereafter. (That's also, by the way, why easy_install also always installs a versioned executable name for itself.)
On Sat, 2008-04-12 at 19:54 -0400, Phillip J. Eby wrote:
At 03:26 PM 4/12/2008 -0700, Cliff Wells wrote:
On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote:
At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote:
PATH is *supposed* to affect applications.
It affects which application you should run, not which interpreter you run the application with.
I think that's splitting hairs and pretty much just made up to defend your position.
Anybody who thinks they can globally change which Python interpreter they're running their scripts with is cruising for a serious bruising. Better they discover their error sooner than later.
If you are on RedHat, yes. On Gentoo, SourceMage, Foresight, etc the system is expected to do a "rolling" upgrade. So what you call an "error" is what is thought of as a "superior method" by many others.
Have you ever wondered why Red Hat has that stupid 'python2' executable? It's because they used #! lines pointing to 'python' rather than referencing a specific *version* of Python... and had to pay for it for years thereafter.
But these are system apps we are discussing, not user apps. System apps are not managed by setuptools, rather they are managed by the system's package manager. I agree they should point to the "system" python, whatever the system decides that should be, and the system should be responsible any upgrading to be done. In fact, it's specifically *because* of many distros depending so heavily on a particular version of Python that userspace Python apps need more flexibility in this area. /usr/bin/python is quite likely going to be out-of-date before the system as a whole is (or in my case, the other way around, as Gentoo moved to 2.5 before all my applications had). Either way, if they system Python and the userspace Python requirements get out of sync, it's clearly the userspace that's going to have to be flexible.
(That's also, by the way, why easy_install also always installs a versioned executable name for itself.)
Actually, if easy_install did something like "#!/usr/bin/python2.4" I think I'd have less of an issue. It would allow me to upgrade the system Python (which on Gentoo, for instance, you can safely do) while allowing legacy apps to continue to function until they can be upgraded. In fact, this would almost incidentally lead people down the right road to fixing it the way you suggest since they will certainly have to easy_install their required libraries under the new interpreter. Regards, Cliff
On Sat, 2008-04-12 at 19:54 -0400, Phillip J. Eby wrote:
(That's also, by the way, why easy_install also always installs a versioned executable name for itself.)
So if setuptools can rely on this name, why doesn't it use it in the shebang line? Regards, Cliff
On Sat, Apr 12, 2008 at 03:26:52PM -0700, Cliff Wells wrote:
On Sat, 2008-04-12 at 17:53 -0400, Phillip J. Eby wrote:
At 12:30 PM 4/12/2008 -0700, Cliff Wells wrote:
PATH is *supposed* to affect applications.
It affects which application you should run, not which interpreter you run the application with.
I think that's splitting hairs and pretty much just made up to defend your position.
Nope, his statement is very correct. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
participants (14)
-
Barry Warsaw -
Ben Finney -
Cliff Wells -
Daniel Krech -
Dave Peterson -
David Cournapeau -
Floris Bruynooghe -
Gael Varoquaux -
Greg Ewing -
Jan Matejek -
Jeff Younker -
Paul Moore -
Phillip J. Eby -
Stephen Waterbury