Remove site-packages?!? [was: [Distutils] PEP 376 - from PyPM's point of view]
Michael Foord wrote:
I agree. People with versioning issues *should* be using virtualenv.
Tarek Ziadé replied (and several people later agreed)
Let's remove site-packages from Python then.
What about those people *without* versioning issues? I have no qualms suggesting that package management programs avoid the use of site-packages. Such programs do need to cater to edge cases. But a single drop-it-in directory works great for the vast majority of *my* needs; requiring me to drink the Kool-Aid from a specific package management system just to get access to any add-ons -- even those I wrote myself in pure python -- would be a huge step backwards. -jJ
At 04:07 PM 7/18/2009 -0400, Jim Jewett wrote:
IIUC, the new "user" directories supported by the 'site' module would fill this purpose. That is, rather than having a site-wide packages directory, there'd just be the user-specific package directories. (That having been said, I'm not actually in favor of the idea.)
On Sat, 18 Jul 2009 16:07:39 -0400, Jim Jewett <jimjjewett@gmail.com> wrote:
I'm just wondering how that is possible? Given that a package management system (pythonpkgmgr - for example) is a tool to assist in the management of packages in site-packages. If you take away the drop in directory (site-packages), effectively you're taking away the place for system drop in packages. I can't see the point in that.
Where you keep your packages is entirely up to you. Even with the Python Package Manager, you can still have unversioned packages in project directories. That doesn't change. And you can still have unversioned packages in site-packages. So I can't really see that having a Package Manager program takes your use case scenerio backwards because it wouldn't really change it. (repeating)
But a single drop-it-in directory works great for the vast majority of *my* needs;
That's exactly what site-packages is..... So it isn't clear why you want to remove the thing that you are advocating works so great.... David
David Lyon wrote:
So it isn't clear why you want to remove the thing that you are advocating works so great....
Jim was quoting someone *else* that had suggested removing it (I'm not sure how serious the original suggestion actually was though). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Sun, 19 Jul 2009 at 16:07, Nick Coghlan wrote:
It was Tarek, and I'm pretty sure it was in the nature of what rhetoricians (among others) call "reductio ad absurdum". But Tarek would have to let us know for sure, since it seems from subsequent discussion that the notion is not _completely_ absurd, just mostly :) --David
On Sun, Jul 19, 2009 at 3:08 PM, R. David Murray<rdmurray@bitdance.com> wrote:
Yes, well, maybe we should go back to Python-ideas to continue this discussion, :) What made me say that, is the unecessary complexity of the situation because we have right now : "Python the application" v.s. "Applications that uses Python the interpreter" So we have: A - a drop-it-in directory were we can put packages and modules B - installers that adds automatically packages and modules in the drop-it-in directory C - applications that uses these installers to install themselves and their dependencies in the drop-it-in directory. D - a tool like virtualenv that duplicates a Python installation so this drop-in-in directory is not shared globally C - installation documentation that tells you to use virtualenv to create a custom environment so you are sure not to break dependencies. (But you can break them nevertheless because using virtualenv is not enforced :) ) E - OS packagers that installs all distributions in the drop-it-in directory (or a similar place) So if you remove the global site-packages, "Python the application" dissapear in favor of "Python the interpreter", and people have to point a local, non-shared drop-it-in directory to run any python application that uses extra modules: 1- For Jim's case, the per-user site packages (PEP 370) can be used 2- For every application it can be a local directory loaded in sys.path at startup 3- For OS packagers a single directory where they maintain a collection of packages and modules for their system. But unfortunately, the problem remains because of (3) : OS packagers will fight against applications (2) that deal with their own dependencies to avoid having duplicate packages in their system. That's why a shared space, not loaded at startup by site.py, that knows how to handle multiple versions for the same distribution would be required. Every script (eg application) would decide which version to use, or use the default version by default. And a smart PEP 302 loader/importer would get the right versions. Regards Tarek
On Sun, 19 Jul 2009 17:15:44 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Well now I completely understand the jist of what you are saying.. And I completely agree with the direction of your comments. This problem is being solved within the pythonpkgmgr project. As we are allowing a distinction between operating system packages and developer packages. I only have ubuntu, and not OS-X at the moment, but let me explain with that. The operating system can install packages into /usr/lib/pythonX.X/ site-packages. The develeloper can install packages into /usr/local/pythonX.x/ site-packages. Therefore providing two seperate areas for packages to reside and eliminating the chance for serious conflict. The main problem imho has been that easy_install by default has installed to /usr/lib/pythonX.X/site-packages when in fact that really is an operating system controlled directory. So naughty naughty to setuptools.... That's why in pythonpkgmgr, we're moving to an entirely different place where there is no chance of conflict with the os package manager and python packages can be installed without su privelages. If the user wishes to install o/s packages that's fine and if they wish to install developer packages that's fine too. The saving grace here is the gui that remembers the appropriate command line switches that are needed to direct this behaviour because it doesn't happen by default in easy_install. Under ubuntu, it seems to be /usr/local/pythonX.X/site-packages and I am checking other posix operating systems and trying to work out what the equivilents might be. When I have it working.. I will let you know.. I'm just trying to work out how to get the permissions going correctly. David
On Sun, Jul 19, 2009 at 7:13 PM, David Lyon<david.lyon@preisshare.net> wrote: [snip]
I've witnessed this a few times... and I completely agree with that statement. :-)
Perhaps I'm not reading what you're saying correctly. Doesn't /usr/local/pythonX.x/site-packages require su privileges to install? FWIW, the equivalent of /usr/local/pythonX.x/site-packages on Mac OS X seems to be /Library/Python/X.x/site-packages -John
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Szakmeister wrote:
That behvior isn't setuptools' fault: it is the behavior of *any* distutils-derived app. "OS controlled directory" is not a concept which the distutils honors at all: $prefix/lib/pytoonX.Y/site-packages is the *intended* installation location for any modules not shipped with Python. Consider the case where the user has built a separate python (e.g., to avoid conflicting with the OS version) in /opt/PythonX.y: are you actually saying that distutils / setuptools should somehow know the difference betweeen /usr/lib/pythonX.y/site-packages and /opt/PythonX.Y/lib/pythonX.Y/site-packages? Or are you asserting that it is somehow invalid for distutils / setuptools to install things into /opt/PythonX.Y/lib/pythonX.y/site-packages, which is certainly not an "OS controlled directory." Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKZf3V+gerLs4ltQ4RAud7AJ9GG9NY4UaKpQ5bBIQ+N6tztPVJSgCdEEIa DUugfdzvBYgPtaas9c+fTTg= =gvLY -----END PGP SIGNATURE-----
On Tue, 21 Jul 2009 13:41:41 -0400, Tres Seaver <tseaver@palladion.com> wrote:
Ok - so naughty distutils...
It already does. If you run the python in /opt/pythonX.X you'll get a different value for sys.path than when you run the other python from /usr/lib/pythonX.X. Try it and see for yourself.. That's because sys.path is calculated relatively within site.py depending on where the interpretor is run from.
Not at all. All I'm saying is that after so many years of python under linux and operating system we should have observed the common use cases and learned a thing or two. If you hadn't noticed, there is mayhem in package installation because the operating system tries to install its packages in the same place as setuptools does in development mode. That has got to be asking for trouble.. When I go into python on ubuntu I see there is /usr/local/pythonX.X/lib/ site-packages and I'm wondering why the hubba setuptools/distutils doesn't put packages there by default. That would solve a lot of problems. Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S. Wrapping up.. most of the functionality that I'm asking for is already there. So I don't require anything extra. I don't want things taken out. All I'm suggesting is that we use what we already have a bit smarter. David
On Jul 21, 2009, at 7:38 PM, David Lyon wrote:
Uh guys, I'm not sure if anyone here noticed, but Debian and Ubuntu have switched to install their distribution-supplied python libraries into: /usr/lib/pythonX.Y/lib/dist-packages and distutils by default will install into /usr/local/lib/pythonX.Y/dist-packages starting with python 2.6. See: http://lists.debian.org/debian-devel/2009/02/msg00431.html Since that email says "Discussed this with Barry Warsaw and Martin v. Loewis", I'd assume this change would be more widely known in the distutils/python-dev community, but apparently not?? James
On Tue, 21 Jul 2009 21:46:11 -0400, James Y Knight <foom@fuhm.net> wrote:
Forgive my ignorance... I'm working with 2.5 on ubuntu and didn't think to try 2.6... It's good to know that the problem is fixed in 2.6 but I still need to make things work for pythonpkgmgr on ubuntu py2.5 as 2.5 is the default for ubuntu 8.10. Take care David
James Y Knight wrote:
Debian has a long history of doing this different, so it's not much of a surprise. They also apply such changes to Python packages. However, all of this is non-standard and will cause problems with tools that rely on the standard site-packages/ location. Such changes should be discouraged. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 22 2009)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Jul 22, 2009, at 4:49 AM, M.-A. Lemburg wrote:
And yet, the change seems to have some strong reasoning, solves the problem discussed in this thread, and was apparently discussed and approved of by some core python developers before being implemented. It seems a bit foolish to me to thus just dismiss it as "evil debian being different"... If anything it seems like it's a failure of the Python project to make easily deployable software, compounded with a failure of communication within the python community. James
James Y Knight wrote:
Maybe I've misunderstood some important detail, but how will their "change" help with anything other than making their distribution a non-standard Python installation ? distutils allows for a great deal of flexibility. For some reason, this does not appear to be known to a larger audience. I can only recommend reading Greg's great write-up about the end-user perspective of installing Python modules: http://docs.python.org/install/ A little known fact is that distutils can easily be customized using config files: http://docs.python.org/install/#distutils-configuration-files Together with the PYTHONHOME and PYTHONPATH environment variables setup in your .bashrc, this gives you full flexibility regarding the Python package setup and permits setting up private Python installations, parallel Python installations, Python installations sharing packages, etc. etc. It really depends a lot on what you want to achieve. There's no one fits all configuration. Using a non-standard Python installation as system Python is certainly not a good approach to solving anything. It only makes it harder for users having to fight problems arising out of this. You have the same problem in other areas as well. A prominent example being Microsoft's tendency (in the past) to tweak standards to better meet their requirements.
I lost you there. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 22 2009)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, Jul 22, 2009 at 5:22 PM, M.-A. Lemburg<mal@egenix.com> wrote:
I think I'm a little confused, too, because Python supports the /usr|/usr/local separation just fine (setup.py install --prefix=/usr/local). It seems like it's also using "dist-packages" instead of "site-packages". That part, I don't understand at all--distribution packages should go in /usr/lib/pythonx.y/site-packages, and "site" packages go in /usr/local/pythonx.y/site-packages; /usr/local *itself* means "non-distribution site-installed stuff". If that's what you're referring to, then at least on first impression I agree. -- Glenn Maynard
On Wed, Jul 22, 2009 at 07:08:26PM -0400, Glenn Maynard wrote:
The way Debian does this isn't that stupid. It solves the problem of the sysadmin intalling Python distibutions for the system Python without writing in the system locations, which is a very reasonable thing to do. Before Debian used dist-packages, /usr/local/lib/pythonX.Y/site-packages was added to the sys.path of the system Python and this was the location where a sysadmin should add packages (although distutils did not default to that location, so it wasn so it was still easy to mess up the system package management). But this caused trouble for people who installed their own Python in /usr/local since now you have the same /usr/local/lib/pythonX.Y/site-packages shared by 2 interpreters. To solve this the system python moved to dist-packages instead, this allows the local admin location to use /usr/local/lib/pythonX.Y/dist-packages which does not conflict with locally installed interpreters. Together with this they changed distutils to install to /usr/local/lib/pythonX.Y/dist-packages by default, so that a sysadmin would have to go out of their way to mess up the system package management. This is a pretty neat solution to the problem. But the weird thing (IMHO) is that distutils-sig completely refuses to see this requirement. Generally it seems accepted that installing modules in the system location (/usr/) is a bad thing, but last time it got discussed there was a point blank refusal to accept that the local admin needs a location in to install packages in. I think it would be great if Python instead provided a general guideline or best practice for this situation which would be explained in the documentation. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Hi Floris, That's exactly how I see it and i totally agree. My contribution is to make a Package Manager Gui that tries to be supportive of what you describe so well. If i have any complaint about the state of affairs it would only be that it takes a newcomer such a long time (months) to fully understand what should be a 5 minute thing as you so well describe below.
Floris Bruynooghe writes:
[dist-packages] is a pretty neat solution to the problem.
To what problem? I admit I am no expert on Python packaging, but my experience with XEmacs suggests that this is the distro trying to help with a *set* of problems that the user/sysadmins really should be handling themselves. There are varied requirements that lead to a need for yet another place to install modules, but there's no guarantee that a given user/sysadmin has *just one*. With the Emacsen, often in trying to satisfy several requirements they end up with conflicts (different libraries in the site-lisp that are compiled for, or even require, different versions of (X)Emacs or even trying to deal with Emacs and XEmacs simultaneously). And it's a pain (as an XEmacs maintainer) to have to field questions about why things don't work in a set up I didn't design, and don't understand the rationale for.
Yeah, so do we all, but I suspect that for Python, like the Emacsen, there is *no* general guideline. You're just asking for various kinds of trouble, all of which require expert attention, by installing a command that is invoked as "python" (as opposed to "pythonX.Y") in /usr/local/bin unless you intend it to be a complete substitute for the system python. And that itself is a serious tour de force (as anybody who remembers Red Hat's love affair with Python 1.5.2 will agree, I suspect). As far as I can see, *in general* /usr/local/lib/python/site-packages is the right place to install local packages for use with the system python, and no other. That doesn't mean that there aren't *other* valid uses for that location, just that I see no general rule for supporting *all* of them at the same time.
On Wed, 22 Jul 2009 23:22:56 +0200, "M.-A. Lemburg" <mal@egenix.com> wrote:
The Debian/ubuntu distribution isn't non-standard. If anything I'd prefer to suggest that it is in many ways "a standard" Here's a sys.pth from a mac... ['', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload', '/Users/david/.local/lib/python2.6/site-packages', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages'] You can see that there are many choices along sys.path for installing packages.
People forget command lines - that's why.
It's good documentation of course. Cheers to Greg but the old method is so tedious. That really is the hard way. pythonpkgmgr offers a much easier solution by wrapping easy_install and/or pip. You just type in parts of the package name into a search box, click [search], a search of pypi is done, click [install] and your package is downloaded and installed. It's a much more modern way of doing package installation and requires absolutely no typing on a command line.
Perphaps. But it seems only for advanced users.. and I couldn't figure out on the face of it what advantage it would have. Take care David
David Lyon wrote:
It's not just Debian mangling their Python installation. Many other Linux distros do the same. Keeping up with all those changes has become so hard, that we have chosen to always build our projects with private (regular) Python installations on Linux and Macs. Disk space is cheap and so is bandwidth. Compared to the hours spent on resolving issues with system provided Python installations, it's a no-brainer.
No, not really. The above is perfectly standard. The directories on sys.path are setup by Python itself and make sure that it can find the stdlib modules. Those entries are not meant to be used as installation targets. Python 2.6 has two standard places for installing packages: 1. In the stdlib site-packages/ subdir 2. In the user home dir's .local/lib/python2.6/site-packages dir The second is a new feature in 2.6.
I think that's the main problem in all this... people just keep on complaining regardless of what you give them. If people don't read documentation, are not willing to at least spend a few minutes on considering what they really want to do and then shout out loud when Python doesn't do what they thought it should do (without telling Python, of course), there's little hope.
You define once and for all where you want your Python packages to install themselves when typing "python setup.py install". After that's done, you can forget about all this and move on to get some real work done ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 23 2009)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Jul 23, 2009 at 10:34:30AM +0200, M.-A. Lemburg wrote:
And is missing a 3rd one. The sysadmin who wants to install packages globally has to use the first option here, but that conflicts with the system package management on UNIXes. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
"M.-A. Lemburg" <mal@egenix.com> writes:
Before this discussion descends further into circular speculation on the Debian Python developers's intentions, perhaps someone should ask them directly: <URL:http://lists.debian.org/debian-python/>. -- \ “Let others praise ancient times; I am glad I was born in | `\ these.” —Ovid (43 BCE–18 CE) | _o__) | Ben Finney
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Lyon wrote:
On Tue, 21 Jul 2009 13:41:41 -0400, Tres Seaver <tseaver@palladion.com> wrote:
You missed my point: I know that sys.prefix etc. is set based on the location of the executable. You were complaining that setuptools / distutils wants to put files in an "OS-controlled directory": I was asking how Python was supposed to know the difference between /usr/lib/python2.x/site-packages (presumably "OS-controlled") and /opt/Python2.x/lib/python2.x/site-packages (definitely not "OS-controlled"). The solution is to avoid doing distutils / setuptools operations as root, *ever*, and instead use either an alternate install location (such as /opt/Python2.x) or use something like virtualenv, which provides a "safe" sandbox for installing stuff.
Users who install anything into /usr/lib/python2.x in "development" mode has already invited all the trouble they can handle into the house.
That is precisely how I think of virtualenv: it sits on top of distutils / setuptools, and makes it sane to do Python development without compromising the core system. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKZyGz+gerLs4ltQ4RAupxAJ95RZQJm6t1FtcFY+DMyHHmwRjmSACglo+W TehyHzxUwHQ5LfdibJsgUcA= =nf13 -----END PGP SIGNATURE-----
On Wed, 22 Jul 2009 10:26:59 -0400, Tres Seaver <tseaver@palladion.com> wrote:
You were complaining that setuptools / distutils wants to put files in an "OS-controlled directory":
I complain about many things - so what.
Well, at an interpreter level it doesn't, and I don't expect it to.
Have you tried this ? When I try to run it without su privelages setuptools crashes... (ubuntu 8.10) ------------------------------------------------------------------------------------------------ Running installer ... /usr/bin/easy_install "/media/Elements/Pythn Packages/html5lib-0.10.zip" ERRORS: Traceback (most recent call last): File "/usr/bin/easy_install", line 5, in <module> from pkg_resources import load_entry_point ImportError: No module named pkg_resources Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 78, in apport_excepthook report_file = open(pr_filename, 'wt') IOError: [Errno 13] Permission denied: '/var/crash/_usr_bin_easy_install.1000.crash' Original exception was: Traceback (most recent call last): File "/usr/bin/easy_install", line 5, in <module> from pkg_resources import load_entry_point ImportError: No module named pkg_resources ------------------------------------------------------------------------------------------------ I can only get it to work with sudo.
For me I prefer using a tool with a GUI interface. Take care David
On Wed, Jul 22, 2009 at 10:26 AM, Tres Seaver<tseaver@palladion.com> wrote:
autoconf--for all its warts--solved this ages ago. On installation, you specify what prefix you want to install stuff into; if you don't, you get a sensible default (/usr/local). If you're creating an OS package, or if installing directly into /usr is otherwise really what you really want to do, you specify --prefix=/usr. The Python core doesn't need to know anything about this; it just needs to have a sys.path that includes both /usr/lib/... and /usr/local/lib/.... This works for everything else--C headers, libraries, manpages, shared libraries. Why is Python special?
This is nasty. I should certainly be able to globally install packages on my own, without creating a whole new Python installation or other hacks, just like I can install anything else into /usr/local. -- Glenn Maynard
At 04:07 PM 7/18/2009 -0400, Jim Jewett wrote:
IIUC, the new "user" directories supported by the 'site' module would fill this purpose. That is, rather than having a site-wide packages directory, there'd just be the user-specific package directories. (That having been said, I'm not actually in favor of the idea.)
On Sat, 18 Jul 2009 16:07:39 -0400, Jim Jewett <jimjjewett@gmail.com> wrote:
I'm just wondering how that is possible? Given that a package management system (pythonpkgmgr - for example) is a tool to assist in the management of packages in site-packages. If you take away the drop in directory (site-packages), effectively you're taking away the place for system drop in packages. I can't see the point in that.
Where you keep your packages is entirely up to you. Even with the Python Package Manager, you can still have unversioned packages in project directories. That doesn't change. And you can still have unversioned packages in site-packages. So I can't really see that having a Package Manager program takes your use case scenerio backwards because it wouldn't really change it. (repeating)
But a single drop-it-in directory works great for the vast majority of *my* needs;
That's exactly what site-packages is..... So it isn't clear why you want to remove the thing that you are advocating works so great.... David
David Lyon wrote:
So it isn't clear why you want to remove the thing that you are advocating works so great....
Jim was quoting someone *else* that had suggested removing it (I'm not sure how serious the original suggestion actually was though). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Sun, 19 Jul 2009 at 16:07, Nick Coghlan wrote:
It was Tarek, and I'm pretty sure it was in the nature of what rhetoricians (among others) call "reductio ad absurdum". But Tarek would have to let us know for sure, since it seems from subsequent discussion that the notion is not _completely_ absurd, just mostly :) --David
On Sun, Jul 19, 2009 at 3:08 PM, R. David Murray<rdmurray@bitdance.com> wrote:
Yes, well, maybe we should go back to Python-ideas to continue this discussion, :) What made me say that, is the unecessary complexity of the situation because we have right now : "Python the application" v.s. "Applications that uses Python the interpreter" So we have: A - a drop-it-in directory were we can put packages and modules B - installers that adds automatically packages and modules in the drop-it-in directory C - applications that uses these installers to install themselves and their dependencies in the drop-it-in directory. D - a tool like virtualenv that duplicates a Python installation so this drop-in-in directory is not shared globally C - installation documentation that tells you to use virtualenv to create a custom environment so you are sure not to break dependencies. (But you can break them nevertheless because using virtualenv is not enforced :) ) E - OS packagers that installs all distributions in the drop-it-in directory (or a similar place) So if you remove the global site-packages, "Python the application" dissapear in favor of "Python the interpreter", and people have to point a local, non-shared drop-it-in directory to run any python application that uses extra modules: 1- For Jim's case, the per-user site packages (PEP 370) can be used 2- For every application it can be a local directory loaded in sys.path at startup 3- For OS packagers a single directory where they maintain a collection of packages and modules for their system. But unfortunately, the problem remains because of (3) : OS packagers will fight against applications (2) that deal with their own dependencies to avoid having duplicate packages in their system. That's why a shared space, not loaded at startup by site.py, that knows how to handle multiple versions for the same distribution would be required. Every script (eg application) would decide which version to use, or use the default version by default. And a smart PEP 302 loader/importer would get the right versions. Regards Tarek
On Sun, 19 Jul 2009 17:15:44 +0200, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Well now I completely understand the jist of what you are saying.. And I completely agree with the direction of your comments. This problem is being solved within the pythonpkgmgr project. As we are allowing a distinction between operating system packages and developer packages. I only have ubuntu, and not OS-X at the moment, but let me explain with that. The operating system can install packages into /usr/lib/pythonX.X/ site-packages. The develeloper can install packages into /usr/local/pythonX.x/ site-packages. Therefore providing two seperate areas for packages to reside and eliminating the chance for serious conflict. The main problem imho has been that easy_install by default has installed to /usr/lib/pythonX.X/site-packages when in fact that really is an operating system controlled directory. So naughty naughty to setuptools.... That's why in pythonpkgmgr, we're moving to an entirely different place where there is no chance of conflict with the os package manager and python packages can be installed without su privelages. If the user wishes to install o/s packages that's fine and if they wish to install developer packages that's fine too. The saving grace here is the gui that remembers the appropriate command line switches that are needed to direct this behaviour because it doesn't happen by default in easy_install. Under ubuntu, it seems to be /usr/local/pythonX.X/site-packages and I am checking other posix operating systems and trying to work out what the equivilents might be. When I have it working.. I will let you know.. I'm just trying to work out how to get the permissions going correctly. David
On Sun, Jul 19, 2009 at 7:13 PM, David Lyon<david.lyon@preisshare.net> wrote: [snip]
I've witnessed this a few times... and I completely agree with that statement. :-)
Perhaps I'm not reading what you're saying correctly. Doesn't /usr/local/pythonX.x/site-packages require su privileges to install? FWIW, the equivalent of /usr/local/pythonX.x/site-packages on Mac OS X seems to be /Library/Python/X.x/site-packages -John
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Szakmeister wrote:
That behvior isn't setuptools' fault: it is the behavior of *any* distutils-derived app. "OS controlled directory" is not a concept which the distutils honors at all: $prefix/lib/pytoonX.Y/site-packages is the *intended* installation location for any modules not shipped with Python. Consider the case where the user has built a separate python (e.g., to avoid conflicting with the OS version) in /opt/PythonX.y: are you actually saying that distutils / setuptools should somehow know the difference betweeen /usr/lib/pythonX.y/site-packages and /opt/PythonX.Y/lib/pythonX.Y/site-packages? Or are you asserting that it is somehow invalid for distutils / setuptools to install things into /opt/PythonX.Y/lib/pythonX.y/site-packages, which is certainly not an "OS controlled directory." Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKZf3V+gerLs4ltQ4RAud7AJ9GG9NY4UaKpQ5bBIQ+N6tztPVJSgCdEEIa DUugfdzvBYgPtaas9c+fTTg= =gvLY -----END PGP SIGNATURE-----
On Tue, 21 Jul 2009 13:41:41 -0400, Tres Seaver <tseaver@palladion.com> wrote:
Ok - so naughty distutils...
It already does. If you run the python in /opt/pythonX.X you'll get a different value for sys.path than when you run the other python from /usr/lib/pythonX.X. Try it and see for yourself.. That's because sys.path is calculated relatively within site.py depending on where the interpretor is run from.
Not at all. All I'm saying is that after so many years of python under linux and operating system we should have observed the common use cases and learned a thing or two. If you hadn't noticed, there is mayhem in package installation because the operating system tries to install its packages in the same place as setuptools does in development mode. That has got to be asking for trouble.. When I go into python on ubuntu I see there is /usr/local/pythonX.X/lib/ site-packages and I'm wondering why the hubba setuptools/distutils doesn't put packages there by default. That would solve a lot of problems. Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S. Wrapping up.. most of the functionality that I'm asking for is already there. So I don't require anything extra. I don't want things taken out. All I'm suggesting is that we use what we already have a bit smarter. David
On Jul 21, 2009, at 7:38 PM, David Lyon wrote:
Uh guys, I'm not sure if anyone here noticed, but Debian and Ubuntu have switched to install their distribution-supplied python libraries into: /usr/lib/pythonX.Y/lib/dist-packages and distutils by default will install into /usr/local/lib/pythonX.Y/dist-packages starting with python 2.6. See: http://lists.debian.org/debian-devel/2009/02/msg00431.html Since that email says "Discussed this with Barry Warsaw and Martin v. Loewis", I'd assume this change would be more widely known in the distutils/python-dev community, but apparently not?? James
On Tue, 21 Jul 2009 21:46:11 -0400, James Y Knight <foom@fuhm.net> wrote:
Forgive my ignorance... I'm working with 2.5 on ubuntu and didn't think to try 2.6... It's good to know that the problem is fixed in 2.6 but I still need to make things work for pythonpkgmgr on ubuntu py2.5 as 2.5 is the default for ubuntu 8.10. Take care David
James Y Knight wrote:
Debian has a long history of doing this different, so it's not much of a surprise. They also apply such changes to Python packages. However, all of this is non-standard and will cause problems with tools that rely on the standard site-packages/ location. Such changes should be discouraged. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 22 2009)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Jul 22, 2009, at 4:49 AM, M.-A. Lemburg wrote:
And yet, the change seems to have some strong reasoning, solves the problem discussed in this thread, and was apparently discussed and approved of by some core python developers before being implemented. It seems a bit foolish to me to thus just dismiss it as "evil debian being different"... If anything it seems like it's a failure of the Python project to make easily deployable software, compounded with a failure of communication within the python community. James
James Y Knight wrote:
Maybe I've misunderstood some important detail, but how will their "change" help with anything other than making their distribution a non-standard Python installation ? distutils allows for a great deal of flexibility. For some reason, this does not appear to be known to a larger audience. I can only recommend reading Greg's great write-up about the end-user perspective of installing Python modules: http://docs.python.org/install/ A little known fact is that distutils can easily be customized using config files: http://docs.python.org/install/#distutils-configuration-files Together with the PYTHONHOME and PYTHONPATH environment variables setup in your .bashrc, this gives you full flexibility regarding the Python package setup and permits setting up private Python installations, parallel Python installations, Python installations sharing packages, etc. etc. It really depends a lot on what you want to achieve. There's no one fits all configuration. Using a non-standard Python installation as system Python is certainly not a good approach to solving anything. It only makes it harder for users having to fight problems arising out of this. You have the same problem in other areas as well. A prominent example being Microsoft's tendency (in the past) to tweak standards to better meet their requirements.
I lost you there. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 22 2009)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, Jul 22, 2009 at 5:22 PM, M.-A. Lemburg<mal@egenix.com> wrote:
I think I'm a little confused, too, because Python supports the /usr|/usr/local separation just fine (setup.py install --prefix=/usr/local). It seems like it's also using "dist-packages" instead of "site-packages". That part, I don't understand at all--distribution packages should go in /usr/lib/pythonx.y/site-packages, and "site" packages go in /usr/local/pythonx.y/site-packages; /usr/local *itself* means "non-distribution site-installed stuff". If that's what you're referring to, then at least on first impression I agree. -- Glenn Maynard
On Wed, Jul 22, 2009 at 07:08:26PM -0400, Glenn Maynard wrote:
The way Debian does this isn't that stupid. It solves the problem of the sysadmin intalling Python distibutions for the system Python without writing in the system locations, which is a very reasonable thing to do. Before Debian used dist-packages, /usr/local/lib/pythonX.Y/site-packages was added to the sys.path of the system Python and this was the location where a sysadmin should add packages (although distutils did not default to that location, so it wasn so it was still easy to mess up the system package management). But this caused trouble for people who installed their own Python in /usr/local since now you have the same /usr/local/lib/pythonX.Y/site-packages shared by 2 interpreters. To solve this the system python moved to dist-packages instead, this allows the local admin location to use /usr/local/lib/pythonX.Y/dist-packages which does not conflict with locally installed interpreters. Together with this they changed distutils to install to /usr/local/lib/pythonX.Y/dist-packages by default, so that a sysadmin would have to go out of their way to mess up the system package management. This is a pretty neat solution to the problem. But the weird thing (IMHO) is that distutils-sig completely refuses to see this requirement. Generally it seems accepted that installing modules in the system location (/usr/) is a bad thing, but last time it got discussed there was a point blank refusal to accept that the local admin needs a location in to install packages in. I think it would be great if Python instead provided a general guideline or best practice for this situation which would be explained in the documentation. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
Hi Floris, That's exactly how I see it and i totally agree. My contribution is to make a Package Manager Gui that tries to be supportive of what you describe so well. If i have any complaint about the state of affairs it would only be that it takes a newcomer such a long time (months) to fully understand what should be a 5 minute thing as you so well describe below.
Floris Bruynooghe writes:
[dist-packages] is a pretty neat solution to the problem.
To what problem? I admit I am no expert on Python packaging, but my experience with XEmacs suggests that this is the distro trying to help with a *set* of problems that the user/sysadmins really should be handling themselves. There are varied requirements that lead to a need for yet another place to install modules, but there's no guarantee that a given user/sysadmin has *just one*. With the Emacsen, often in trying to satisfy several requirements they end up with conflicts (different libraries in the site-lisp that are compiled for, or even require, different versions of (X)Emacs or even trying to deal with Emacs and XEmacs simultaneously). And it's a pain (as an XEmacs maintainer) to have to field questions about why things don't work in a set up I didn't design, and don't understand the rationale for.
Yeah, so do we all, but I suspect that for Python, like the Emacsen, there is *no* general guideline. You're just asking for various kinds of trouble, all of which require expert attention, by installing a command that is invoked as "python" (as opposed to "pythonX.Y") in /usr/local/bin unless you intend it to be a complete substitute for the system python. And that itself is a serious tour de force (as anybody who remembers Red Hat's love affair with Python 1.5.2 will agree, I suspect). As far as I can see, *in general* /usr/local/lib/python/site-packages is the right place to install local packages for use with the system python, and no other. That doesn't mean that there aren't *other* valid uses for that location, just that I see no general rule for supporting *all* of them at the same time.
On Wed, 22 Jul 2009 23:22:56 +0200, "M.-A. Lemburg" <mal@egenix.com> wrote:
The Debian/ubuntu distribution isn't non-standard. If anything I'd prefer to suggest that it is in many ways "a standard" Here's a sys.pth from a mac... ['', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload', '/Users/david/.local/lib/python2.6/site-packages', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages'] You can see that there are many choices along sys.path for installing packages.
People forget command lines - that's why.
It's good documentation of course. Cheers to Greg but the old method is so tedious. That really is the hard way. pythonpkgmgr offers a much easier solution by wrapping easy_install and/or pip. You just type in parts of the package name into a search box, click [search], a search of pypi is done, click [install] and your package is downloaded and installed. It's a much more modern way of doing package installation and requires absolutely no typing on a command line.
Perphaps. But it seems only for advanced users.. and I couldn't figure out on the face of it what advantage it would have. Take care David
David Lyon wrote:
It's not just Debian mangling their Python installation. Many other Linux distros do the same. Keeping up with all those changes has become so hard, that we have chosen to always build our projects with private (regular) Python installations on Linux and Macs. Disk space is cheap and so is bandwidth. Compared to the hours spent on resolving issues with system provided Python installations, it's a no-brainer.
No, not really. The above is perfectly standard. The directories on sys.path are setup by Python itself and make sure that it can find the stdlib modules. Those entries are not meant to be used as installation targets. Python 2.6 has two standard places for installing packages: 1. In the stdlib site-packages/ subdir 2. In the user home dir's .local/lib/python2.6/site-packages dir The second is a new feature in 2.6.
I think that's the main problem in all this... people just keep on complaining regardless of what you give them. If people don't read documentation, are not willing to at least spend a few minutes on considering what they really want to do and then shout out loud when Python doesn't do what they thought it should do (without telling Python, of course), there's little hope.
You define once and for all where you want your Python packages to install themselves when typing "python setup.py install". After that's done, you can forget about all this and move on to get some real work done ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 23 2009)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Jul 23, 2009 at 10:34:30AM +0200, M.-A. Lemburg wrote:
And is missing a 3rd one. The sysadmin who wants to install packages globally has to use the first option here, but that conflicts with the system package management on UNIXes. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
"M.-A. Lemburg" <mal@egenix.com> writes:
Before this discussion descends further into circular speculation on the Debian Python developers's intentions, perhaps someone should ask them directly: <URL:http://lists.debian.org/debian-python/>. -- \ “Let others praise ancient times; I am glad I was born in | `\ these.” —Ovid (43 BCE–18 CE) | _o__) | Ben Finney
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Lyon wrote:
On Tue, 21 Jul 2009 13:41:41 -0400, Tres Seaver <tseaver@palladion.com> wrote:
You missed my point: I know that sys.prefix etc. is set based on the location of the executable. You were complaining that setuptools / distutils wants to put files in an "OS-controlled directory": I was asking how Python was supposed to know the difference between /usr/lib/python2.x/site-packages (presumably "OS-controlled") and /opt/Python2.x/lib/python2.x/site-packages (definitely not "OS-controlled"). The solution is to avoid doing distutils / setuptools operations as root, *ever*, and instead use either an alternate install location (such as /opt/Python2.x) or use something like virtualenv, which provides a "safe" sandbox for installing stuff.
Users who install anything into /usr/lib/python2.x in "development" mode has already invited all the trouble they can handle into the house.
That is precisely how I think of virtualenv: it sits on top of distutils / setuptools, and makes it sane to do Python development without compromising the core system. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKZyGz+gerLs4ltQ4RAupxAJ95RZQJm6t1FtcFY+DMyHHmwRjmSACglo+W TehyHzxUwHQ5LfdibJsgUcA= =nf13 -----END PGP SIGNATURE-----
On Wed, 22 Jul 2009 10:26:59 -0400, Tres Seaver <tseaver@palladion.com> wrote:
You were complaining that setuptools / distutils wants to put files in an "OS-controlled directory":
I complain about many things - so what.
Well, at an interpreter level it doesn't, and I don't expect it to.
Have you tried this ? When I try to run it without su privelages setuptools crashes... (ubuntu 8.10) ------------------------------------------------------------------------------------------------ Running installer ... /usr/bin/easy_install "/media/Elements/Pythn Packages/html5lib-0.10.zip" ERRORS: Traceback (most recent call last): File "/usr/bin/easy_install", line 5, in <module> from pkg_resources import load_entry_point ImportError: No module named pkg_resources Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 78, in apport_excepthook report_file = open(pr_filename, 'wt') IOError: [Errno 13] Permission denied: '/var/crash/_usr_bin_easy_install.1000.crash' Original exception was: Traceback (most recent call last): File "/usr/bin/easy_install", line 5, in <module> from pkg_resources import load_entry_point ImportError: No module named pkg_resources ------------------------------------------------------------------------------------------------ I can only get it to work with sudo.
For me I prefer using a tool with a GUI interface. Take care David
On Wed, Jul 22, 2009 at 10:26 AM, Tres Seaver<tseaver@palladion.com> wrote:
autoconf--for all its warts--solved this ages ago. On installation, you specify what prefix you want to install stuff into; if you don't, you get a sensible default (/usr/local). If you're creating an OS package, or if installing directly into /usr is otherwise really what you really want to do, you specify --prefix=/usr. The Python core doesn't need to know anything about this; it just needs to have a sys.path that includes both /usr/lib/... and /usr/local/lib/.... This works for everything else--C headers, libraries, manpages, shared libraries. Why is Python special?
This is nasty. I should certainly be able to globally install packages on my own, without creating a whole new Python installation or other hacks, just like I can install anything else into /usr/local. -- Glenn Maynard
participants (15)
-
Ben Finney -
David Lyon -
david.lyon＠preisshare.net
-
Floris Bruynooghe -
Glenn Maynard -
James Y Knight -
Jim Jewett -
John Szakmeister -
M.-A. Lemburg -
Nick Coghlan -
P.J. Eby -
R. David Murray -
Stephen J. Turnbull -
Tarek Ziadé -
Tres Seaver