I've released version 0.1.1 of distil, downloadable from [1]. It's based on distlib 0.1.2. The other changes are as follows: * Added "distil init" to support creating an initial version of package.json metadata, which can then be added to during development. * Added "distil link" to support "editable" installations, similar to pip install -e local_dir. * Take into account pre-confirmation (-y) during uninstallation when dists that are no longer needed are found. These are now removed automatically when -y is specified. * Fixed error in setting up SSL certificate verification, and adjusted PyPI URLs to be https:// where specified as http:// in metadata. Successful SSL verification is now logged. * Added --local-dists option to allow local wheels and sdists to be used for installation. * Fixed a bug in the handling of local archives (e.g. those returned through a configured DirectoryLocator). Local archives shouldn’t be deleted after unpacking. * Added --python-tags argument to distil package when building wheels to configure the tags for the built wheel. * Added --no-unpack option to distil download. * Fixed problem with rollback on error caused by not recording SHARED and RECORD correctly. * Fixed bug in writing entry points (EXPORTS) file. * Use of 2to3 now defaults to True when run under 3.x. * Fixed bug when run in venvs which caused e.g. dist-packages to be used instead of site-packages. * Improved error message when run with Python 2.5 (not supported, but this is now clear from the error message). Your feedback will be gratefully received! Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py
On 30 April 2013 14:23, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
've released version 0.1.1 of distil, downloadable from [1]. It's based on distlib 0.1.2. The other changes are as follows:
[...]
Your feedback will be gratefully received!
One thought - it would be nice to have an equivalent of pip's --target flag (install packages into the specified directory) for distil install. I use this fairly frequently to install packages outside of sys.path, when creating standalone scripts bundled with their dependencies. Actually, in that case the metadata isn't much use either, so I often delete that. An option to not install the metadata directory would also be useful. I know it's an unusual use-case, but I thought I might as well ask :-) Paul
On 30 April 2013 14:23, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Your feedback will be gratefully received!
One thing that does slightly bother me. If distil is invoked without the -p or -e option, it defaults to installing into the system Python. (At least on Windows, on Unix I guess it gets the Python currently first on $PATH, due to the use of /usr/bin/env). Pretty much all of the time, that's *not* what I want (at least for install/uninstall type actions). Would it be worth requiring the use of -e or -p, when running distil to install/uninstall? (There could be a --system flag to say "yes, I really do want the system Python" if that's worth having as well as an explicit -p). Paul
From: Paul Moore <p.f.moore@gmail.com> One thing that does slightly bother me. If distil is invoked without the -p or -e option, it defaults to installing into the system Python.
That's not supposed to happen - it's supposed to install to the user site (PEP 370 Per user site-packages directory) unless you specify --system. It will use the system Python unless you specify -p. So: distil install xyz should install xyz to the per-user site-packages using the default Python, say 2.7. distil -p python3.2 install xyz should install xyz to the per-user site-packages, using Python 3.2. Specifying --system should install to the system-wide Python site-packages for the running Python. In a venv, installations should always be to the venv. I will see if the problem is Windows specific - the behaviour seems OK on POSIX. Regards, Vinay Sajip
Sorry, I knew that it installed to the per-user site packages directory, but I never use that so I don't fully understand the implications. But nevertheless, for my current usage, distil install foo is 99% of the time a user error, as I maintain a completely clean system Python, and do all my package installs in virtualenvs. Maybe the per-user site packages is something I should be using, and if I do, then I can afford to be less obsessive about keeping my system Python clean. But until distil became available, I haven't had the tools to work with the user site packages directory, so it's new ground for me. The one thing that *is* different between Unix and Windows is that on Unix, the hashbang line of #!/usr/bin/env python will use the currently active Python (system or virtualenv as appropriate) but under Windows it will always use the system Python (2 or 3 depending on py.ini). Maybe the launcher should treat /sur/bin/env python differently, and try looking on PATH before using the default in that case, but that's not how it works, sadly (indeed, there's no way of getting that behaviour with the current launcher). I can imagine having a virtualenv active and running distil install foo and being surprised that the install doesn't go to the active virtualenv. Paul On 2 May 2013 17:11, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
If you do all your package installs into venvs, then isn't "distil -e venvdir install foo" enough? You don't need to activate the venv, just tell distil which venv to install into via the -e.
I have implemented path searching in the launcher, but it's currently disabled. I can re-enable it (such that the path is only searched if the PYLAUNCH_SEARCHPATH environment variable is set) and create a new release of the standalone launcher - are you willing to be a guinea pig for this functionality? Regards, Vinay Sajip
On 2 May 2013 18:29, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
It's not that -e is insufficient, rather that *forgetting* the -e doesn't fail, but rather does the wrong thing silently (and has an unintended "global" impact). If distil used the "active" Python by default (which it does on Unix) then this wouldn't concern me so much. But it is a small point, I'd admit.
What I would like is for #!/usr/bin/env python to be treated specially as a path search (it's the only case on Unix that does a path search after all), with a fallback of using the default python (like #!python) if there is no python on $PATH. That completely replicates the Unix behaviour (i.e., the author's intention, I would assume). I would argue that this does not need to be conditionally enabled, but given that it deviates from the PEP I can live with the requirement. Although I'd prefer it if the override was via the ini file rather than being via an environment variable. Oh, and can I have a pony? :-) Seriously, I'd be happy to test this. Paul
On 3 May 2013 03:09, "Paul Moore" <p.f.moore@gmail.com> wrote:
Sorry, I knew that it installed to the per-user site packages directory,
but I never use that so I don't fully understand the implications.
But nevertheless, for my current usage, distil install foo is 99% of the
time a user error, as I maintain a completely clean system Python, and do all my package installs in virtualenvs. Maybe the per-user site packages is something I should be using, and if I do, then I can afford to be less obsessive about keeping my system Python clean. But until distil became available, I haven't had the tools to work with the user site packages directory, so it's new ground for me. "pip --user" manipulates the per-user directories (and I believe I filed a ticket a while back suggesting it should migrate to the model distil now uses). Cheers, Nick.
The one thing that *is* different between Unix and Windows is that on
Unix, the hashbang line of #!/usr/bin/env python will use the currently active Python (system or virtualenv as appropriate) but under Windows it will always use the system Python (2 or 3 depending on py.ini). Maybe the launcher should treat /sur/bin/env python differently, and try looking on PATH before using the default in that case, but that's not how it works, sadly (indeed, there's no way of getting that behaviour with the current launcher). I can imagine having a virtualenv active and running distil install foo and being surprised that the install doesn't go to the active virtualenv.
the -p or -e option, it defaults to installing into the system Python.
On 3 May 2013 09:08, "Donald Stufft" <donald@stufft.io> wrote:
On May 2, 2013, at 6:48 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
"pip --user" manipulates the per-user directories (and I believe I filed
a ticket a while back suggesting it should migrate to the model distil now uses).
Language level installers should leave "/usr" alone by default, as that part of the filesystem is the domain of system packages. Cheers, Nick.
On May 2, 2013, at 7:17 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Out of the three primary OSs only one of them has a concept of system packages. Regardless it's a major change in behavior and isn't a clear cut right or wrong choice. It's going to need a stronger justification than that to break backwards compatibility in such a large way.
On Fri, May 3, 2013 at 9:30 AM, Donald Stufft <donald@stufft.io> wrote:
How about: I won't approve PEP 439 as long as pip still writes to the system site-packages by default when running as an ordinary user on Linux systems? "sudo pip" needs to be banished from the command line lexicon of Python developers. The transition isn't that hard: add --system now, switch the default from --system to --user for anyone *other than root* some time before Python 3.4. Anyone using virtual environments or an explicit --user or --system will be utterly unaffected when the default changes, as will anyone running an explicit "sudo pip" (in order to get write access to the system Python site-packages). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On May 2, 2013, at 9:02 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
So your answer to my wanting stronger justification is "because I said so"? *Why* does ``sudo pip`` need to be banished? ``pip install --user`` is not any safer, it's not any more or less wrong. It's an option. If pip was brand new I'd probably be of the opinion that I prefer the user scheme by default. However pip is not new, and it has a long history of working in this way. Changes can be made but I'm loath to support a large change in behavior like this without strong justification as to *why*. I also think your proposed solution of --user being the default unless we are running as root is suboptimal. Conditional defaults are confusing to end users, even more so when a common "error" condition (e.g. I forgot to type sudo before I pip installed, or I forgot to activate my virtualenv) is going to move from being an error to installing to the user scheme and *working*… right up until the time you actually try and use it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 3 May 2013 11:39, "Donald Stufft" <donald@stufft.io> wrote:
filed that part than that
Because sysadmins don't participate in upstream development and non-platform provided tools messing with /usr is wrong, wrong, wrong. The system Python is the domain of distro tools, we should be leaving it the hell alone. Cheers, Nick.
*Why* does ``sudo pip`` need to be banished? ``pip install --user`` is
not any safer, it's not
On May 3, 2013, at 6:29 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
To be honest your proposal doesn't really do that since you want it to install into the site-packages if they are running it as root. If that's you're argument you should at least be consistent within your own proposal. One of my main problems with your proposal is that we suddenly have surprising behavior about where packages are being installed to: Linux + root user = site-packages Linux + regular user = user-packages (<-- Going to confuse people because it's different) Linux + regular user + virtualenv = site-packages I don't know much about RHEL, but Debian "installing globally" means installing into /usr/local/lib/pythonX.Y/site-packages/ which is exactly where Debian wants you to put python libraries not installed via apt. Apt drops things into /usr/local/lib/pythonX.Y/dist-packages. Even if installing into the "system" python globally is wrong is there even a way too determine if we are attempting to install into a *system* python or not? I could be wrong but afaik there's not a good way to determine if this is a python that is managed by the system or one that was installed otherwise. If that's the case then we break by default one of the viable methods of isolation *from* system. This is very popular on OSX where people tend to install a non system python via homebrew to isolate themselves. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I would also be relatively happy for pip to refuse the temptation to guess if run globally and require an explicit --user or --system whenever it is run outside a virtual environment. However, I think it's better to make the typical "pip install whatever" work for most unprivileged users without requiring elevated privileges. I agree the proposed exception for root doesn't make sense so I withdraw that idea, even though installing things into root's home directory is a little strange. As far as Debian's dist-packages setup goes, that's their workaround for this misfeature of the current Python packaging ecosystem. Cheers, Nick.
On May 3, 2013, at 11:14 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
As someone responsible for working with Python app deployment tools, this _will_ break the universe. Yes people should be using virtualenv, however many don't, deal with it. People expect "package install" as root to work like every other package system (yum, apt, take your pick) where they run "sudo pip install a b c d" and then run their app as a service-specific user. Magically writing to ~root will clearly not work in this case unless you also run your app as root (though I know some people do that too, but not behavior that should be encouraged). This proposal is entirely non-viable for anything but 100% best-practices users, full stop. --Noah
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/03/2013 02:24 PM, Noah Kantrowitz wrote:
I don't know any of those people: either folks have learned *not* to install as root, or else they run as root too. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGEAP4ACgkQ+gerLs4ltQ6odACfXcO7KHBYvPJhWOZCXMtMTZsn bxIAn3j9jwRe70de+O0mIY4K2kfDrfzs =WgtN -----END PGP SIGNATURE-----
On May 3, 2013, at 2:25 PM, Tres Seaver <tseaver@palladion.com> wrote:
I know a fair number of people like that. The common wisdom is *not* to install python packages from the system package mangers. because they generally either don't have everything so you'll need to install from PyPI anyways, or they'll have really old versions. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Hi Nick On 05/03/2013 12:14 PM, Nick Coghlan wrote:
IMO requiring either --user or --system is bad UI (we should have a sane default). Defaulting to --user is tempting, but also a bad idea, both because of the massive adjustment that it would require from existing users, and because it does the wrong thing in many cases where there are multiple Pythons compiled on the system. I think the best solution to this problem is quite easily implemented and non-intrusive: an improved error message when pip finds that it lacks permission to install to site-packages. This error message could recommend --user and/or virtualenv, could warn quite strongly against installing to this location unless you really know what you are doing, etc. Also, I think that for the specific situation of a Linux distribution's "system Python" (which, as Donald has pointed out, is only one mode of Python installation out of many), Debian's solution is better than just a workaround, it is conceptually more or less the Right Thing: separate distro-managed Python code from user-managed Python code, without preventing independent installation of Python-installation-wide code by the user. (Though some aspects of the implementation of Debian's solution are sub-optimal and have required ugly workarounds in pip. It would be better if multiple site-packages were a first-class concept in Python's own site.py so Debian's solution could be implemented via configuration rather than patches.) Carl
What is the proposal for Windows here? Would the default behaviour change? There's no concept of "sudo" on Windows. Personally, I don't care much as I only use pip in virtualenvs, but backward compatibility *is* important, even if the current behaviour is considered wrong. Also, don't forget that the current release of pip supports Python 2.5, which doesn't have per-user site-packages. The next release will drop support for 2.5, so we *could* start the switch to the user directory at the same time, but up until now that wasn't an option. PEP 439 may be about Python 3.4, but Paul.
participants (7)
-
Carl Meyer -
Donald Stufft -
Nick Coghlan -
Noah Kantrowitz -
Paul Moore -
Tres Seaver -
Vinay Sajip