problems with import
One problem I often encounter is with import search. An example of the problem is with the package mercurial. It has extensions in a subdirectory called 'hgext'. On fedora, I install mercurial using the vendor package. This creates /usr/lib64/python2.7/site-packages/hgext/ Later, I want to try out an extension as a non-privileged user. cd <extension> python setup.py install --user Now I also have ~/.local/lib/python2.7/site-packages/hgext but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one. Any thoughts?
Neal Becker wrote:
One problem I often encounter is with import search.
An example of the problem is with the package mercurial. It has extensions in a subdirectory called 'hgext'.
On fedora, I install mercurial using the vendor package. This creates
/usr/lib64/python2.7/site-packages/hgext/
Later, I want to try out an extension as a non-privileged user.
cd <extension> python setup.py install --user
Now I also have ~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one.
Any thoughts?
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages? http://legacy.python.org/dev/peps/pep-0420/
On Fri, Jun 27, 2014 at 11:34 PM, Peter Otten <__peter__@web.de> wrote:
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages?
Unfortunately for the OP, that doesn't seem to be applicable to Python 2.x; also, changes made to Python in response to this list don't usually apply to 2.x either. Mercurial doesn't work with Python 3, currently, although some of the improvements to the latest Pythons have been in response to reported issues with hg, so it's possible that hg might support 3.5 or 3.6 at some point. I don't know if there's a 2.x-compatible solution to this problem. ChrisA
Chris Angelico wrote:
On Fri, Jun 27, 2014 at 11:34 PM, Peter Otten <__peter__@web.de> wrote:
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages?
Unfortunately for the OP, that doesn't seem to be applicable to Python 2.x; also, changes made to Python in response to this list don't usually apply to 2.x either. Mercurial doesn't work with Python 3, currently, although some of the improvements to the latest Pythons have been in response to reported issues with hg, so it's possible that hg might support 3.5 or 3.6 at some point.
I don't know if there's a 2.x-compatible solution to this problem.
http://legacy.python.org/dev/peps/pep-0420/#namespace-packages-today But this is likely off-topic for python-ideas.
On Fri, Jun 27, 2014 at 09:05:48AM -0400, Neal Becker wrote: [...]
Now I also have ~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one.
Re-arrange sys.path so that the local site-packages comes first, before the global site-packages. (I'm surprised Python doesn't already do this.) -- Steven
Le 27/06/2014 13:12, Steven D'Aprano a écrit :
On Fri, Jun 27, 2014 at 09:05:48AM -0400, Neal Becker wrote: [...]
Now I also have ~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one.
Re-arrange sys.path so that the local site-packages comes first, before the global site-packages. (I'm surprised Python doesn't already do this.)
Then he would have the reverse problem: once he installs a user-local hg extension, the bundled (official) hg extensions wouldn't be reachable anymore. The answer here comes into two possibilities, both of which have to do with Mercurial and none with Python itself: 1) Mercurial could make hgext a namespace package (see Peter Otten's answer above) 2) third-party extensions for Mercurial should never install into the "hgext" package but rather in a separate top-level package or module (presumable called "hgsomething") Regards Antoine.
On Fri, Jun 27, 2014 at 02:33:07PM -0400, Antoine Pitrou wrote:
Le 27/06/2014 13:12, Steven D'Aprano a écrit :
On Fri, Jun 27, 2014 at 09:05:48AM -0400, Neal Becker wrote: [...]
Now I also have ~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one.
Re-arrange sys.path so that the local site-packages comes first, before the global site-packages. (I'm surprised Python doesn't already do this.)
Then he would have the reverse problem: once he installs a user-local hg extension, the bundled (official) hg extensions wouldn't be reachable anymore.
Naturally, but I assumed that the only reason you would install something locally was if you intended it to over-ride the global version. If that's not the case, then you're right, it's an issue for Mercurial to solve. -- Steven
Steven D'Aprano wrote:
On Fri, Jun 27, 2014 at 02:33:07PM -0400, Antoine Pitrou wrote:
Le 27/06/2014 13:12, Steven D'Aprano a écrit :
On Fri, Jun 27, 2014 at 09:05:48AM -0400, Neal Becker wrote: [...]
Now I also have ~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one.
Re-arrange sys.path so that the local site-packages comes first, before the global site-packages. (I'm surprised Python doesn't already do this.)
Then he would have the reverse problem: once he installs a user-local hg extension, the bundled (official) hg extensions wouldn't be reachable anymore.
Naturally, but I assumed that the only reason you would install something locally was if you intended it to over-ride the global version. If that's not the case, then you're right, it's an issue for Mercurial to solve.
I don't think this is unique to mercurial. I'd like to have 2 areas for installing extensions to a package: a system wide and a local. I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict
Yeah, so in Python 3.3 this is possible through namespace packages (see PEP 420 -- tldr: remove the empty __init__.py). It has been supported for a long time in Python 2 by setuptools, and you can even do it yourself by setting the package's __path__ attribute. See also pkgutil.py in the Python 2 stdlib. On Fri, Jun 27, 2014 at 2:02 PM, Neal Becker <ndbecker2@gmail.com> wrote:
Steven D'Aprano wrote:
On Fri, Jun 27, 2014 at 02:33:07PM -0400, Antoine Pitrou wrote:
Le 27/06/2014 13:12, Steven D'Aprano a écrit :
On Fri, Jun 27, 2014 at 09:05:48AM -0400, Neal Becker wrote: [...]
Now I also have ~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one.
Re-arrange sys.path so that the local site-packages comes first, before the global site-packages. (I'm surprised Python doesn't already do this.)
Then he would have the reverse problem: once he installs a user-local hg extension, the bundled (official) hg extensions wouldn't be reachable anymore.
Naturally, but I assumed that the only reason you would install something locally was if you intended it to over-ride the global version. If that's not the case, then you're right, it's an issue for Mercurial to solve.
I don't think this is unique to mercurial.
I'd like to have 2 areas for installing extensions to a package: a system wide and a local.
I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
Le 27/06/2014 17:02, Neal Becker a écrit :
I don't think this is unique to mercurial.
I'd like to have 2 areas for installing extensions to a package: a system wide and a local.
I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict
Who is "we"? Mercurial extensions have to be enabled manually in your .hgrc, so whether they live in the hgext namespace or anywhere else is quite irrelevant. Regards Antoine.
Neal Becker writes:
I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict
Maybe. XEmacs does such overriding, and my experience is that users expect DWIM behavior (the "best" version gets used). Typically local trees get out of date and may not be compatible with newer versions of the main tree updated by the OS's PMS, etc. It makes things hard to diagnose.
On Saturday, June 28, 2014 8:48 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Neal Becker writes:
I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict
Maybe. XEmacs does such overriding, and my experience is that users expect DWIM behavior (the "best" version gets used). Typically local trees get out of date and may not be compatible with newer versions of the main tree updated by the OS's PMS, etc. It makes things hard to diagnose.
Isn't Python already flexible enough here? Whichever one comes first in sys.path wins. If you (as a distro packager, a sysadmin, a user, or even a program at runtime) want to customize that order, it's trivial to do so. If even that isn't good enough, you can replace almost any piece of the import machinery pretty easily—you could write a custom finder that finds all versions and picks the one with the newest timestamp, or whatever you think is better. So, what do people want here that Python doesn't do? Meanwhile, what I personally prefer is to let local beat global, as long as I have an easy way to check when that may not be a good idea anymore. For example, I had PyObjC 2.5.0 installed for my system Python 2.7, back when Apple was using 2.4.something. At some point, Apple switched to 2.5.1, so after installing that system upgrade, my local copy was no longer needed, or wanted. Fortunately, I have a script that I wrote for just that occasion that told me about it so I could uninstall it. I definitely wouldn't want Python to automatically start ignoring my 2.5.0 because there's a 2.5.1. I'd love it if something (Python, Apple's installer, a script that came with either Python or OS X, whatever) would alert me to the problem so I didn't need my own script, but I'm not expecting that. Also, what's right for my primary dev machine is not necessarily what's right for my company's testing systems or live deployed servers, or our customers' disparate systems; all I can really do there is require PyObjC 2.5.0 and let whoever's in charge of the machine figure out that they may be able to get that by uninstalling instead of upgrading. Anything that tries to DWIM its way out of that problem is going to get things mysteriously wrong as often as it helps.
On 28 Jun 2014 05:20, "Steven D'Aprano" <steve@pearwood.info> wrote:
Naturally, but I assumed that the only reason you would install something locally was if you intended it to over-ride the global version. If that's not the case, then you're right, it's an issue for Mercurial to solve.
Local installs are supported to *add* non-conflicting user specific packages to the system Python installation, not to override them. The fact it's tricky to override the standard library and system provided libraries helps reduce the attack surface when running software with elevated access, but still as a specific user. Packages *can* opt-in to allowing contributions of submodules from later sys.path entries by declaring that package as a namespace package. In Python 3.3+ that's as simple as leaving __init__.py out entirely. In any version, it can be done explicitly using pkgutil.extend_path() (standard library) or pkg_resources.declare_namespace() (published as part of setuptools) Even with namespace packages, though, *modules* earlier in sys.path still take precedence over later entries. Cheers, Nick.
-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Jun 27, 2014 at 7:05 AM, Neal Becker <ndbecker2@gmail.com> wrote:
One problem I often encounter is with import search.
An example of the problem is with the package mercurial. It has extensions in a subdirectory called 'hgext'.
On fedora, I install mercurial using the vendor package. This creates
/usr/lib64/python2.7/site-packages/hgext/
Later, I want to try out an extension as a non-privileged user.
cd <extension> python setup.py install --user
Now I also have ~/.local/lib/python2.7/site-packages/hgext
but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one.
Any thoughts?
Use ~/.hgrc to enable extensions: http://www.selenic.com/mercurial/hgrc.5.html#extensions In your case give the explicit path. I've been doing this for years and it works great. -eric
participants (10)
-
Andrew Barnert -
Antoine Pitrou -
Chris Angelico -
Eric Snow -
Guido van Rossum -
Neal Becker -
Nick Coghlan -
Peter Otten -
Stephen J. Turnbull -
Steven D'Aprano