problems with import

One problem I often encounter is with import search. An example of the problem is with the package mercurial. It has extensions in a subdirectory called 'hgext'. On fedora, I install mercurial using the vendor package. This creates /usr/lib64/python2.7/site-packages/hgext/ Later, I want to try out an extension as a non-privileged user. cd <extension> python setup.py install --user Now I also have ~/.local/lib/python2.7/site-packages/hgext but python won't search there for extensions. Once if finds the system hgext directory, it won't look also in the local one. Any thoughts?

Neal Becker wrote:
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages? http://legacy.python.org/dev/peps/pep-0420/

On Fri, Jun 27, 2014 at 11:34 PM, Peter Otten <__peter__@web.de> wrote:
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages?
Unfortunately for the OP, that doesn't seem to be applicable to Python 2.x; also, changes made to Python in response to this list don't usually apply to 2.x either. Mercurial doesn't work with Python 3, currently, although some of the improvements to the latest Pythons have been in response to reported issues with hg, so it's possible that hg might support 3.5 or 3.6 at some point. I don't know if there's a 2.x-compatible solution to this problem. ChrisA

Chris Angelico wrote:
http://legacy.python.org/dev/peps/pep-0420/#namespace-packages-today But this is likely off-topic for python-ideas.

Le 27/06/2014 13:12, Steven D'Aprano a écrit :
Then he would have the reverse problem: once he installs a user-local hg extension, the bundled (official) hg extensions wouldn't be reachable anymore. The answer here comes into two possibilities, both of which have to do with Mercurial and none with Python itself: 1) Mercurial could make hgext a namespace package (see Peter Otten's answer above) 2) third-party extensions for Mercurial should never install into the "hgext" package but rather in a separate top-level package or module (presumable called "hgsomething") Regards Antoine.

On Fri, Jun 27, 2014 at 02:33:07PM -0400, Antoine Pitrou wrote:
Naturally, but I assumed that the only reason you would install something locally was if you intended it to over-ride the global version. If that's not the case, then you're right, it's an issue for Mercurial to solve. -- Steven

Steven D'Aprano wrote:
I don't think this is unique to mercurial. I'd like to have 2 areas for installing extensions to a package: a system wide and a local. I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict

Yeah, so in Python 3.3 this is possible through namespace packages (see PEP 420 -- tldr: remove the empty __init__.py). It has been supported for a long time in Python 2 by setuptools, and you can even do it yourself by setting the package's __path__ attribute. See also pkgutil.py in the Python 2 stdlib. On Fri, Jun 27, 2014 at 2:02 PM, Neal Becker <ndbecker2@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

Neal Becker writes:
I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict
Maybe. XEmacs does such overriding, and my experience is that users expect DWIM behavior (the "best" version gets used). Typically local trees get out of date and may not be compatible with newer versions of the main tree updated by the OS's PMS, etc. It makes things hard to diagnose.

On Saturday, June 28, 2014 8:48 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Isn't Python already flexible enough here? Whichever one comes first in sys.path wins. If you (as a distro packager, a sysadmin, a user, or even a program at runtime) want to customize that order, it's trivial to do so. If even that isn't good enough, you can replace almost any piece of the import machinery pretty easily—you could write a custom finder that finds all versions and picks the one with the newest timestamp, or whatever you think is better. So, what do people want here that Python doesn't do? Meanwhile, what I personally prefer is to let local beat global, as long as I have an easy way to check when that may not be a good idea anymore. For example, I had PyObjC 2.5.0 installed for my system Python 2.7, back when Apple was using 2.4.something. At some point, Apple switched to 2.5.1, so after installing that system upgrade, my local copy was no longer needed, or wanted. Fortunately, I have a script that I wrote for just that occasion that told me about it so I could uninstall it. I definitely wouldn't want Python to automatically start ignoring my 2.5.0 because there's a 2.5.1. I'd love it if something (Python, Apple's installer, a script that came with either Python or OS X, whatever) would alert me to the problem so I didn't need my own script, but I'm not expecting that. Also, what's right for my primary dev machine is not necessarily what's right for my company's testing systems or live deployed servers, or our customers' disparate systems; all I can really do there is require PyObjC 2.5.0 and let whoever's in charge of the machine figure out that they may be able to get that by uninstalling instead of upgrading. Anything that tries to DWIM its way out of that problem is going to get things mysteriously wrong as often as it helps.

On 28 Jun 2014 05:20, "Steven D'Aprano" <steve@pearwood.info> wrote:
Local installs are supported to *add* non-conflicting user specific packages to the system Python installation, not to override them. The fact it's tricky to override the standard library and system provided libraries helps reduce the attack surface when running software with elevated access, but still as a specific user. Packages *can* opt-in to allowing contributions of submodules from later sys.path entries by declaring that package as a namespace package. In Python 3.3+ that's as simple as leaving __init__.py out entirely. In any version, it can be done explicitly using pkgutil.extend_path() (standard library) or pkg_resources.declare_namespace() (published as part of setuptools) Even with namespace packages, though, *modules* earlier in sys.path still take precedence over later entries. Cheers, Nick.

On Fri, Jun 27, 2014 at 7:05 AM, Neal Becker <ndbecker2@gmail.com> wrote:
Use ~/.hgrc to enable extensions: http://www.selenic.com/mercurial/hgrc.5.html#extensions In your case give the explicit path. I've been doing this for years and it works great. -eric

Neal Becker wrote:
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages? http://legacy.python.org/dev/peps/pep-0420/

On Fri, Jun 27, 2014 at 11:34 PM, Peter Otten <__peter__@web.de> wrote:
Isn't that addressed with "PEP 420 -- Implicit Namespace Packages?
Unfortunately for the OP, that doesn't seem to be applicable to Python 2.x; also, changes made to Python in response to this list don't usually apply to 2.x either. Mercurial doesn't work with Python 3, currently, although some of the improvements to the latest Pythons have been in response to reported issues with hg, so it's possible that hg might support 3.5 or 3.6 at some point. I don't know if there's a 2.x-compatible solution to this problem. ChrisA

Chris Angelico wrote:
http://legacy.python.org/dev/peps/pep-0420/#namespace-packages-today But this is likely off-topic for python-ideas.

Le 27/06/2014 13:12, Steven D'Aprano a écrit :
Then he would have the reverse problem: once he installs a user-local hg extension, the bundled (official) hg extensions wouldn't be reachable anymore. The answer here comes into two possibilities, both of which have to do with Mercurial and none with Python itself: 1) Mercurial could make hgext a namespace package (see Peter Otten's answer above) 2) third-party extensions for Mercurial should never install into the "hgext" package but rather in a separate top-level package or module (presumable called "hgsomething") Regards Antoine.

On Fri, Jun 27, 2014 at 02:33:07PM -0400, Antoine Pitrou wrote:
Naturally, but I assumed that the only reason you would install something locally was if you intended it to over-ride the global version. If that's not the case, then you're right, it's an issue for Mercurial to solve. -- Steven

Steven D'Aprano wrote:
I don't think this is unique to mercurial. I'd like to have 2 areas for installing extensions to a package: a system wide and a local. I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict

Yeah, so in Python 3.3 this is possible through namespace packages (see PEP 420 -- tldr: remove the empty __init__.py). It has been supported for a long time in Python 2 by setuptools, and you can even do it yourself by setting the package's __path__ attribute. See also pkgutil.py in the Python 2 stdlib. On Fri, Jun 27, 2014 at 2:02 PM, Neal Becker <ndbecker2@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)

Neal Becker writes:
I think the semantics we'd want is that the 2 trees are effectively merged, with the local overriding in the event of a conflict
Maybe. XEmacs does such overriding, and my experience is that users expect DWIM behavior (the "best" version gets used). Typically local trees get out of date and may not be compatible with newer versions of the main tree updated by the OS's PMS, etc. It makes things hard to diagnose.

On Saturday, June 28, 2014 8:48 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Isn't Python already flexible enough here? Whichever one comes first in sys.path wins. If you (as a distro packager, a sysadmin, a user, or even a program at runtime) want to customize that order, it's trivial to do so. If even that isn't good enough, you can replace almost any piece of the import machinery pretty easily—you could write a custom finder that finds all versions and picks the one with the newest timestamp, or whatever you think is better. So, what do people want here that Python doesn't do? Meanwhile, what I personally prefer is to let local beat global, as long as I have an easy way to check when that may not be a good idea anymore. For example, I had PyObjC 2.5.0 installed for my system Python 2.7, back when Apple was using 2.4.something. At some point, Apple switched to 2.5.1, so after installing that system upgrade, my local copy was no longer needed, or wanted. Fortunately, I have a script that I wrote for just that occasion that told me about it so I could uninstall it. I definitely wouldn't want Python to automatically start ignoring my 2.5.0 because there's a 2.5.1. I'd love it if something (Python, Apple's installer, a script that came with either Python or OS X, whatever) would alert me to the problem so I didn't need my own script, but I'm not expecting that. Also, what's right for my primary dev machine is not necessarily what's right for my company's testing systems or live deployed servers, or our customers' disparate systems; all I can really do there is require PyObjC 2.5.0 and let whoever's in charge of the machine figure out that they may be able to get that by uninstalling instead of upgrading. Anything that tries to DWIM its way out of that problem is going to get things mysteriously wrong as often as it helps.

On 28 Jun 2014 05:20, "Steven D'Aprano" <steve@pearwood.info> wrote:
Local installs are supported to *add* non-conflicting user specific packages to the system Python installation, not to override them. The fact it's tricky to override the standard library and system provided libraries helps reduce the attack surface when running software with elevated access, but still as a specific user. Packages *can* opt-in to allowing contributions of submodules from later sys.path entries by declaring that package as a namespace package. In Python 3.3+ that's as simple as leaving __init__.py out entirely. In any version, it can be done explicitly using pkgutil.extend_path() (standard library) or pkg_resources.declare_namespace() (published as part of setuptools) Even with namespace packages, though, *modules* earlier in sys.path still take precedence over later entries. Cheers, Nick.

On Fri, Jun 27, 2014 at 7:05 AM, Neal Becker <ndbecker2@gmail.com> wrote:
Use ~/.hgrc to enable extensions: http://www.selenic.com/mercurial/hgrc.5.html#extensions In your case give the explicit path. I've been doing this for years and it works great. -eric
participants (10)
-
Andrew Barnert
-
Antoine Pitrou
-
Chris Angelico
-
Eric Snow
-
Guido van Rossum
-
Neal Becker
-
Nick Coghlan
-
Peter Otten
-
Stephen J. Turnbull
-
Steven D'Aprano