Brett Cannon wrote:

On 1/30/07, Ron Adam <rrr@ronadam.com> wrote:

...

In order to resolve a path conflict where I'm working on several copies of the same package. I found it useful to add the following near the top of modules in a package or sub package.

Module in package:

import sys sys.path = ['..'] + sys.path

import package.module # Imports module in "this!" package.

Note: There could still be conflicts if a module with the same name is in the same directory as the package. But that's much less likely than one in the rest of the path.

Module in sub-package:

import sys sys.path = ['../..'] + sys.path

import package.subpackage.module # finds "self" (subpackage) reliably.

By explicitly adding the packages parent directory to the *front* of sys.path it resolves cases where imports using absolute imports, import modules from another package because they are found first in the search path.

Why aren't you using relative imports (e.g., ``from . import module``)? That should be doing exactly what you want. That uses __path__ which is set to the path of the package.

-Brett

But if you try to run a module in a package, vs from a package, with relative from . imports, you will get: ValueError: Attempted relative import in non-package And there is no __path__ attribute to look at since no package has been imported yet. So you need to import the package first, and then you may not import the correct package even then if there is another package or module with the same name. For example if you are running a test module from test sub-package that has a "if __name__=='__main__': _test()" at the bottom. Something that is not uncommon. It may attempt to import modules from a different package. In my case it was a package with the same name in a different SVN branch. The tests actually ran, but with errors because it wasn't exactly the same. It wasn't obvious what was going on either. Cheers, Ron

Brett Cannon wrote:

On 1/30/07, Ron Adam <rrr@ronadam.com> wrote:

...

...

On 1/30/07, Ron Adam <rrr@ronadam.com> wrote:

...

In order to resolve a path conflict where I'm working on several copies of the same package. I found it useful to add the following near the top of modules in a package or sub package.

Module in package:

import sys sys.path = ['..'] + sys.path

import package.module # Imports module in "this!"

Brett Cannon wrote: package.

...

...

Note: There could still be conflicts if a module with the same name is in the same directory as the package. But that's much less likely than one in the rest of the path.

Module in sub-package:

import sys sys.path = ['../..'] + sys.path

import package.subpackage.module # finds "self" (subpackage) reliably.

By explicitly adding the packages parent directory to the *front* of sys.path it resolves cases where imports using absolute imports, import modules from another package because they are found first in the search path.

Why aren't you using relative imports (e.g., ``from . import module``)? That should be doing exactly what you want. That uses __path__ which is set to the path of the package.

-Brett

But if you try to run a module in a package, vs from a package, with relative from . imports, you will get:

ValueError: Attempted relative import in non-package

And there is no __path__ attribute to look at since no package has been imported yet.

Yeah, I have been bitten by that. You need to have that initial package import. I got nailed by this when using '-m' and a module contained within a package for executing tests.

And you can't set __path__ manually, that only changes the Exception to an Import Error.

...

So you need to import the package first, and then you may not import the correct package even then if there is another package or module with the same name.

Well, just don't do that. =) You really shouldn't have multiple things with the same name in sys.path. But as you noticed, you can control it with sys.path if you really need to.

I just don't see this as a common enough problem to warrant documenting it somewhere (and I have no clue where that "somewhere" would be).

I expected that someone would say that. How about adding a note at the bottom of PEP 328. That was one of the places I looked for a way to resolve this. Or possibly posting a message on python-ideas. Consider it done ;-) A note on PEP 328 might be a good idea still. I have no idea what the best wording would be. The import tutorial is another place where a note clarifying relative imports can't be used for stand alone scripts is needed. The nice thing about adjusting sys.path directly is you can then move back and forth from different same named versions in development without doing renames. If you do a rename, you also have to remember to rename all the submodule absolute references in the package as well. Another alternative is to hide the ones you don't want by renaming them. Then you have to do renames to switch back to them. That makes it more difficult to do side by side comparisons as well. Altering PYTHONPATH is another way to do it, but you also need to switch it back if you switch versions. But if you forget to edit it back, and use a version not on the path, it will import modules from the version on the path. In this case the package I'm working on may replace pydoc.py module in pythons library. (no promises though) The name matters so that site.py can locate the help function. It's convenient for me to be able to easily and dependably compare different working versions. A few things to consider... ----- According to PEP 328: You may use relative imports freely. In Python 2.6, any import statement that results in an intra-package import will raise DeprecationWarning (this also applies to from <> import that fails to use the relative import syntax). In Python 2.7, import will always be an absolute import (and the __future__ directive will no longer be needed). ----- So It will probably come up more often as more people start using absolute imports and the '.' style relative imports. Or as you put it, more people get "bitten". Take a look at how much __name__ == '__main__' is used in the standard library. Those modules won't be able to use relative imports. They will need to be updated and can use only absolute imports. Some people will probably want a way to have relative imports work with modules meant to be run as scripts (in packages). An optional function (or property) may be a good way to resolve both of these package self reference situations without any changes to the default behavior. Cheers, Ron

Brett Cannon wrote:

On 1/30/07, Ron Adam <rrr@ronadam.com> wrote:

...

Brett Cannon wrote: [SNIP]

...

I just don't see this as a common enough problem to warrant documenting it somewhere (and I have no clue where that "somewhere" would be).

I expected that someone would say that.

How about adding a note at the bottom of PEP 328. That was one of the places I looked for a way to resolve this. Or possibly posting a message on python-ideas. Consider it done ;-)

A note on PEP 328 might be a good idea still. I have no idea what the best wording would be. The import tutorial is another place where a note clarifying relative imports can't be used for stand alone scripts is needed.

Sure, but that is not my PEP. That is something to bug Thomas Wouters about.

Ok, Maybe later. I looked into it further and I think it needs more thought in any case.

...

The nice thing about adjusting sys.path directly is you can then move back and forth from different same named versions in development without doing renames. If you do a rename, you also have to remember to rename all the submodule absolute references in the package as well. Another alternative is to hide the ones you don't want by renaming them. Then you have to do renames to switch back to them. That makes it more difficult to do side by side comparisons as well.

Right but the point of relative imports is to help prevent renaming. It's a catch-22 situation of what matters more to you.

Right, thats what I tried first. It would have been nice if that worked.

...

Take a look at how much __name__ == '__main__' is used in the standard library. Those modules won't be able to use relative imports. They will need to be updated and can use only absolute imports.

It's possible. Maybe this will finally push forward the idea of having a proper 'main' function or something.

<clip>

Yeah, some built-in or something might do it that lets you decorate what function is to be the 'main' function and maybe do some magic to make it resolve properly.

It's not the main function that is the problem, it's the imports at the top of the file before the main function. If they import ok, then any main function will work also. I did some experimenting and just adding '../..' in front of the path has some problems of its own. It's relative to the directory you run it from instead of the directory the script is in. So it may or may not allow the script to run. What is needed is to append the actual root package location to the front os sys.path. If absolute imports are used for intra-package imports, then there is not chance of shadowing the library by doing this. This is what I came up with, but there may be an easier way. (and shorter) def pkg_dir(file): import os file = os.path.abspath(file) path = os.path.dirname(file) # Next line should check for (.pyo, .pyc) files too. while os.path.exists(os.path.join(path, '__init__.py')): path = os.path.abspath(os.path.join(path, os.pardir)) return path # Library imports import sys sys.path = [pkg_dir(__file__)] + sys.path # Absolute package imports here ... tests(): ... if __name__ == '__main__': tests() This does what you would expect it to do. It always finds it's own packages modules and it's not sensitive to what location you run it from. But it's bit bulky to be putting everywhere. The real problem isn't that you might get errors, it is that you *may not*. For example, if you decide to modify an existing package, and so copy it to a new location for editing. Then you do your edits, and after everything looks good you run the tests in the tests sub module. It passes, Yay! So then you copy it back to it's original location. Unfortunately the tests may have imported the unedited modules in the live package. So the edited package you replaced it with may have errors. How serious that is depends on what the package is used for. Those mars satellites can get pretty expensive after you loose 2 or 3 of them. ;-) I admit that's pretty unlikely because the package would probably go though some pretty rigorous testing in that case. But still, there is the chance of something like that happening in more casual circumstances. I don't think this is that far fetched. And the other benefit is it just makes it easier to run modules in different circumstances without problems. On the path, directly, indirectly... while editing. That is really what I want at the moment. It should *just work*. Cheers, Ron