Yes, the https://github.com/pypa/pip/issues/3 definitely sounds like my issue. Seems there is some concern over the nspkg.pth there as well. It seems taht the nspkg.pth is a fine idea to replace the install of the __init__.py, but that it just doesn’t work to fully extend the locations in which a namespace can reside.
I now understand what the issue is and can help perhaps with some advice below.
Either way, there a few other workarounds posted there as well, I will check them out to see if any of them are more palatable than the one I posted here.
You need to make foo.bar-1.0.0-py2.7-nspkg.pth look like this:
import sys, types, os, pkgutil; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('foo',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('foo',types.ModuleType('foo')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p); mp[:] = m and pkgutil.extend_path(mp, 'foo') or mp
Notice the difference from the default in use of pkgutil.extend_path() that helps with editable packages.
How to make this happen automatically is explained below.
Thanks Lele for sending that link!
-Kevin
I also want to thank Lele for pointing out the possible cause of your issues.
Now, I’d like to separate two things here:
1. Defining namespaces correctly — that should be done as we talked before in the standard way.
2. Issue with the tooling — in this case pip is the tool that causes issues.
Let’s talk about it more. These are the reasons I didn’t recognize Kevin’s issue. :-)
- The namespaces work great when defined correctly (#1 above) with zc.buildout and other tools.
There are good open-source build systems out there that take care of namespaces properly.
- When one makes a good custom build system, it can fix the pip deficiencies too.
For example, the build system specifically used in a company, team, …
So, my advice is to:
1. Make a custom distutils Distribution extension, for example, MyDistribution (or some better name)
(a) make custom install_egg_info class that overrides either:
- two templates for pth files (for newer versions of setuptools) or
- install_namespaces method (for older versions of setuptools)
(b) override get_command_class method in the MyDistribution class with your own install_egg_info
2. Build this package and install it.
3. Use distclass=MyDistribution in your foo package setup.py (import it from your package).
Using this approach everything will work for every of your packages automatically.
I attached your original code from foo_test.zip and added to it a minimal mydist package + a sample __init__.py file that I think you should use in your namespace packages (if you want).
Kevin, you can try this by doing the following:
- create a virtual environment
- source bin/activate
- go to mydist package and run “python setup.py sdist”
- pip install /path/to/mydist-1.0.0.tar.gz
- go to your foo_bar package and run “python setup.py sdist”
- pip install /path/to/foo.bar-1.0.0.tar.gz
- pip install -e /path/to/foo_more
- start the python interpreter in your virtual environment
- confirm that it works
>>> import foo.bar
foo.bar imported
>>> import foo.more
foo.more imported
I know that this is not a fix for pip, but it is a generic solution that I hope helps you.
Perhaps I should make a little better example (better names, better __init__.py, …) and post it as a little project on GitHub that people can refer to.
All the best,
Zvezdan