PYTHONPATH issue with sibling package names

[Stuart Moffatt]

On Sep 10, 10:12 am, "Diez B. Roggisch" <de... at nospam.web.de> wrote:
> Stuart Moffatt wrote:
> > Environment: Eclipse 3.4.2, Windows XP Pro SP2, Pydev 1.4.4, python
> > 2.6
>
> > When I work in eclipse with java, I like to break up my client and
> > server packages, like this:
>
> > client-project/src/org/me/client
>
> > server-project/src/org/me/api
> > server-project/src/org/me/dao
> > server-project/src/org/me/entity
> > server-project/src/org/me/<etc>
>
> > Then, when I need to call API code from the client, I make sure the
> > API src is on the path.
>
> > I am trying setup pydev projects to do the same thing, but running
> > into an ImportError because my client code can't see the server src,
> > even though it is on the path.
>
> > Specifically, I am trying to import an entity from the server code
> > into the client, like this:
>
> >    from org.me.entity import MyEntity
>
> > If I do this from any module in the server project it is fine (because
> > the src path is in the same eclipse project). But if I do it from
> > anywhere in the client code I get the ImportError
>
> > From what I can tell, python asks for the "closest" module path, which
> > is the current project. It finds org.me, but there is only the client
> > sub-package. The org.me.entity sibling is in another eclipse project,
> > but even though that path is on the PYTHONPATH, python stopped looking
> > after it found a similarly named parent package.
>
> > Is there a trusted way to make sure python looks through all paths for
> > sibling packages? Can I load 'org.me.client' from one path in
> > PYTHONPATH and 'org.me.*' from another path in PYTHONPATH? I imagine
> > if I try to force python to load the second package first that the
> > same thing will happen in reverse.
>
> The solution you are searching for is called "namespace packages", and you
> can read more about it here:
>
>  http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
>
> Do yourself a favor though, and don't use those several-steps-namespaces.
> This is Python, not Java - two levels at *most*, normally a
> project-namespace should be enough.
>
> Diez


Diez,

Thanks for the tips re: namespace packages. Yeah, I know this is
python, but for very large projects (or multiple projects for very
large clients) it is just more flexible to stick to the reverse-dot
java notation of "domain.organization.project.namespace".

All I had to do was make sure that top- and mid-level folders had an
__init__.py with this line:

__import__('pkg_resources').declare_namespace(__name__)

in it (and nothing else).

So, for my example, I had:

client-project/src/org/__init__.py
client-project/src/org/me/__init__.py
server-project/src/org/__init__.py
server-project/src/org/me/__init__.py

...all with the special namespace declaration line above.

Note that the lowest level folders in the package:

client-project/src/org/me/client
server-project/src/org/me/api
server-project/src/org/me/<etc>

...all have __init__.py files with actual module code, and therefore
do NOT have the namespace declaration line.

To expose the server code (org.me.api, org.me.dao, etc) I have a
setup.py in the server-project/src folder with:

from setuptools import setup
setup(name='myserver',
version='1.0',
namespace_packages = ['org','org.me'],
packages=['org.me.api','org.me.dao','org.me.entity'],
)

A similar setup.py can be created for the client library too. Notice
that the setup has every level (except the last) in the
namespace_packages list argument. This is how it walks down the tree
and builds the namespace properly.

With this setup file I can make an .egg or .zip distribution using:

  cd server-project/src
  python setup.py bdist

  or

  python setup.py bdist_egg

For my dev environment in eclipse I also wanted the convenience of
editing server code AND have the latest code on the path for my client
project (without rebuilding a dist). To do this I just:

  cd server-project/src
  python setup.py develop

...which puts a symlink (even does it properly in Windows) in my
python/lib/site-packages folder. Now in my client code I can do a
regular import, even though the code lives in another project, and
though portions of the code base are at different PYTHONPATHs:

  from org.me.entity import MyEntity # from server-project/src
  from org.me.api import MyAPI # from server-project/src
  from org.me.client import MyClient # from client-project/src

When my code is complete, the myserver-1.0-py2.6.egg and myclient-1.0-
py2.6.egg can be included in other python modules by loading compiled
source from inside the egg with:

  from pkg_resources import require
  require("myserver>=1.0")
  require("myclient>=1.0")

and then:

  from org.me.entity import MyEntity # from compiled code in
myserver-1.0-py2.6.egg
  from org.me.api import MyAPI # from compiled code in myserver-1.0-
py2.6.egg
  from org.me.client import MyClient # from compiled code in
myclient-1.0-py2.6.egg


Again, it might seem like a headache to some python developers, but it
replicates fairly well the java way of keeping code organized, but
"merging" it into a virtual namespace (encapsulated in an egg archive)
at run-time.

Hope that helps other java developers in packaging python code.

Stuart