[Distutils] Namespace packages and egg-info confusion when using bdist_rpm

[Stanley A. Klein]

On Wed, 2007-07-25 at 18:10 -0400, Phillip J. Eby wrote:
> At 04:42 PM 7/25/2007 -0400, Stanley A. Klein wrote:
> >I've been trying to build rpms of enthought system components.  Some of
them use namespace packages.  Those packages have the required
__init__.py
> >files containing
> >"__import__('pkg_resources').declare_namespace(__name__)".
> >
> >According to the Setuptools Guide, these __init__.py files are not to
be
> >packaged when using "system" packaging (such as bdist_rpm) and when I
run
> >bdist_rpm they are not included in the rpms.  However, there are
egg-info
> >directories and nspkg.pth files produced, included in the rpms, and
installed in site-packages.
>
> So far, so good - that's exactly how it's supposed to work.
>
> >   When I tried to run the enthought example
> >applications, I had various kinds of import failures.  I found that
fixing
> >them required putting the unpackaged __init__.py files into the
relevant
> >directories in the namespace packages.
> >
> >The various rpms were being placed in the proper locations where they
should have been found for importing.  There were some strange errors,
such as in one case when I tried to test the imports interactively an
"import a.b.c" worked, but an "import a.b.c as c" failed.
> >
> >Before putting in the __init__.py files, I tried removing the egg-info
and
> >nspkg.pth files from site_packages.  That did not improve things.
>
> It sure as heck wouldn't.  They need to be there, and the nspkg.pth 
files need to be processed by Python during its startup.
>
>
> >I think setuptools is somehow not properly processing bdist_rpm if
there
> >are namespace packages involved.  Also, the setuptools don't seem to
have
> >an option to not create the egg info and nspkg.pth files and/or to not
package them if bdist_rpm is being used.
>
> This is by design.  It is almost certainly the case that either:
>
> 1. the nspkg.pth files are not being processed when Python starts
>
> 2. something is deleting the namespace packages from sys.modules 
between the time the .pth files are processed, and the time you  attempt
to import them.
>
> To verify this, I would say, start Python with the bdist_rpms
> installed as directed (i.e., don't mess with what's installed), and  see
if there are sys.modules entries for the namespace
> package(s).  If there are, then the problem is happening later -  i.e.,
something being done by your code.  If the namespace packages  aren't in
sys.modules, then there is a problem with the .pth files at  startup;
try using "python -v" to see if there are any errors shown.

I think the "almost" allows more than the two alternative possibilities
you provide, and I think that is the situation.  Something else must be
going on.

I disabled the __init__.py's on the namespace packages by renaming them. A
sys.modules.keys() shows the namespace packages in the list.  Then I
interactively did the first three statements of the test_traits.py
program:

import unittest
from enthought.traits.api import *
import enthought.traits.standard as standard

The last statement failed with an error:

Traceback (most recent call last):
  File "test_traits.py", line 23, in ?
    import enthought.traits.standard as standard
AttributeError: 'module' object has no attribute 'traits'

If I then interactively do "import enthought.traits.standard", it works
without error.

If I redo the same steps using python -v, I get the error message only
after a lot of imports.  The last output before the error message is

# /usr/lib/python2.4/site-packages/enthought/traits/ui/editors.pyc matches
/usr/lib/python2.4/site-packages/enthought/traits/ui/editors.py import
enthought.traits.ui.editors # precompiled
from /usr/lib/python2.4/site-packages/enthought/traits/ui/editors.pyc
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'traits'

The file editors.py first does:

from toolkit \
    import toolkit

which points to toolkit.py.  That file contains some statements that were
the failure point when I tried to run an example from the Traits UI Guide.
 The relevant statements in toolkit.py are:


from enthought.traits.api \
    import HasTraits, HasPrivateTraits, TraitError

from ui_traits \
    import SequenceTypes

#-------------------------------------------------------------------------------#
 Constants:
#-------------------------------------------------------------------------------

# List of implemented UI toolkits
TraitUIToolkits = [ 'wx', 'null' ]

#-------------------------------------------------------------------------------
#  Data:
#-------------------------------------------------------------------------------

# The current GUI toolkit object being used
_toolkit = None

#-------------------------------------------------------------------------------
#  Low-level GUI toolkit selection function:
#-------------------------------------------------------------------------------

def toolkit ( *toolkits ):
    """ Selects and returns a low-level GUI toolkit.

    Use this function to get a reference to the current toolkit.
    """
    global _toolkit

    if len( toolkits ) == 0:
        if _toolkit is not None:
            return _toolkit
        toolkits = TraitUIToolkits
    for toolkit_name in toolkits:
        try:
            package  = 'enthought.traits.ui.' + toolkit_name
            module   = __import__( package )
            _toolkit = getattr( module.traits.ui, toolkit_name ).toolkit
return _toolkit
        except ImportError:
            pass
    else:
        raise TraitError, ("Could not find any UI toolkit called: %s" %
                           ', '.join( toolkits ))

When I restart the interactive session as python -v, do the import
statements in toolkit.py , and then do

toolkit_name = "wx"
package  = 'enthought.traits.ui.' + toolkit_name
module   = __import__( package )
_toolkit = getattr( module.traits.ui, toolkit_name ).toolkit

it finds the enthought.traits.ui.wx toolkit, does the imports required by
the "module   = __import__( package )" statement and then fails on the
next statement with an error:

>>> _toolkit = getattr( module.traits.ui, toolkit_name ).toolkit
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'module' object has no attribute 'traits'


If I manually restore the __init__.py files in the three namespace
packages, I don't see the namespace packages on python startup when I do
the sys.modules.keys(), but I can run the test_traits.py and the
examples from the Traits UI Guide without a problem.  In particular, I
don't get the attribute error that I got without the __init__.py files in
place.

Also, I don't know how Ryan May "hand-rolled" his Gentoo packages, but he
said he had a similar problem and fixed it by putting in the
__init__.py files.

It looks like without the __init__.py files, the imports are being
properly found and imported, but some lack of information somewhere is
causing the getattr statement to fail.  The purpose of the statement
appears to be to return which lower level GUI toolkit was found and
imported.

I hope this provides everyone enough information to figure out what is
happening.


Stan Klein


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070726/9270e050/attachment.html