Namespace packages and egg-info confusion when using bdist_rpm
I've been trying to build rpms of enthought system components. Some of them use namespace packages. Those packages have the required __init__.py files containing "__import__('pkg_resources').declare_namespace(__name__)". According to the Setuptools Guide, these __init__.py files are not to be packaged when using "system" packaging (such as bdist_rpm) and when I run bdist_rpm they are not included in the rpms. However, there are egg-info directories and nspkg.pth files produced, included in the rpms, and installed in site-packages. When I tried to run the enthought example applications, I had various kinds of import failures. I found that fixing them required putting the unpackaged __init__.py files into the relevant directories in the namespace packages. The various rpms were being placed in the proper locations where they should have been found for importing. There were some strange errors, such as in one case when I tried to test the imports interactively an "import a.b.c" worked, but an "import a.b.c as c" failed. Before putting in the __init__.py files, I tried removing the egg-info and nspkg.pth files from site_packages. That did not improve things. I also noticed that some other packages had pth files in site-packages, so that may be what is needed if the egg-info and nspkg.pth are removed, although I don't know why or what the pth files are supposed to contain. I think setuptools is somehow not properly processing bdist_rpm if there are namespace packages involved. Also, the setuptools don't seem to have an option to not create the egg info and nspkg.pth files and/or to not package them if bdist_rpm is being used. Stan Klein
At 04:42 PM 7/25/2007 -0400, Stanley A. Klein wrote:
I've been trying to build rpms of enthought system components. Some of them use namespace packages. Those packages have the required __init__.py files containing "__import__('pkg_resources').declare_namespace(__name__)".
According to the Setuptools Guide, these __init__.py files are not to be packaged when using "system" packaging (such as bdist_rpm) and when I run bdist_rpm they are not included in the rpms. However, there are egg-info directories and nspkg.pth files produced, included in the rpms, and installed in site-packages.
So far, so good - that's exactly how it's supposed to work.
When I tried to run the enthought example applications, I had various kinds of import failures. I found that fixing them required putting the unpackaged __init__.py files into the relevant directories in the namespace packages.
The various rpms were being placed in the proper locations where they should have been found for importing. There were some strange errors, such as in one case when I tried to test the imports interactively an "import a.b.c" worked, but an "import a.b.c as c" failed.
Before putting in the __init__.py files, I tried removing the egg-info and nspkg.pth files from site_packages. That did not improve things.
It sure as heck wouldn't. They need to be there, and the nspkg.pth files need to be processed by Python during its startup.
I think setuptools is somehow not properly processing bdist_rpm if there are namespace packages involved. Also, the setuptools don't seem to have an option to not create the egg info and nspkg.pth files and/or to not package them if bdist_rpm is being used.
This is by design. It is almost certainly the case that either: 1. the nspkg.pth files are not being processed when Python starts 2. something is deleting the namespace packages from sys.modules between the time the .pth files are processed, and the time you attempt to import them. To verify this, I would say, start Python with the bdist_rpms installed as directed (i.e., don't mess with what's installed), and see if there are sys.modules entries for the namespace package(s). If there are, then the problem is happening later - i.e., something being done by your code. If the namespace packages aren't in sys.modules, then there is a problem with the .pth files at startup; try using "python -v" to see if there are any errors shown.
At 04:42 PM 7/25/2007 -0400, Stanley A. Klein wrote:
I've been trying to build rpms of enthought system components. Some of
files containing "__import__('pkg_resources').declare_namespace(__name__)".
According to the Setuptools Guide, these __init__.py files are not to be packaged when using "system" packaging (such as bdist_rpm) and when I run bdist_rpm they are not included in the rpms. However, there are egg-info directories and nspkg.pth files produced, included in the rpms, and installed in site-packages.
So far, so good - that's exactly how it's supposed to work.
When I tried to run the enthought example applications, I had various kinds of import failures. I found that fixing them required putting the unpackaged __init__.py files into the relevant directories in the namespace packages.
The various rpms were being placed in the proper locations where they should have been found for importing. There were some strange errors, such as in one case when I tried to test the imports interactively an "import a.b.c" worked, but an "import a.b.c as c" failed.
Before putting in the __init__.py files, I tried removing the egg-info and nspkg.pth files from site_packages. That did not improve things.
It sure as heck wouldn't. They need to be there, and the nspkg.pth files need to be processed by Python during its startup.
I think setuptools is somehow not properly processing bdist_rpm if
are namespace packages involved. Also, the setuptools don't seem to have an option to not create the egg info and nspkg.pth files and/or to not
On Wed, 2007-07-25 at 18:10 -0400, Phillip J. Eby wrote: them use namespace packages. Those packages have the required __init__.py there package them if bdist_rpm is being used.
This is by design. It is almost certainly the case that either:
1. the nspkg.pth files are not being processed when Python starts
2. something is deleting the namespace packages from sys.modules
between the time the .pth files are processed, and the time you attempt to import them.
To verify this, I would say, start Python with the bdist_rpms installed as directed (i.e., don't mess with what's installed), and see
package(s). If there are, then the problem is happening later - i.e., something being done by your code. If the namespace packages aren't in sys.modules, then there is a problem with the .pth files at startup;
if there are sys.modules entries for the namespace try using "python -v" to see if there are any errors shown. I think the "almost" allows more than the two alternative possibilities you provide, and I think that is the situation. Something else must be going on. I disabled the __init__.py's on the namespace packages by renaming them. A sys.modules.keys() shows the namespace packages in the list. Then I interactively did the first three statements of the test_traits.py program: import unittest from enthought.traits.api import * import enthought.traits.standard as standard The last statement failed with an error: Traceback (most recent call last): File "test_traits.py", line 23, in ? import enthought.traits.standard as standard AttributeError: 'module' object has no attribute 'traits' If I then interactively do "import enthought.traits.standard", it works without error. If I redo the same steps using python -v, I get the error message only after a lot of imports. The last output before the error message is # /usr/lib/python2.4/site-packages/enthought/traits/ui/editors.pyc matches /usr/lib/python2.4/site-packages/enthought/traits/ui/editors.py import enthought.traits.ui.editors # precompiled from /usr/lib/python2.4/site-packages/enthought/traits/ui/editors.pyc Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'module' object has no attribute 'traits' The file editors.py first does: from toolkit \ import toolkit which points to toolkit.py. That file contains some statements that were the failure point when I tried to run an example from the Traits UI Guide. The relevant statements in toolkit.py are: from enthought.traits.api \ import HasTraits, HasPrivateTraits, TraitError from ui_traits \ import SequenceTypes #-------------------------------------------------------------------------------# Constants: #------------------------------------------------------------------------------- # List of implemented UI toolkits TraitUIToolkits = [ 'wx', 'null' ] #------------------------------------------------------------------------------- # Data: #------------------------------------------------------------------------------- # The current GUI toolkit object being used _toolkit = None #------------------------------------------------------------------------------- # Low-level GUI toolkit selection function: #------------------------------------------------------------------------------- def toolkit ( *toolkits ): """ Selects and returns a low-level GUI toolkit. Use this function to get a reference to the current toolkit. """ global _toolkit if len( toolkits ) == 0: if _toolkit is not None: return _toolkit toolkits = TraitUIToolkits for toolkit_name in toolkits: try: package = 'enthought.traits.ui.' + toolkit_name module = __import__( package ) _toolkit = getattr( module.traits.ui, toolkit_name ).toolkit return _toolkit except ImportError: pass else: raise TraitError, ("Could not find any UI toolkit called: %s" % ', '.join( toolkits )) When I restart the interactive session as python -v, do the import statements in toolkit.py , and then do toolkit_name = "wx" package = 'enthought.traits.ui.' + toolkit_name module = __import__( package ) _toolkit = getattr( module.traits.ui, toolkit_name ).toolkit it finds the enthought.traits.ui.wx toolkit, does the imports required by the "module = __import__( package )" statement and then fails on the next statement with an error:
_toolkit = getattr( module.traits.ui, toolkit_name ).toolkit Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'module' object has no attribute 'traits'
If I manually restore the __init__.py files in the three namespace packages, I don't see the namespace packages on python startup when I do the sys.modules.keys(), but I can run the test_traits.py and the examples from the Traits UI Guide without a problem. In particular, I don't get the attribute error that I got without the __init__.py files in place. Also, I don't know how Ryan May "hand-rolled" his Gentoo packages, but he said he had a similar problem and fixed it by putting in the __init__.py files. It looks like without the __init__.py files, the imports are being properly found and imported, but some lack of information somewhere is causing the getattr statement to fail. The purpose of the statement appears to be to return which lower level GUI toolkit was found and imported. I hope this provides everyone enough information to figure out what is happening. Stan Klein
At 12:34 PM 7/26/2007 -0400, Stanley A. Klein wrote:
I disabled the __init__.py's on the namespace packages by renaming them. A sys.modules.keys() shows the namespace packages in the list. Then I interactively did the first three statements of the test_traits.py program:
import unittest from enthought.traits.api import * import enthought.traits.standard as standard
The last statement failed with an error:
Traceback (most recent call last): File "test_traits.py", line 23, in ? import enthought.traits.standard as standard AttributeError: 'module' object has no attribute 'traits'
If I then interactively do "import enthought.traits.standard", it works without error.
Aha. Try this. In each nspkg.pth file, add "import x" lines, where "x" is any namespace package referenced in that file. That is, if the file refers to enthought.traits, add a line saying "import enthought.traits" at the *end* of the file. Do this in every file, for every module mentioned in that file. Get rid of the __init__.py's and give it a whirl. I think that what's happening is that the .pth files I'm generating are not settting sys.modules['enthought'].traits = sys.modules['enthought.traits']; adding the import statements should fix that. If it works, I'll change setuptools to generate the files with the added import statements. Thanks for taking the time to help track this down. It looks like I only tested this with top--level namespace packages (e.g. peak.*, zope.*) and not sub-packages (like enthought.traits).
On Thu, July 26, 2007 12:46 pm, Phillip J. Eby wrote:
At 12:34 PM 7/26/2007 -0400, Stanley A. Klein wrote:
I disabled the __init__.py's on the namespace packages by renaming them. A sys.modules.keys() shows the namespace packages in the list. Then I interactively did the first three statements of the test_traits.py program:
import unittest from enthought.traits.api import * import enthought.traits.standard as standard
The last statement failed with an error:
Traceback (most recent call last): File "test_traits.py", line 23, in ? import enthought.traits.standard as standard AttributeError: 'module' object has no attribute 'traits'
If I then interactively do "import enthought.traits.standard", it works without error.
Aha. Try this. In each nspkg.pth file, add "import x" lines, where "x" is any namespace package referenced in that file. That is, if the file refers to enthought.traits, add a line saying "import enthought.traits" at the *end* of the file. Do this in every file, for every module mentioned in that file. Get rid of the __init__.py's and give it a whirl.
I think that what's happening is that the .pth files I'm generating are not settting sys.modules['enthought'].traits = sys.modules['enthought.traits']; adding the import statements should fix that. If it works, I'll change setuptools to generate the files with the added import statements.
Thanks for taking the time to help track this down. It looks like I only tested this with top--level namespace packages (e.g. peak.*, zope.*) and not sub-packages (like enthought.traits).
The only three namespace packages are enthought, enthought.traits, and enthought.traits.ui. Here is the full list of pth files: [stan@localhost ~]$ ls /usr/lib/python2.4/site-packages/enthought.*pth /usr/lib/python2.4/site-packages/enthought.etsconfig-2.0b1.dev_r12883-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.io-2.0b1.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.naming-2.0b2.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.pyface-2.0b2.dev_r12984-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.resource-2.0b1.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.resource_type-2.0b2.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.sweet_pickle-2.0b2.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.traits-2.0b2.dev_r12984-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.traits.ui.wx-2.0b2.dev_r12984-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.type_manager-2.0b1.dev_r12810-py2.4-nspkg.pth /usr/lib/python2.4/site-packages/enthought.util-2.0b2.dev_r12981-py2.4-nspkg.pth Most of the files looked like: import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought',new.module('enthought')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) I left those alone, and did only the following two: [stan@localhost ~]$ cat /usr/lib/python2.4/site-packages/enthought.traits-2.0b2.dev_r12984-py2.4-nspkg.pth import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought',new.module('enthought')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought', 'traits')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought.traits',new.module('enthought.traits')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought', 'traits', 'ui')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought.traits.ui',new.module('enthought.traits.ui')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import enthought import enthought.traits import enthought.traits.ui and [stan@localhost ~]$ cat /usr/lib/python2.4/site-packages/enthought.traits.ui.wx-2.0b2.dev_r12984-py2.4-nspkg.pth import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought',new.module('enthought')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought', 'traits')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought.traits',new.module('enthought.traits')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('enthought', 'traits', 'ui')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('enthought.traits.ui',new.module('enthought.traits.ui')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import enthought import enthought.traits import enthought.traits.ui I had the same error. Should I have put in an "import enthought" in all the others? Stan Klein
At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote:
I had the same error. Should I have put in an "import enthought" in all the others?
No, apparently the manual import doesn't help. Presumably the ones you changed need something more like this: import enthought; enthought.traits = sys.modules['enthought.traits'] import enthought.traits; enthought.traits.ui = sys.modules['enthought.traits.ui'] I thought the import would be sufficient, but apparently it's not. :( Again, let me know if it works so I can change setuptools accordingly.
It worked. Problem solved. Thanks. Stan Klein On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote:
At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote:
I had the same error. Should I have put in an "import enthought" in all the others?
No, apparently the manual import doesn't help. Presumably the ones you changed need something more like this:
import enthought; enthought.traits = sys.modules['enthought.traits'] import enthought.traits; enthought.traits.ui = sys.modules['enthought.traits.ui']
I thought the import would be sufficient, but apparently it's not. :(
Again, let me know if it works so I can change setuptools accordingly.
--
On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote:
At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote:
I had the same error. Should I have put in an "import enthought" in all the others? No, apparently the manual import doesn't help. Presumably the ones you changed need something more like this:
import enthought; enthought.traits = sys.modules['enthought.traits'] import enthought.traits; enthought.traits.ui = sys.modules['enthought.traits.ui']
I thought the import would be sufficient, but apparently it's not. :(
Again, let me know if it works so I can change setuptools accordingly.
This solution worked for me initially, but now I'm getting warnings when I run any python script I get: warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits': missing __init__.py warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits/ui': missing __init__.py This is when I added the above lines to the .pth files for both traits and traits.ui. What am I doing wrong? Thanks, Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma
Ryan May wrote:
On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote:
At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote:
I had the same error. Should I have put in an "import enthought" in all the others? No, apparently the manual import doesn't help. Presumably the ones you changed need something more like this:
import enthought; enthought.traits = sys.modules['enthought.traits'] import enthought.traits; enthought.traits.ui = sys.modules['enthought.traits.ui']
I thought the import would be sufficient, but apparently it's not. :(
Again, let me know if it works so I can change setuptools accordingly.
This solution worked for me initially, but now I'm getting warnings when I run any python script I get:
warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits': missing __init__.py warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits/ui': missing __init__.py
This is when I added the above lines to the .pth files for both traits and traits.ui. What am I doing wrong?
Turns out that another package (kiva) decided it would be a good idea to put a __init__.py in the enthought directory, which made python angry. Sorry for the noise, distutils. Enthought guys, anybody wanna take a crack at why kiva (and only kiva) insisted on adding a __init__.py to enthought? Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma
Ryan May wrote:
Ryan May wrote:
On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote:
At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote:
I had the same error. Should I have put in an "import enthought" in all the others?
No, apparently the manual import doesn't help. Presumably the ones you changed need something more like this:
import enthought; enthought.traits = sys.modules['enthought.traits'] import enthought.traits; enthought.traits.ui = sys.modules['enthought.traits.ui']
I thought the import would be sufficient, but apparently it's not. :(
Again, let me know if it works so I can change setuptools accordingly.
This solution worked for me initially, but now I'm getting warnings when I run any python script I get:
warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits': missing __init__.py warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits/ui': missing __init__.py
This is when I added the above lines to the .pth files for both traits and traits.ui. What am I doing wrong?
Turns out that another package (kiva) decided it would be a good idea to put a __init__.py in the enthought directory, which made python angry.
Sorry for the noise, distutils.
Enthought guys, anybody wanna take a crack at why kiva (and only kiva) insisted on adding a __init__.py to enthought?
Well, I just checked the source and 'enthought' is declared as a namespace package in setup.py, and the enthought/__init__.py contains nothing but: try: __import__('pkg_resources').declare_namespace(__name__) except: pass which is what our other projects have. So, since I'm not a bdist_rpm expert, I'm at a loss of even where to begin on this one. -- Dave
On Thu, August 9, 2007 4:02 pm, Ryan May wrote:
On Thu, July 26, 2007 2:01 pm, Phillip J. Eby wrote:
At 01:37 PM 7/26/2007 -0400, Stanley A. Klein wrote:
I had the same error. Should I have put in an "import enthought" in all the others? No, apparently the manual import doesn't help. Presumably the ones you changed need something more like this:
import enthought; enthought.traits = sys.modules['enthought.traits'] import enthought.traits; enthought.traits.ui = sys.modules['enthought.traits.ui']
I thought the import would be sufficient, but apparently it's not. :(
Again, let me know if it works so I can change setuptools accordingly.
This solution worked for me initially, but now I'm getting warnings when I run any python script I get:
warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits': missing __init__.py warning: Not importing directory '/usr/lib64/python2.5/site-packages/enthought/traits/ui': missing __init__.py
This is when I added the above lines to the .pth files for both traits and traits.ui. What am I doing wrong?
Thanks,
With traits and traits.ui, you have to fix the pth file for all three namespace packages. Most of the others only have one namespace package (enthought). BTW, there is now a totally crazy error I get when trying to do bdist_rpm on kiva. If I do "python setup.py build", it works properly. If I do "python setup.py bdist_rpm" it gives a traceback on the build step that uses (or can be made to use) the same command that worked when run alone, but now complains that it can't fine agg_wrap.c, which doesn't exist (the file is agg_wrap.cpp). Stan Klein
participants (4)
-
Dave Peterson -
Phillip J. Eby -
Ryan May -
Stanley A. Klein