Re: [Distutils] How can I get setuptools to include files beyond what find_packages() finds?
Robert Kern wrote: Stanley A. Klein wrote:
I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I got an error message I didn't understand.
I'd like to help, but if you don't copy the exact error message here, I can't.
OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in numpy/distutils/commands/install.py: import os import sys if 'setuptools' in sys.modules: import setuptools.command.bdist_rpm as old_bdist_rpm class bdist_rpm(old_bdist_rpm): pass else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm class bdist_rpm(old_bdist_rpm): def _make_spec_file(self): spec_file = old_bdist_rpm._make_spec_file(self) # Replace hardcoded setup.py script name # with the real setup script name. setup_py = os.path.basename(sys.argv[0]) if setup_py == 'setup.py': return spec_file new_spec_file = [] for line in spec_file: line = line.replace('setup.py',setup_py) new_spec_file.append(line) return new_spec_file And here is the error message I got (using the kiva setup.py): [stan@localhost enthought.kiva_2.0]$ python setup.py bdist_rpm Traceback (most recent call last): File "setup.py", line 2, in ? from numpy.distutils.core import setup File "/usr/lib/python2.4/site-packages/numpy/distutils/core.py", line 32, in ? from numpy.distutils.command import bdist_rpm File "/usr/lib/python2.4/site-packages/numpy/distutils/command/bdist_rpm.py", line 6, in ? class bdist_rpm(old_bdist_rpm): TypeError: Error when calling the metaclass bases module.__init__() takes at most 2 arguments (3 given) BTW, below is what the unpackaged files error looks like (with a traceback generated by having the DISTUTILS_DEBUG environment variable set). In a much earlier thread I learned that this can usually be resolved by setting [install] optimize=1 in setup.cfg. --------------------------------------------------------------------------- Installed (but unpackaged) file(s) found: /usr/lib/python2.4/site-packages/enthought/__init__.pyc /usr/lib/python2.4/site-packages/enthought/__init__.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/benchmark.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/benchmark.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/conv.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/conv.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/dash.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/dash.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/lion.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/lion.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/lion_data.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/lion_data.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/polygon_hit_test.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/polygon_hit_test.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/rect.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/rect.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple2.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple2.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple_clip.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/simple_clip.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star1.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star1.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star2.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star2.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star_path.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/star_path.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/sub_path.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/sub_path.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/text_ex.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/text_ex.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/tk_ex.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/tk_ex.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_brain.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_brain.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_demo.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_demo.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_image_copy.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_image_copy.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_lines.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_lines.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_lion.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/examples/wx_lion.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/affine_matrix_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/clip_to_rect_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/compiled_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/gcmemtest.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/graphics_context_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/image_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/join_stroke_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/join_stroke_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/points_in_polygon_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/points_in_polygon_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/rgba_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/rgba_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/stroke_path_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/stroke_path_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_arc.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_arc.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_draw_dash.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_draw_dash.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_ft.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_ft.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_graphics_context_system.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_graphics_context_system.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_image3.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_image3.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_save.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_save.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_utils.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/test_utils.pyo /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/unicode_font_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/agg/tests/unicode_font_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/affine_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/affine_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/basecore2d_test_case.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/basecore2d_test_case.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/mac_simpletest.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/mac_simpletest.pyo /usr/lib/python2.4/site-packages/enthought/kiva/tests/macport_test.pyc /usr/lib/python2.4/site-packages/enthought/kiva/tests/macport_test.pyo Traceback (most recent call last): File "setup.py", line 108, in ? version = '2.0b1', File "/usr/lib/python2.4/site-packages/numpy/distutils/core.py", line 174, in setup return old_setup(**new_attr) File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/distutils/command/bdist_rpm.py", line 365, in run self.spawn(rpm_cmd) File "/usr/lib/python2.4/distutils/cmd.py", line 398, in spawn spawn(cmd, search_path, dry_run= self.dry_run) File "/usr/lib/python2.4/distutils/spawn.py", line 37, in spawn _spawn_posix(cmd, search_path, dry_run=dry_run) File "/usr/lib/python2.4/distutils/spawn.py", line 165, in _spawn_posix raise DistutilsExecError, \ distutils.errors.DistutilsExecError: command 'rpmbuild' failed with exit status 1
Stanley A. Klein wrote:
Robert Kern wrote: Stanley A. Klein wrote:
I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I got an error message I didn't understand.
I'd like to help, but if you don't copy the exact error message here, I can't.
OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in numpy/distutils/commands/install.py:
import os import sys
if 'setuptools' in sys.modules: import setuptools.command.bdist_rpm as old_bdist_rpm class bdist_rpm(old_bdist_rpm): pass
else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm
Note the difference between these two lines. You imported a module, not the class inside it. Try this instead: if 'setuptools' in sys.modules: from setuptools.command.bdist_rpm import bdist_rpm as old_bdist_rpm else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
At 05:16 PM 7/31/2007 -0400, Stanley A. Klein wrote:
Robert Kern wrote: Stanley A. Klein wrote:
I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I got an error message I didn't understand.
I'd like to help, but if you don't copy the exact error message here, I can't.
OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in numpy/distutils/commands/install.py:
Please see my earlier message about "setup.py install --root --record" and try that -- it will give us the information to know whether there's even a problem with numpy's bdist_rpm, or if it's in numpy's install_lib or build_ext commands. There's no point in changing the commands around before that.
At 05:16 PM 7/31/2007 -0400, Stanley A. Klein wrote:
Robert Kern wrote: Stanley A. Klein wrote:
I tried to do something to fix the numpy distutils bdist_rpm.py (by
On Tue, 2007-07-31 at 18:05 -0400, Phillip J. Eby wrote: trying to follow what was done in install.py) but it didn't work and I
got an error message I didn't understand.
I'd like to help, but if you don't copy the exact error message here, I can't.
OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in numpy/distutils/commands/install.py:
Please see my earlier message about "setup.py install --root --record" and try that -- it will give us the information to know whether there's even a problem with numpy's bdist_rpm, or if it's in numpy's install_lib or build_ext commands. There's no point in changing the commands around before that.
I ran it and think I have figured out what is happening. I first ran the build, using "python setup.py build" rather than the "env CFLAGS="$RPM_OPT_FLAGS" python setup.py build" that appears in the rpm spec file built by bdist_rpm. I don't know what difference that makes. The build looked OK. I then created a buildroot directory and ran "python setup.py install --root=build/buildroot --record=INSTALLED_FILES". The install looked OK as well. It included the .so files and also built the .pyo and .pyc files for the .py files that are used during kiva execution and resulted from the [install] optimize=1 in setup.cfg. I saved the buildroot directory and INSTALLED_FILES. I then cleared the build directory and ran "python setup.py bdist_rpm". It gave me the unpackaged files error as before. I looked at exactly what files were unpackaged. They turned out to include two kinds of files: (1) some .pyc and .pyo files for .py files in example and test directories, and (2) a pair of .pyc and .pyo files for the __init__.py at the top level of the enthought directory that contains the statement for proper processing of namespace packages. I'm sure these .pyc and .pyo files are coming from brp-python-bytecompile that Fedora rpmbuild runs without requiring __init__.py files in the directories it processes. The workaround for the example and test directories is to remove them from the package. Under the Unix File Hierarchy Standard, they don't belong in /usr/lib/python/site-packages but in /usr/share/packagename+ or /usr/share/doc/packagename+ (where the "+" indicates adding version information). I had to do this with another enthought component, and for that I put them in a directory intended for building a package to install them in the right location per the FHS. I don't know what to do about the .pyc and .pyo files for the __init__.py. It looks like what might be happening is that the rpm spec file is being built by a combination of the distutils and numpy distutils functions (which differ from setuptools in the scripts they put in the spec file, including the env statement build difference shown above and omitting " --single-version-externally-managed" from the install options). Then it looks like the build might be executed by the numpy distutils but the install is being done by setuptools. The install (both what I executed alone and when done by bdist_rpm) includes the egg-info directory and the nspkg.pth file. BTW, differencing the bdist_rpm INSTALLED_FILES and my earlier test INSTALLED_FILES showed the only difference being the naming of the files in the egg-info and nspkg.pth files. I think this clarifies what is happening, although I'm not sure how to fix the issue with the __init__.py file. Many thanks for insisting that I run the test you suggested. I hope this helps make the issues clear. Stan Klein
Robert Kern wrote:
Stanley A. Klein wrote:
Robert Kern wrote: Stanley A. Klein wrote:
I tried to do something to fix the numpy distutils bdist_rpm.py (by trying to follow what was done in install.py) but it didn't work and I got an error message I didn't understand.
I'd like to help, but if you don't copy the exact error message here, I can't.
OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in numpy/distutils/commands/install.py:
import os import sys
if 'setuptools' in sys.modules: import setuptools.command.bdist_rpm as old_bdist_rpm class bdist_rpm(old_bdist_rpm): pass
else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm
Note the difference between these two lines. You imported a module, not the class inside it. Try this instead:
if 'setuptools' in sys.modules: from setuptools.command.bdist_rpm import bdist_rpm as old_bdist_rpm else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm
Thanks. I tried it and it ran. I'm still having problems getting things to work, but as you can see from my previous email I've closed in on the issues, that now appear to involve the packaging of documentation, tests, and examples, and the handling of the namespace package __init__.py. Right now, I think my problems involve svn, relating to my moving the tests and examples to a different directory but somehow (probably because setuptools looks at svn information) having files I moved out of the enthought.kiva directory still in the build and creating problems, or files not in the current version expected for the build and found missing. I didn't follow proper svn procedure in moving files out, and I think that is coming back to bite me, although what I did seemed to work with another component. My best bet at this point is to re-checkout the enthought.branches directory and do it right from an svn viewpoint. Stan Klein
Robert Kern wrote:
Stanley A. Klein wrote:
Robert Kern wrote: Stanley A. Klein wrote:
I tried to do something to fix the numpy distutils bdist_rpm.py (by
got an error message I didn't understand.
I'd like to help, but if you don't copy the exact error message here, I can't.
OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying -- obviously not well -- to follow what was done in numpy/distutils/commands/install.py:
import os import sys
if 'setuptools' in sys.modules: import setuptools.command.bdist_rpm as old_bdist_rpm class bdist_rpm(old_bdist_rpm): pass
else: from distutils.command.bdist_rpm import bdist_rpm as
Note the difference between these two lines. You imported a module, not
class inside it. Try this instead:
if 'setuptools' in sys.modules: from setuptools.command.bdist_rpm import bdist_rpm as old_bdist_rpm else: from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm
Thanks.
I tried it and it ran.
I'm still having problems getting things to work, but as you can see from my previous email I've closed in on the issues, that now appear to involve the packaging of documentation, tests, and examples, and the handling of
On Thu, 2007-08-02 at 12:11 -0400, Stanley A. Klein wrote: trying to follow what was done in install.py) but it didn't work and I old_bdist_rpm the the namespace package __init__.py.
Right now, I think my problems involve svn, relating to my moving the
setuptools looks at svn information) having files I moved out of the enthought.kiva directory still in the build and creating problems, or files not in the current version expected for the build and found missing.
I didn't follow proper svn procedure in moving files out, and I think
tests and examples to a different directory but somehow (probably because that
is coming back to bite me, although what I did seemed to work with another component. My best bet at this point is to re-checkout the enthought.branches directory and do it right from an svn viewpoint.
I may have spoken too soon about it working. I'm now getting a completely incomprehensible error during the build step. If I independently run "python setup.py build" it works. If I run "python setup.py bdist_rpm" it creates an rpm spec file with substantially the same statement for the build and I get the traceback below. If I make the build statement exactly the same by specifying --no-rpm-opt-flags the behavior doesn't change. The traceback is (picking up slightly earlier in the output): ------------ adding 'enthought/kiva/agg/freetype2/src/sfnt/sfnt.c' to sources. adding 'enthought/kiva/agg/freetype2/src/smooth/smooth.c' to sources. adding 'enthought/kiva/agg/freetype2/src/truetype/truetype.c' to sources. adding 'enthought/kiva/agg/freetype2/src/type1/type1.c' to sources. adding 'enthought/kiva/agg/freetype2/src/type42/type42.c' to sources. adding 'enthought/kiva/agg/freetype2/src/winfonts/winfnt.c' to sources. adding 'enthought/kiva/agg/freetype2/src/gzip/ftgzip.c' to sources. building library "agg24_src" sources building library "kiva_src" sources building extension "enthought.kiva.agg._agg" sources Traceback (most recent call last): File "setup.py", line 117, in ? version = '2.0b1', File "/usr/lib/python2.4/site-packages/numpy/distutils/core.py", line 174, in setup return old_setup(**new_attr) File "/usr/lib/python2.4/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands self.run_command(cmd) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/lib/python2.4/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command cmd_obj.run() File "/usr/lib/python2.4/site-packages/numpy/distutils/command/build_src.py", line 87, in run self.build_sources() File "/usr/lib/python2.4/site-packages/numpy/distutils/command/build_src.py", line 106, in build_sources self.build_extension_sources(ext) File "/usr/lib/python2.4/site-packages/numpy/distutils/command/build_src.py", line 216, in build_extension_sources sources = self.swig_sources(sources, ext) File "/usr/lib/python2.4/site-packages/numpy/distutils/command/build_src.py", line 564, in swig_sources raise ValueError("%r missing" % (target_file,)) ValueError: 'agg_wrap.c' missing error: Bad exit status from /var/tmp/rpm-tmp.37529 (%build) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.37529 (%build) error: command 'rpmbuild' failed with exit status 1 ----------------------------- The problem is that when the build is done under the rpmbuild spec build script (which is the same as the command line used for an ordinary build), somehow it wants to look for a file agg_wrap.c. There is no file agg_wrap.c. There *is* a file agg_wrap.cpp, that it properly finds and uses when running "python setup.py build". I've looked at the numpy distutils code and can't imagine what would cause such a weird problem. Any suggestions? Stan Klein
participants (3)
-
Phillip J. Eby -
Robert Kern -
Stanley A. Klein