How can I get setuptools to include files beyond what find_packages() finds?
I'm trying to package an rpm for enthought kiva. The regular setup.py uses the numpy distutils because of some cpp functions that have to be compiled and are somehow tied to numpy. Someone recently did a "build in place" program that uses the existing setup.py (that I renamed setup.original.py) and builds the .so files in the regular source directory hierarchy. I did that and then tried to run setuptools (python setup.py bdist_rpm) using a straightforward setup.py. It included all the python files but missed the *.so files. I can run kiva examples if I manually put the *.so files into the proper place in site-packages, so I know they are needed. I tried including packages = find_packages(), package_data = {'': ['*.so']}, in the setup.py, but it still missed the *.so files. What am I doing wrong? Stan Klein
At 03:47 PM 7/27/2007 -0400, Stanley A. Klein wrote:
I'm trying to package an rpm for enthought kiva. The regular setup.py uses the numpy distutils because of some cpp functions that have to be compiled and are somehow tied to numpy.
Someone recently did a "build in place" program that uses the existing setup.py (that I renamed setup.original.py) and builds the .so files in the regular source directory hierarchy. I did that and then tried to run setuptools (python setup.py bdist_rpm) using a straightforward setup.py. It included all the python files but missed the *.so files. I can run kiva examples if I manually put the *.so files into the proper place in site-packages, so I know they are needed.
I tried including
packages = find_packages(), package_data = {'': ['*.so']},
in the setup.py, but it still missed the *.so files.
What am I doing wrong?
You don't want package_data. setup() needs to actually list the extensions. More precisely, the distutils "install_lib" command needs to install the extensions when requested to do so by bdist_rpm. The preceding applies to any distutils setup, not just setuptools.
At 03:47 PM 7/27/2007 -0400, Stanley A. Klein wrote:
I'm trying to package an rpm for enthought kiva. The regular setup.py uses the numpy distutils because of some cpp functions that have to be compiled and are somehow tied to numpy.
Someone recently did a "build in place" program that uses the existing setup.py (that I renamed setup.original.py) and builds the .so files in
On Fri, 2007-07-27 at 15:55 -0400, Phillip J. Eby wrote: the regular source directory hierarchy. I did that and then tried to run
setuptools (python setup.py bdist_rpm) using a straightforward setup.py. It included all the python files but missed the *.so files. I can run kiva examples if I manually put the *.so files into the proper place in site-packages, so I know they are needed.
I tried including
packages = find_packages(), package_data = {'': ['*.so']},
in the setup.py, but it still missed the *.so files.
What am I doing wrong?
You don't want package_data. setup() needs to actually list the extensions. More precisely, the distutils "install_lib" command needs to install the extensions when requested to do so by bdist_rpm.
The preceding applies to any distutils setup, not just setuptools.
This is very confusing. The documentation of distutils is very sparse in this area and doesn't really cover this issue. I looked at the code and couldn't figure out anything. I got a little bit out of the "--help install_lib" command line option, but couldn't figure it out from that. It appears that for extensions, distutils expects to build/compile them. There is some kind of option called skip-build that I don't know how to invoke and am not sure how will affect optimize=1 in the setup.cfg file, that calls for it to build the pyc and pyo files that are needed. I also couldn't figure out how to specify the .so files to be included. I don't need to build the .so files. They are already built. That had to be done using the build-in-place and the numpy distutils for reasons I don't fully understand but are related to the use of numpy. The reason I tried to use package_data was because I wanted setuptools to include the files the same as would be done if they were data files -- just put them into the package as they are. The find_packages() function identifies .py files to be included in the package. I want the already-built .so files to be included also. How do I make that happen? Stan Klein
At 03:15 PM 7/30/2007 -0400, Stanley A. Klein wrote:
I don't need to build the .so files. They are already built. That had to be done using the build-in-place and the numpy distutils for reasons I don't fully understand but are related to the use of numpy.
Have you tried building them with setuptools, using the numpy distutils 'build_ext' command, using: setup( cmdclass = dict(build_ext = numpy_build_ext_class_here), ext_modules = list_of_Extension_objects, ... ) Unless there is a radical difference between numpy distutils and the regular distutils, you should be able to do this. Just find numpy's "build_ext" class, and define the appropriate Extension() objects (for the things to build) in your setup script. Setuptools will then delegate the building to numpy, but handle the installing itself. Again, this is assuming that numpy's distutils extensions don't do anything unfriendly like completely redefine how extension objects work or assume that their commands will be only mixed with other numpy commands. (Setuptools doesn't make such assumptions, and tries to leave the normal distutils stuff alone as much as possible.)
Phillip J. Eby wrote:
At 03:15 PM 7/30/2007 -0400, Stanley A. Klein wrote:
I don't need to build the .so files. They are already built. That had to be done using the build-in-place and the numpy distutils for reasons I don't fully understand but are related to the use of numpy.
Have you tried building them with setuptools, using the numpy distutils 'build_ext' command, using:
setup( cmdclass = dict(build_ext = numpy_build_ext_class_here), ext_modules = list_of_Extension_objects, ... )
Unless there is a radical difference between numpy distutils and the regular distutils, you should be able to do this. Just find numpy's "build_ext" class, and define the appropriate Extension() objects (for the things to build) in your setup script. Setuptools will then delegate the building to numpy, but handle the installing itself.
Again, this is assuming that numpy's distutils extensions don't do anything unfriendly like completely redefine how extension objects work or assume that their commands will be only mixed with other numpy commands. (Setuptools doesn't make such assumptions, and tries to leave the normal distutils stuff alone as much as possible.)
I think we're getting into confusing territory by trying to get workarounds for workarounds. Let me try to take us a step back and focus on the initial problem which is that bdist_rpm is not working with enthought.kiva. The existing setup script already does build extensions just fine; they're just not being picked up by bdist_rpm. A suggestion from a coworker of mine prompted Stanley to look at using a script that we have for building enthought.kiva inplace (there are a few more options that are needed beyond "python setup.py develop"); however, it wasn't really a suggestion to use that as basis for building an RPM. numpy.distutils extends distutils in three ways which are important for enthought.kiva: * automatically adds the location of the numpy headers to the include_dirs of Extensions. (easily replaced) * adds a build_src command that allows users to give Python functions in the sources list of an Extension. These functions will be called to actually generate the real source files. (hard to replace) * allows subpackages to have their own build information which is assembled by the top-level setup.py script. This is mostly legacy from when the enthought package was monolithic and doesn't strictly need to continue. I won't go into details since I don't think it's part of the problem. (straightforward, but time-consuming to replace) numpy.distutils tries hard to not step on setuptools' toes. We actually check if setuptools is in sys.modules and use its command classes instead of distutils' as the base classes for our commands. However, it's possible that neglect of our bdist_rpm implementation has caused the implementations to diverge and some toe-stepping has taking place. The main problem is that bdist_rpm is not working on enthought.kiva. Most likely, this is the fault of numpy.distutils. However, this is a bug that needs to be caught and fixed. Working around it by doing an --inplace build and then trying to include the extension modules as package_data is not likely to work and is not a solution. I'm not usually a Redhat guy, so I don't have much experience with bdist_rpm; however, numpy.distutils has had problems with bdist_rpm in the past. I'm trying to get an environment working on a Redhat machine, and will try to build an RPM for enthought.kiva and try to see the problem first-hand. I've looked over Stanley's emails on the subject, and don't see enough information for me to really pin down the problem. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
At 05:17 PM 7/30/2007 -0500, Robert Kern wrote:
The main problem is that bdist_rpm is not working on enthought.kiva. Most likely, this is the fault of numpy.distutils. However, this is a bug that needs to be caught and fixed. Working around it by doing an --inplace build and then trying to include the extension modules as package_data is not likely to work and is not a solution.
Yep; that's why I recommend finding the actual problem -- thanks for pointing it out more specifically.
I'm not usually a Redhat guy, so I don't have much experience with bdist_rpm; however, numpy.distutils has had problems with bdist_rpm in the past.
The keys to making bdist_rpm work for a distutils extension are: 1. Make sure you install stuff to the right place 2. Implement get_outputs() correctly The most likely source of the problem is if your get_outputs() isn't listing the .so files; in that case they'd won't be listed in INSTALLED_FILES, which is used by bdist_rpm. The other possibility is that the files aren't getting installed by install_lib, as install_lib expects to just copy the build tree to the target. If the built files aren't in the right place, then they won't get installed even if they're listed in the files. You can easily test this by running an install command using --root=/some/dir --record=SOME_FILE. If this doesn't install something like "/some/dir/usr/lib/python2.x/site-packages/whatever/something.so", or the file isn't listed in SOME_FILE, you can track it down further from there -- it's not a problem specific to bdist_rpm in that case. You can also see if it's correct with or without setuptools, to see if the interaction is in fact there.
I'm trying to get an environment working on a Redhat machine, and will try to build an RPM for enthought.kiva and try to see the problem first-hand. I've looked over Stanley's emails on the subject, and don't see enough information for me to really pin down the problem.
I'd suggest trying an install --root/--record test first, since if that doesn't work right, you know RPMs aren't even involved. So that'll help narrow down where the actual issue lies.
participants (3)
-
Phillip J. Eby -
Robert Kern -
Stanley A. Klein