[Distutils] How can I get setuptools to include files beyond what find_packages() finds?

[Stanley A. Klein]

Oops, I forgot to copy the subject line.

Phillip J. Eby wrote:
> At 03:15 PM 7/30/2007 -0400, Stanley A. Klein wrote:
>>> I don't need to build the .so files.  They are already built.  That
had to
>>> be done using the build-in-place and the numpy distutils for reasons I
don't fully understand but are related to the use of numpy.
>
>> Have you tried building them with setuptools, using the numpy
>> distutils 'build_ext' command, using:
>>
>>     setup(
>>        cmdclass = dict(build_ext = numpy_build_ext_class_here),
ext_modules = list_of_Extension_objects,
>>        ...
>>     )
>>
>> Unless there is a radical difference between numpy distutils and the
regular distutils, you should be able to do this.  Just find numpy's
"build_ext" class, and define the appropriate Extension() objects (for
the things to build) in your setup script.  Setuptools will then
delegate the building to numpy, but handle the installing itself.
>
>> Again, this is assuming that numpy's distutils extensions don't do
anything unfriendly like completely redefine how extension objects work
or assume that their commands will be only mixed with other numpy
commands.  (Setuptools doesn't make such assumptions, and tries to
leave the normal distutils stuff alone as much as possible.)

>I think we're getting into confusing territory by trying to get
workarounds >for workarounds. Let me try to take us a step back and focus
on the initial >problem which is that bdist_rpm is not working with
enthought.kiva. The >existing setup script already does build extensions
just fine; they're just >not being picked up by bdist_rpm. A suggestion
from a coworker of mine >prompted Stanley to look at using a script that
we have for building >enthought.kiva inplace (there are a few more options
that are needed beyond >"python setup.py develop"); however, it wasn't
really a suggestion to use >that as basis for building an RPM.
>
>numpy.distutils extends distutils in three ways which are important for
enthought.kiva:
>
>  * automatically adds the location of the numpy headers to the
>include_dirs of Extensions. (easily replaced)
>
>  * adds a build_src command that allows users to give Python functions
in >the sources list of an Extension. These functions will be called to
>actually
>generate the real source files. (hard to replace)
>
>  * allows subpackages to have their own build information which is
>assembled by the top-level setup.py script. This is mostly legacy from
when >the enthought package was monolithic and doesn't strictly need to
continue. >I won't go into details since I don't think it's part of the
problem. >(straightforward, but time-consuming to replace)

>numpy.distutils tries hard to not step on setuptools' toes. We actually
check if setuptools is in sys.modules and use its command classes
instead >of distutils' as the base classes for our commands. However, it's
possible >that neglect of our bdist_rpm implementation has caused the
implementations >to diverge and some toe-stepping has taking place.

>The main problem is that bdist_rpm is not working on enthought.kiva. Most
likely, this is the fault of numpy.distutils. However, this is a bug that
needs to be caught and fixed. Working around it by doing an --inplace
build >and then trying to include the extension modules as package_data is
not >likely to work and is not a solution.
>
>I'm not usually a Redhat guy, so I don't have much experience with
bdist_rpm; however, numpy.distutils has had problems with bdist_rpm in
the >past. I'm trying to get an environment working on a Redhat machine,
and >will try to build an RPM for enthought.kiva and try to see the
problem >first-hand. I've looked over Stanley's emails on the subject, and
don't see >enough information for me to really pin down the problem.

Robert -

Thanks for illuminating the issue.

The problem I had was as follows.  Fedora (also RedHat) uses SE-Linux,
which needs to know all the files expected to be in sensitive directories
such as the Python site-packages.  This includes the pyc and pyo files
ordinarily generated by Python as the .py files are executed.

It turns out that to do a bdist_rpm for Fedora, it is necessary to create
a setup.cfg file containing the lines:

[install]
optimize = 1

or to add these lines to the existing setup.cfg file.

If that is not done, the result in Fedora is an unpackaged files error. 
This is due to the fact that if distutils/setuptools doesn't cause the pyc
and pyo files to be created, Fedora will create them but they won't be
properly handled in the spec file created by distutils/setuptools.

In trying to do bdist_rpm with kiva, I got the unpackaged files error. 
This implies that numpy distutils did not properly handle the optimize=1
in the setup.cfg (when I did "python setup.py bdist_rpm").  That's when I
went to the workaround that resulted in this thread.

I hope this clarifies the problem.

Thanks.


Stan Klein



-- 
Stanley A. Klein, D.Sc.
Managing Principal
Open Secure Energy Control Systems, LLC
8070 Georgia Avenue
Silver Spring, MD 20910
301-565-4025