Jack Jansen has been hacking on the Mac OS support in the Distutils, and brings up an interesting -- but annoying point: installing extension-related header files under Python's include directory could be perceived as interfering with Python's source. I disagree, but let me see if I can summarize what's been going on in email between Jack, Corran Webster (the other person working on Distutils Mac support), and me.
I mentioned how files are installed on Mac OS (by quoting the source, of course):
'mac': { 'purelib': '$base:Lib:site-packages', 'platlib': '$base:Lib:site-packages', 'headers': '$base:Include:$dist_name', 'scripts': '$base:Scripts',
Regarding the 'headers' install dir -- where extension-related header files go -- Jack says:
This one is wrong. Where do you put these things on Unix? I don't particularly like it if distutils scripts go mucking about in my source directories (which is what $base:Include would be).
On Unix, the answer is:
'unix_prefix': { 'purelib': '$base/lib/python$py_version_short/site-packages', 'platlib': '$platbase/lib/python$py_version_short/site-packages', 'headers': '$base/include/python$py_version_short/$dist_name', 'scripts': '$base/bin', 'data' : '$base', },
In the "prefix" installation scheme, $base expands to sys.prefix, so in a standard Python 2.0 installation on Unix, the 'headers' install dir is /usr/local/include/python2.0/$dist_name. $dist_name in turn expands to the name of the Distribution -- "Numerical", "Imaging", whatever.
So yes, we are installing files *under* one of Python's directories, but not *into* it -- the include/python2.0 directory continues to provide just the header files installed by Python, namely Python.h and friends (abstract.h, bitset.h, ..., unicodeobject.h). And that's why I think this isn't a bad thing to do.
The real reason it was done this way, of course, was laziness. Any time you compile a Python extension, you have to include "-I/usr/local/include/python2.0" (or local equivalent). With Numerical Python's headers installed under this location, an extension that needs access to Numerical's C structures just has to say #include <Numerical/arrayobject.h>
and it will Just Work.
Jack had an interesting suggestion:
Moreover, for packages to use one another I would guess that more than just include files is needed (like the location of the shared libraries). I think that my preference (but I haven't thought about it very long) would be to put "distribution data for other distributions" all together in a single place, something like $base/Extensions/packagename/Include $base/Extensions/packagename/Lib
If the shared libraries that go in Extensions/$dist_name/Lib (Jack misspelled it ;-) are C libraries, that's fine -- if they're Python extensions, I'm confused.
This makes sense, but it does require adding another directory to the standard include search path when building Python extensions. And the directory would probably be named something different on Unix. But it's in interesting idea, and I'm not going to throw it out willy-nilly. What does everyone else think?
Corran's response to this was:
I agree, though, that this would be a natural way to go. It might make more sense to have it all inside the site-packages directory, though: headers live in
$base:Lib:site-packages:$package:Include
while modules (both python and C shared libraries) live in
$base:Lib:site-packages:$package:
I disagree: site-packages should be for Python modules and extensions. If we have to install C headers and libraries -- and I suspect we probably do -- then they should find their own home. On Unix, perhaps another directory under <exec-prefix>/lib/python2.0, but I'm not sure what to call it that makes it clear these are for *C* programming, not Python programming. (Granted it's the specialized domain of C programming for developing Python extensions, but still.)
On Mac OS and Windows, it's easy -- I'd say put 'em in Extensions/Includes and Extensions/Lib under <prefix> (where that / is really a : or a , of course).
Jack later follows up:
Ok, so then we have (assuming we're running in a source tree) an "Include" which contains standard Python include files and "includes" which contains package-deposited include files. I'm not too thrilled with the similar names, but I guess I can live with this.
Yechh -- I would be *very* wary of depending on case distinction in filenames! I still don't see what's so wrong with Include/*.h being standard Python headers, and Include/*/*.h being headers provided by whatever extensions are installed on the system.
Greg
On Thu, 14 Sep 2000, Greg Ward wrote:
I disagree: site-packages should be for Python modules and extensions. If we have to install C headers and libraries -- and I suspect we probably do -- then they should find their own home. On Unix, perhaps another directory under <exec-prefix>/lib/python2.0, but I'm not sure what to call it that makes it clear these are for *C* programming, not Python programming. (Granted it's the specialized domain of C programming for developing Python extensions, but still.)
I disagree as well. To vastly over-simplify, why would a python package be installing headers and libraries for C only? Just because someone is installing stuff to help them code python, doesn't mean they want to load everything necessary for C development. This is why the bdist stuff distributes compiled libraries, so the installer doesn't have to deal with rebuilding.
So, if someone wants to provide C interfaces, it should be done seperate from the python interfaces, obviously with the python package dependent on the C libraries. The C interface should also have seperate run-time and development sub-packages, so most people, including those using the python package would only require the run-time. For example,you have gkt+, gtk+-devel, and python-pygtk. python-pygtk would require gkt+, but not gtk+-devel.
I think getting distutils involved with C headers and (C only) libraries is going beyond it's scope without good cause.
Mark Alexander mwa@gate.net
On Fri, Sep 15, 2000 at 10:04:53AM -0400, Mark W. Alexander wrote:
I disagree as well. To vastly over-simplify, why would a python package be installing headers and libraries for C only? Just
Counterexample: ExtensionClass from Digital Creations is a Python-only C module for writing Python classes in C. It needs to install an ExtensionClass.h, which is then used by other C modules (such as ZODB's cPersistence,
I think installing the files under the Python include directory is fine, at least on Unix, since it means if C code can find Python.h, it can also find ExtensionClass/ExtensionClass.h. I'm not sure what to do for Jack's problem.
--amk
On Fri, 15 Sep 2000, Andrew Kuchling wrote:
On Fri, Sep 15, 2000 at 10:04:53AM -0400, Mark W. Alexander wrote:
I disagree as well. To vastly over-simplify, why would a python package be installing headers and libraries for C only? Just
Counterexample: ExtensionClass from Digital Creations is a Python-only C module for writing Python classes in C. It needs to install an ExtensionClass.h, which is then used by other C modules (such as ZODB's cPersistence,
Then maybe I'm lost, but wouldn't the same concept apply?
The only "users" of the C headers/libs would be other C developers; even if they are doing C development for python. Do python users who use the Extension class need the headers, or just the shared libraries that distutils already handles properly?
I think installing the files under the Python include directory is fine, at least on Unix, since it means if C code can find Python.h, it can also find ExtensionClass/ExtensionClass.h. I'm not sure what to do for Jack's problem.
Since the python tree already contains stuff to do C modules, I don't see anything at all "wrong" with putting additional stuff there for use by C developers. I'm just questioning whether it's something that distutils should be responsible for. I think the answer is a definite maybe. It seems like a lot of effort to provide something that may not be a good fit.
Mark Mark
On 15 September 2000, Mark W. Alexander said:
Since the python tree already contains stuff to do C modules, I don't see anything at all "wrong" with putting additional stuff there for use by C developers. I'm just questioning whether it's something that distutils should be responsible for. I think the answer is a definite maybe. It seems like a lot of effort to provide something that may not be a good fit.
Installing C headers is definitely something the Distutils has to do. Just as people write modules that depend on other modules, they also write extensions that depend on other extensions. In C, that kind of dependence usually means two things: include a header file and link against a library. In the specialized domain of writing Python extensions in C, I *think* all the header file is the only problem; the linking issue will be taken care of when both extensions are loaded into the Python interpreter. (This is merely informed speculation; I've never actually written an extension that depends on facilities provided by another extension! Things could well be different on platforms that don't follow the usual Unix model of shared libraries, where .so files can depend on each other, but the dependencies don't have to be resolved until load-time.)
That's why the "install" command can install C header files (using the "install_headers" command). We install .py and .so/.pyd files because they will be needed by Python programmers writing Python modules and applications; we install .h files because they might be needed by C programmers writing Python extensions. We doesn't install any C libraries, either shared or static, because I have not yet seen a need for them.
Greg
On Fri, 15 Sep 2000, Greg Ward wrote:
On 15 September 2000, Mark W. Alexander said:
Since the python tree already contains stuff to do C modules, I don't see anything at all "wrong" with putting additional stuff there for use by C developers. I'm just questioning whether it's something that distutils should be responsible for. I think the answer is a definite maybe. It seems like a lot of effort to provide something that may not be a good fit.
Installing C headers is definitely something the Distutils has to do. Just as people write modules that depend on other modules, they also write extensions that depend on other extensions. In C, that kind of dependence usually means two things: include a header file and link against a library. In the specialized domain of writing Python extensions in C, I *think* all the header file is the only problem; the linking issue will be taken care of when both extensions are loaded into the Python interpreter. (This is merely informed speculation; I've never actually written an extension that depends on facilities provided by another extension! Things could well be different on platforms that don't follow the usual Unix model of shared libraries, where .so files can depend on each other, but the dependencies don't have to be resolved until load-time.)
Ok, I WAS taking my position for the sake of simplicity. Now I'm just confused. Distutils installs headers, but not libraries, because a C module developer may be making a module that requires another module. But, if he's only got the headers of the required module, what does he link against? If I follow what you're saying, you're suggesting that somehow this occurs when python loads both modules. I don't think so. If the required module has it's own headers, it follows that it must also have it's own library which actually implements the functions defined in the headers. Python's really good, but it can't resolve symbols that don't exist, so it needs both libraries to resolve the dependencies. So, it doesn't make sense to me to have headers without libraries.
The exceptions to this being the headers that provide the python extension api. I just double checked, and on my system, those and libpython are in the python-devel package. So even those are not required on the typical user machine. Why would they need headers for any other C mods they install?
That's why the "install" command can install C header files (using the "install_headers" command). We install .py and .so/.pyd files because they will be needed by Python programmers writing Python modules and applications; we install .h files because they might be needed by C programmers writing Python extensions. We doesn't install any C libraries, either shared or static, because I have not yet seen a need for them.
I'm only providing opinions here, not code, so I'm very easily out-voted, but I think distutils should do neither or both. I still feel that the libraries are better provided by a generic (non-python) run-time package, the headers (and usage docs) by a devel package, and the python implementation by distutils. My bigest concern is that if distutils starts doing the devel stuff, why shouldn't is do the run-time stuff .....It's just a matter of time before distutils becomes the universal build/install/package everything tool. I recommend you don't take that first step.
If you're going to do headers, though, then you need to do the libraries that implement them also. This would be a nice feature for developers who want to share C modules, but I tend to think they would just to tarballs or cvs.
Mark
On 15 September 2000, Mark W. Alexander said:
I'm only providing opinions here, not code, so I'm very easily out-voted, but I think distutils should do neither or both. I still feel that the libraries are better provided by a generic (non-python) run-time package, the headers (and usage docs) by a devel package, and the python implementation by distutils.
But the headers installed by the Distutils are *specifically* for Python extensions: we install Numeric/*.h so that other extension authors can use Numeric's C structs and functions, not so that general C programmers can do so. Thus, this is a concern for a Python extension-building and -installing system.
I do not know for certain, but I suspect that installing the Python extension -- eg. _numpy.so -- suffices for the "library" portion. I have not heard any reaction, positive or negative, from anyone using the installed Numeric/*.h files. So who knows if it's actually working?
.....It's just a matter of time before distutils becomes the universal build/install/package everything tool.
Curses! You've seen to the heart of my devious master plan. All who read this post will have to swear to uttermost silence, else my minions will track them down and... SILENCE THEM!! bwa-hAHAHAHAHAA!!!
I recommend you don't take that first step.
Oops, too late. ;-)
Greg
-
Andrew Kuchling -
Greg Ward -
Mark W. Alexander