PATCH: scipy_distutils/system_info.py
Ok, I've taken another look at the system_info.py code. The attached patch does the following: - The wrong things were being set for static and shared libraries. Static libraries should be linked as 'extra_objects', and shared libraries using 'libraries' and 'library_dirs'. - X11 is only linked statically if static_first=1 is set in the x11 section of the site.cfg file (i.e., only when you explicitly ask for it, which you probably won't) - The environment variables ATLAS and FFTW can now be set to a list of directories, if you don't want to use the site.cfg method. - The doc strings for {Atlas,FFTW}NotFoundError now mention site.cfg as a method of setting paths -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke |cookedm@mcmaster.ca
On Thu, 23 May 2002, David M. Cooke wrote:
Ok, I've taken another look at the system_info.py code. The attached patch does the following:
Thanks for the patch!
- The wrong things were being set for static and shared libraries. Static libraries should be linked as 'extra_objects', and shared libraries using 'libraries' and 'library_dirs'.
Are you sure about this? What is your situation that static libraries should not be in 'libraries'? I found this http://users.actcom.co.il/~choo/lupg/tutorials/libraries/unix-c-libraries.ht... very useful, e.g. see the last section. In fact, I am not sure anymore why we need to be specific about static libraries. Also, we shouldn't use 'extra_objects' at all. Pearu
At some point, Pearu Peterson <pearu@cens.ioc.ee> wrote:
On Thu, 23 May 2002, David M. Cooke wrote:
Ok, I've taken another look at the system_info.py code. The attached patch does the following:
Thanks for the patch!
- The wrong things were being set for static and shared libraries. Static libraries should be linked as 'extra_objects', and shared libraries using 'libraries' and 'library_dirs'.
Are you sure about this? What is your situation that static libraries should not be in 'libraries'?
This is what I get from system_info.py (before my patch) with static_first = 1: atlas_info: FOUND: include_dirs = ['/usr/local/include'] library_dirs = ['/usr/lib/3dnow/atlas', '/usr/lib/3dnow'] libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] fftw_info: FOUND: include_dirs = ['/usr/include'] define_macros = [('SCIPY_FFTW_H', 1)] library_dirs = ['/usr/lib'] extra_objects = ['/usr/lib/libsfftw.so.2', '/usr/lib/libsrfftw.so.2', '/usr/lib/libsfftw_threads.so.2', '/usr/lib/libsrfftw_threads.so.2'] libraries = ['fftw', 'rfftw', 'fftw_threads', 'rfftw_threads'] x11_info: FOUND: include_dirs = ['/usr/X11R6/include'] library_dirs = ['/usr/X11R6/lib'] libraries = ['X11'] This is what I get with static_first = 0: atlas_info: FOUND: include_dirs = ['/usr/local/include'] extra_objects = ['/usr/lib/3dnow/atlas/liblapack.so', '/usr/lib/3dnow/libf77blas.so', '/usr/lib/3dnow/libcblas.so', '/usr/lib/3dnow/libatlas.so'] fftw_info: FOUND: include_dirs = ['/usr/include'] extra_objects = ['/usr/lib/libfftw.so', '/usr/lib/librfftw.so', '/usr/lib/libfftw_threads.so', '/usr/lib/librfftw_threads.so', '/usr/lib/libsfftw.so.2', '/usr/lib/libsrfftw.so.2', '/usr/lib/libsfftw_threads.so.2', '/usr/lib/libsrfftw_threads.so.2'] define_macros = [('SCIPY_FFTW_H', 1)] x11_info: FOUND: include_dirs = ['/usr/X11R6/include'] extra_objects = ['/usr/X11R6/lib/libX11.so'] Note that the static_first=1 (which should be 'static libraries are preferred') would generate a link line like '-L/usr/lib/3dnow -L/usr/lib/3dnow/atlas -llapack -lf77blas -lcblas -latlas'. However, this will link in the *shared* versions of these libraries, as the linker, if there are static and shared libraries available that satisfy the link line, will preferentially select the shared library. The static_first=0 situation will have a link line like '/usr/lib/3dnow/atlas/liblapack.so /usr/lib/3dnow/libf77blas.so ...' which (I just tried it out) works for linking in shared libraries, but looks weird and isn't the way it's usually done. Now, with my patch, static_first=1 will use for the link line '/usr/lib/3dnow/atlas/liblapack.a /usr/lib/3dnow/libf77blas.a ...' which links in static versions, and static_first=0 will use '-L/usr/lib/3dnow/atlas -L/usr/lib/3dnow -llapack -lf77blas -lcblas -latlas' which will link in shared versions (or static versions, if the shared versions don't exist).
I found this
http://users.actcom.co.il/~choo/lupg/tutorials/libraries/unix-c-libraries.ht...
very useful, e.g. see the last section.
In fact, I am not sure anymore why we need to be specific about static libraries. Also, we shouldn't use 'extra_objects' at all.
If you want to use a static library instead of a shared one (for speed reasons, maybe), you will need to specify the filename of the static library as an 'extra object file', so it will have to go in 'extra_objects'. Either that or specify an option to the linker to prefer static to shared libraries. Be as it may, I prefer to use shared libraries, as only one copy of the library is in memory at one time, and my python modules don't suddenly become 5 megs in size... -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke |cookedm@mcmaster.ca
David, On Fri, 24 May 2002, David M. Cooke wrote: <snip>
Note that the static_first=1 (which should be 'static libraries are preferred') would generate a link line like '-L/usr/lib/3dnow -L/usr/lib/3dnow/atlas -llapack -lf77blas -lcblas -latlas'. However, this will link in the *shared* versions of these libraries, as the linker, if there are static and shared libraries available that satisfy the link line, will preferentially select the shared library.
The static_first=0 situation will have a link line like '/usr/lib/3dnow/atlas/liblapack.so /usr/lib/3dnow/libf77blas.so ...' which (I just tried it out) works for linking in shared libraries, but looks weird and isn't the way it's usually done.
I agree that static_first=1 did not work as expected but it worked as designed (see below).
Now, with my patch, static_first=1 will use for the link line '/usr/lib/3dnow/atlas/liblapack.a /usr/lib/3dnow/libf77blas.a ...' which links in static versions, and static_first=0 will use '-L/usr/lib/3dnow/atlas -L/usr/lib/3dnow -llapack -lf77blas -lcblas -latlas' which will link in shared versions (or static versions, if the shared versions don't exist).
Since I don't have shared versions of atlas libraries (in fact, ATLAS provides no support for that and users are in their own in building shared versions of atlas), I get with static_first=0 and your patch: atlas_info: FOUND: extra_objects = ['/usr/local/lib/atlas/liblapack.a', '/usr/local/lib/atlas/libf77blas.a', '/usr/local/lib/atlas/libcblas.a', '/usr/local/lib/atlas/libatlas.a'] include_dirs = ['/usr/include'] which is not desired either because blas and lapack extension modules will be very large, they will contain symbols that they do not need. Even if there are no .so libraries, the proper inclusion of .a libraries would be to use '-llapack -lf77blas -lcblas -latlas' so that only needed symbols get into extension modules.
If you want to use a static library instead of a shared one (for speed reasons, maybe), you will need to specify the filename of the static library as an 'extra object file', so it will have to go in 'extra_objects'. Either that or specify an option to the linker to prefer static to shared libraries.
Be as it may, I prefer to use shared libraries, as only one copy of the library is in memory at one time, and my python modules don't suddenly become 5 megs in size...
I agree with you completely. Initially static_first hooks were meant to be used only to affect the search path of libraries when static and shared libriaries are in different directories. Obviously this is not always the case and then the use of 'extra_objects' was introduced. IMO this was a mistake because 1) it is hackish and 2) it does not work neither with or without the patch. Note also that one cannot use linker flags to force using static libraries because python extension modules _must_ be shared. My suggestion is to stop using 'extra_objects' in static_first hooks and even if static_first=1 is specified, the shared libraries are preferred (by the system, not by scipy_distutils [*]) in the case .so and .a files are in the same directory. If people really want to force static libraries, they can collect (the links of) static libraries into a separate directory and add it to site.cfg. I don't think that scipy should do that for users that do not like default behaviour of their system. What do you think? [*] Fighting with the default behaviours of systems that cannot be tested directly is difficult. IMHO, by default scipy_distutils should use default settings of the system and let the users fight with the system if they insist, scipy_distutils can only provide some weapons (mainly site.cfg) to ease their task. Pearu
Hi, Regarding this patch I have made the following changes. On Thu, 23 May 2002, David M. Cooke wrote:
Ok, I've taken another look at the system_info.py code. The attached patch does the following:
- The wrong things were being set for static and shared libraries. Static libraries should be linked as 'extra_objects', and shared libraries using 'libraries' and 'library_dirs'.
Dropped using 'extra_objects' as it was buggy from the start and unified hooks for static (.a) and dynamic (.so) libriaries.
- X11 is only linked statically if static_first=1 is set in the x11 section of the site.cfg file (i.e., only when you explicitly ask for it, which you probably won't)
Renamed static_first to search_static_first and set its default value to 0. If someone really needs it, let me know. Otherwise search_static_first hooks will be removed in future. I find it quite useless because systems prefer shared libraries to static libraries (if they are in the same directory) and Python extension modules are always shared libraries anyway and I don't believe in speed improvement when linking against static libraries in this situation.
- The environment variables ATLAS and FFTW can now be set to a list of directories, if you don't want to use the site.cfg method.
Applied. In addition, one can use also LAPACK and BLAS environment variables.
- The doc strings for {Atlas,FFTW}NotFoundError now mention site.cfg as a method of setting paths
Applied. In addition, updated comments in sample_site.cfg. I'll look forward to scoreboard to see if my changes broke something, especially in win32 platform. Pearu
At some point, Pearu Peterson <pearu@cens.ioc.ee> wrote:
Hi,
Regarding this patch I have made the following changes.
[...]
Dropped using 'extra_objects' as it was buggy from the start and unified hooks for static (.a) and dynamic (.so) libriaries. [...] Renamed static_first to search_static_first and set its default value to 0. If someone really needs it, let me know. Otherwise search_static_first hooks will be removed in future. I find it quite useless because systems prefer shared libraries to static libraries (if they are in the same directory) and Python extension modules are always shared libraries anyway and I don't believe in speed improvement when linking against static libraries in this situation.
Ok, I looked over your changes and they look ok. But I have one quibble: the use of shortest_path(). If there are two copies of a library 'libx.so' in different directories, say /usr/local/lib and /opt/lib, and the search paths have been specified so that /usr/local/lib comes before /opt/lib (as it does in default_lib_dirs), then shortest_path() will select '/opt/lib/libx.so' instead of the preferred '/usr/local/lib/libx.so'. But here's a replacement that should choose the right one, preferring '/usr/local/lib/libx.so' over '/opt/lib/libx.so' and '/usr/local/lib/libx.so.6' def shortest_path(pths): if not pths: return None dir = os.path.dirname(pths[0]) pths = [(len(p), p) for p in pths if os.path.dirname(p) == dir] return min(pths)[1] -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke |cookedm@physics.mcmaster.ca
On Mon, 27 May 2002, David M. Cooke wrote:
At some point, Pearu Peterson <pearu@cens.ioc.ee> wrote:
Hi,
Regarding this patch I have made the following changes.
[...]
Dropped using 'extra_objects' as it was buggy from the start and unified hooks for static (.a) and dynamic (.so) libriaries. [...] Renamed static_first to search_static_first and set its default value to 0. If someone really needs it, let me know. Otherwise search_static_first hooks will be removed in future. I find it quite useless because systems prefer shared libraries to static libraries (if they are in the same directory) and Python extension modules are always shared libraries anyway and I don't believe in speed improvement when linking against static libraries in this situation.
Ok, I looked over your changes and they look ok. But I have one quibble: the use of shortest_path(). If there are two copies of a library 'libx.so' in different directories, say /usr/local/lib and /opt/lib,
Note that this situation is not possible because lib_dir in self._lib_list is always a string containing the name of a directory. So, shortest_path is used to return actually the shortest basename as the directory is always the same. To avoid breaking these conditions in future, I have added assert type(lib_dir) is type('') to self._lib_list method. Is that ok now? Pearu
At some point, Pearu Peterson <pearu@cens.ioc.ee> wrote:
On Mon, 27 May 2002, David M. Cooke wrote:
At some point, Pearu Peterson <pearu@cens.ioc.ee> wrote:
Hi,
Regarding this patch I have made the following changes.
[...]
Dropped using 'extra_objects' as it was buggy from the start and unified hooks for static (.a) and dynamic (.so) libriaries. [...] Renamed static_first to search_static_first and set its default value to 0. If someone really needs it, let me know. Otherwise search_static_first hooks will be removed in future. I find it quite useless because systems prefer shared libraries to static libraries (if they are in the same directory) and Python extension modules are always shared libraries anyway and I don't believe in speed improvement when linking against static libraries in this situation.
Ok, I looked over your changes and they look ok. But I have one quibble: the use of shortest_path(). If there are two copies of a library 'libx.so' in different directories, say /usr/local/lib and /opt/lib,
Note that this situation is not possible because lib_dir in self._lib_list is always a string containing the name of a directory. So, shortest_path is used to return actually the shortest basename as the directory is always the same. To avoid breaking these conditions in future, I have added assert type(lib_dir) is type('') to self._lib_list method. Is that ok now?
All right. It's just that it's somewhat deceptive to call shortest_path(), which (as written) doesn't take into account the preferred ordering. I suppose now I worried more about the look of the code :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke |cookedm@physics.mcmaster.ca
On Mon, 27 May 2002, Pearu Peterson wrote:
Note that this situation is not possible because lib_dir in self._lib_list is always a string containing the name of a directory. So, shortest_path is used to return actually the shortest basename as the directory is always the same. To avoid breaking these conditions in future, I have added assert type(lib_dir) is type('') to self._lib_list method. Is that ok now?
hmm, now I look that there is no need for shortest_path as e.g. p==[libx.so.6, libx.so] never occurs. Earlier it was possible because exts = ['.a',so_ext+'*'] equivalent was used. But now, detecting e.g. libx.so.6 is senseless as linking -lx looks always for libx.so. Earlier it made sense only because libx.so.6 was used in extra_objects list. So, I am considering not using the shortest_path at all and the code will hopefully look good for you :-) Pearu
participants (2)
-
cookedm＠physics.mcmaster.ca -
Pearu Peterson