Bug with how numpy.distutils.system_info handles the site.cfg
I had this problem back in 2009 when building Enthought Enable, and was happy with a work around. It just bit me again, and I finally got around to drilling down to the problem. On linux, if one uses the numpy/site.cfg [default] section when building from source to specifiy local library directories, the x11 libs won't be found by NumPy. The relevant section of the site.cfg.example reads as follows: # Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # #[DEFAULT] #library_dirs = /usr/local/lib #include_dirs = /usr/local/include Now, I build NumPy with Atlas and my Atlas libs are installed in /usr/local, so my [default] section of site.cfg looks like this (as suggested by the site.cfg.example): # Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # [DEFAULT] library_dirs = /usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/local/include NumPy builds and works fine with this. The problem occurs when other libraries use numpy.distutils.system_info.get_info('x11') (ala Enthought Enable). That function eventually calls numpy.distutils.system_info.system_info.parse_config_files which has the following definition: def parse_config_files(self): self.cp.read(self.files) if not self.cp.has_section(self.section): if self.section is not None: self.cp.add_section(self.section) When self.cp is instantiated (when looking for the x11 libs), it is provided the following defaults: {'libraries': '', 'src_dirs': '.:/usr/local/src', 'search_static_first': '0', 'library_dirs': '/usr/X11R6/lib64:/usr/X11R6/lib:/usr/X11/lib64:/usr/X11/lib:/usr/lib64:/usr/lib', 'include_dirs': '/usr/X11R6/include:/usr/X11/include:/usr/include'} As is clearly seen, the 'library_dirs' contains the proper paths to find the x11 libs. But since the config file has [default] section, these paths get trampled and replaced with whatever is contained in the site.cfg [default] section. In my case, this is /usr/local/lib:/usr/local/lib/atlas. Thus, my x11 libs aren't found and the Enable build fails. The workaround is to include an [x11] section in site.cfg with the appropriate paths, but I don't really feel this should be necessary. Would the better behavior be to look for a [default] section in the config file in the parse_config_files method and add those paths to the already specified defaults? Changing the site.cfg [default] section to read as follows: [DEFAULT] library_dirs = /usr/lib:/usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/include:/usr/local/include is not an option because then NumPy will find and use the system atlas, which in my case is not threaded nor optimized for my machine. If you want me to patch the parse_config_files method, just let me know. Cheers, Chris
On Wed, May 12, 2010 at 11:06 PM, Chris Colbert
I had this problem back in 2009 when building Enthought Enable, and was happy with a work around. It just bit me again, and I finally got around to drilling down to the problem.
On linux, if one uses the numpy/site.cfg [default] section when building from source to specifiy local library directories, the x11 libs won't be found by NumPy.
The relevant section of the site.cfg.example reads as follows:
# Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # #[DEFAULT] #library_dirs = /usr/local/lib #include_dirs = /usr/local/include
Now, I build NumPy with Atlas and my Atlas libs are installed in /usr/local, so my [default] section of site.cfg looks like this (as suggested by the site.cfg.example):
# Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # [DEFAULT] library_dirs = /usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/local/include
NumPy builds and works fine with this. The problem occurs when other libraries use numpy.distutils.system_info.get_info('x11') (ala Enthought Enable). That function eventually calls numpy.distutils.system_info.system_info.parse_config_files which has the following definition:
def parse_config_files(self): self.cp.read(self.files) if not self.cp.has_section(self.section): if self.section is not None: self.cp.add_section(self.section)
When self.cp is instantiated (when looking for the x11 libs), it is provided the following defaults:
{'libraries': '', 'src_dirs': '.:/usr/local/src', 'search_static_first': '0', 'library_dirs': '/usr/X11R6/lib64:/usr/X11R6/lib:/usr/X11/lib64:/usr/X11/lib:/usr/lib64:/usr/lib', 'include_dirs': '/usr/X11R6/include:/usr/X11/include:/usr/include'}
As is clearly seen, the 'library_dirs' contains the proper paths to find the x11 libs. But since the config file has [default] section, these paths get trampled and replaced with whatever is contained in the site.cfg [default] section. In my case, this is /usr/local/lib:/usr/local/lib/atlas. Thus, my x11 libs aren't found and the Enable build fails.
The workaround is to include an [x11] section in site.cfg with the appropriate paths, but I don't really feel this should be necessary. Would the better behavior be to look for a [default] section in the config file in the parse_config_files method and add those paths to the already specified defaults?
Then again, another workaround could be to add the atlas directory paths to the [blas_opt] and [lapack_opt] sections. This would work for my case, but it doesn't solve the larger problem of any directories put in [default] trouncing any of the other standard dirs that would otherwise be used.
Changing the site.cfg [default] section to read as follows:
[DEFAULT] library_dirs = /usr/lib:/usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/include:/usr/local/include
is not an option because then NumPy will find and use the system atlas, which in my case is not threaded nor optimized for my machine.
If you want me to patch the parse_config_files method, just let me know.
Cheers,
Chris
On Wed, 2010-05-12 at 23:06 -0400, Chris Colbert wrote:
I had this problem back in 2009 when building Enthought Enable, and was happy with a work around. It just bit me again, and I finally got around to drilling down to the problem.
On linux, if one uses the numpy/site.cfg [default] section when building from source to specifiy local library directories, the x11 libs won't be found by NumPy.
The relevant section of the site.cfg.example reads as follows:
# Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # #[DEFAULT] #library_dirs = /usr/local/lib #include_dirs = /usr/local/include
Now, I build NumPy with Atlas and my Atlas libs are installed in /usr/local, so my [default] section of site.cfg looks like this (as suggested by the site.cfg.example):
# Defaults # ======== # The settings given here will apply to all other sections if not overridden. # This is a good place to add general library and include directories like # /usr/local/{lib,include} # [DEFAULT] library_dirs = /usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/local/include
NumPy builds and works fine with this. The problem occurs when other libraries use numpy.distutils.system_info.get_info('x11') (ala Enthought Enable). That function eventually calls numpy.distutils.system_info.system_info.parse_config_files which has the following definition:
def parse_config_files(self): self.cp.read(self.files) if not self.cp.has_section(self.section): if self.section is not None: self.cp.add_section(self.section)
When self.cp is instantiated (when looking for the x11 libs), it is provided the following defaults:
{'libraries': '', 'src_dirs': '.:/usr/local/src', 'search_static_first': '0', 'library_dirs': '/usr/X11R6/lib64:/usr/X11R6/lib:/usr/X11/lib64:/usr/X11/lib:/usr/lib64:/usr/lib', 'include_dirs': '/usr/X11R6/include:/usr/X11/include:/usr/include'}
As is clearly seen, the 'library_dirs' contains the proper paths to find the x11 libs. But since the config file has [default] section, these paths get trampled and replaced with whatever is contained in the site.cfg [default] section. In my case, this is /usr/local/lib:/usr/local/lib/atlas. Thus, my x11 libs aren't found and the Enable build fails.
The workaround is to include an [x11] section in site.cfg with the appropriate paths, but I don't really feel this should be necessary. Would the better behavior be to look for a [default] section in the config file in the parse_config_files method and add those paths to the already specified defaults?
Changing the site.cfg [default] section to read as follows:
[DEFAULT] library_dirs = /usr/lib:/usr/local/lib:/usr/local/lib/atlas include_dirs = /usr/include:/usr/local/include
is not an option because then NumPy will find and use the system atlas, which in my case is not threaded nor optimized for my machine.
If you want me to patch the parse_config_files method, just let me know.
Cheers,
Chris
Anyone have thoughts on this? Thinking more about it, I feel the appropriate behavior would be for numpy to prepend everything in the [default] section to its internal default paths, rather than override the internal defaults as it is currently doing.
participants (2)
-
Chris Colbert -
S. Chris Colbert