Hi David,

...

...

Basically, I'm trying to understand the library discovery, linking steps - ATLAS in particular. Don't trust the included doc: it is not up-to-date, and that's the part which I totally redesigned since I wrote the initial doc.

...

- For a perflib check to succeed, I try to compile, link and run code snippets. Any failure means the Check* function will return failure. - When a perflib Check* succeeds, the meta library check checks that another code snippet can be compiled/linked/run.

Thanks - and I agree, for all the reasons you've given in later emails that this seems like the way to go.

But, in the mean time, I've got problems with the perflib stuff. Specifically, the scons build is not picking up the atlas library for blas and lapack that works for the standard build. I'm on a 64 bit system, providing an extra layer of complexity. I have never tested the thing on 64 bits (I don't have easy access to 64 bits machine, which should change soon, hopefully), so it is not surprising it does not work now. I have taken this into account for the design, though (see below).

I've attached the build logs. I noticed that, for atlas, you check for atlas_enum.c - but do you in fact need this for the build? Now. I just wanted one header specific to atlas. It looks like not all version of ATLAS install this one, unfortunately (3.8, for example). numpy.distutils seemed to be satisfied with cblas.h and clapack.h in /usr/local/include/atlas. It's no big deal to copy it from sources, but was there a reason you chose that file? No reason, but it cannot be cblas.h (it has to be atlas specific; otherwise, it does not make sense). The list of headers to check can be empty, though.

The test for linking to blas and lapack from atlas fails too - is this a false positive? Hmm, if atlas check does not work, it sounds like a right negative to me :) If ATLAS is not detected correctly, it won't be used by blas/lapack checkers. Or do you mean something else ?

For both numpy.distutils and numpyscons, default locations of libraries do not include the lib64 libraries like /usr/local/lib64 that us 64 bit people use. Is that easy to fix? Yes, it is easy, in the sense that nothing in the checkers code is harcoded: all the checks internally uses BuildConfig instances, which is

Matthew Brett wrote: like a dictionary with default values and a restricted set of keys (the keys are library path, libraries, headers, etc...). Those BuildConfig instances are created from a config file (perflib.cfg), and should always be customizable from site.cfg The options which can be customized can be found in the perflib.cfg file. For example, having: [atlas] htc = in your site.cfg should say to CheckATLAS to avoid looking for atlas_enum.h To make 64 bits work by default is a bit more complicated. I thought a bit about the problem: that's why the checkers do not use BuildConfig instances directly, but request them through a BuildConfigFactory. One problem is that I don't understand how 64 bits libraries work; more precisely, what is the convention of library path ? Is there a lib64 for each lib directory (/lib64, /usr/lib64, /usr/local/li64) ? Should the standard ones (/lib, /usr/lib) checked at all, after the 64 bits counterpart ? cheers, David

On Jan 25, 2008 3:21 AM, David Cournapeau wrote:

...

Hi David,

...

...

Basically, I'm trying to understand the library discovery, linking steps - ATLAS in particular. Don't trust the included doc: it is not up-to-date, and that's the part which I totally redesigned since I wrote the initial doc.

...

- For a perflib check to succeed, I try to compile, link and run code snippets. Any failure means the Check* function will return

failure.

...

- When a perflib Check* succeeds, the meta library check checks

...

...

another code snippet can be compiled/linked/run.

Thanks - and I agree, for all the reasons you've given in later emails that this seems like the way to go.

But, in the mean time, I've got problems with the perflib stuff. Specifically, the scons build is not picking up the atlas library for blas and lapack that works for the standard build. I'm on a 64 bit system, providing an extra layer of complexity. I have never tested the thing on 64 bits (I don't have easy access to 64 bits machine, which should change soon, hopefully), so it is not surprising it does not work now. I have taken this into account for the design, though (see below).

I've attached the build logs. I noticed that, for atlas, you check for atlas_enum.c - but do you in fact need this for the build? Now. I just wanted one header specific to atlas. It looks like not all version of ATLAS install this one, unfortunately (3.8, for example). numpy.distutils seemed to be satisfied with cblas.h and clapack.h in /usr/local/include/atlas. It's no big deal to copy it from sources, but was there a reason you chose that file? No reason, but it cannot be cblas.h (it has to be atlas specific; otherwise, it does not make sense). The list of headers to check can be empty, though.

The test for linking to blas and lapack from atlas fails too - is this a false positive? Hmm, if atlas check does not work, it sounds like a right negative to me :) If ATLAS is not detected correctly, it won't be used by blas/lapack checkers. Or do you mean something else ?

For both numpy.distutils and numpyscons, default locations of libraries do not include the lib64 libraries like /usr/local/lib64 that us 64 bit people use. Is that easy to fix? Yes, it is easy, in the sense that nothing in the checkers code is harcoded: all the checks internally uses BuildConfig instances, which is

Matthew Brett wrote: that like a dictionary with default values and a restricted set of keys (the keys are library path, libraries, headers, etc...). Those BuildConfig instances are created from a config file (perflib.cfg), and should always be customizable from site.cfg

The options which can be customized can be found in the perflib.cfg file. For example, having:

[atlas] htc =

in your site.cfg should say to CheckATLAS to avoid looking for atlas_enum.h

To make 64 bits work by default is a bit more complicated. I thought a bit about the problem: that's why the checkers do not use BuildConfig instances directly, but request them through a BuildConfigFactory. One problem is that I don't understand how 64 bits libraries work; more precisely, what is the convention of library path ? Is there a lib64 for each lib directory (/lib64, /usr/lib64, /usr/local/li64) ? Should the standard ones (/lib, /usr/lib) checked at all, after the 64 bits counterpart ?

It varies with the Linux distro. The usual convention (LSB, I think) uses /usr/local/lib64, but Debian and distros derived from Debian use /usr/local/lib instead. That's how it was the last time I checked, anyway. I don't know what Gentoo and all the others do. Chuck

On Jan 25, 2008 11:30 PM, David Cournapeau wrote:

Charles R Harris wrote:

<snip>

...

It varies with the Linux distro. The usual convention (LSB, I think) uses /usr/local/lib64, but Debian and distros derived from Debian use /usr/local/lib instead. That's how it was the last time I checked, anyway. I don't know what Gentoo and all the others do.

Grrr, that's annoying. Do you know any resource which has clear explanation on that ? (reading LSB documents do not appeal much to me, and I would only do that as a last resort).

The Debian folks will tell you that they did it right and everyone else screwed up horribly O_o. Anyway, it looks like they put the 32 bit stuff in lib32, so if a distro has a lib32, then you can figure the 64 bit stuff is in lib. Those are the only two variations I know of. What 64 bit windows does, I have no idea. 32-bit (compat) libraries 64-bit (native) libraries Fedora Core /lib/, /usr/lib/ /lib64/, /usr/lib64/ Debian /lib32/, /usr/lib32/ /lib/, /usr/lib/ Chuck

On Jan 26, 2008 1:05 AM, David Cournapeau wrote:

Charles R Harris wrote:

...

On Jan 25, 2008 11:30 PM, David Cournapeau mailto:david@ar.media.kyoto-u.ac.jp> wrote:

Charles R Harris wrote:

<snip>

> > It varies with the Linux distro. The usual convention (LSB, I

think)

...

> uses /usr/local/lib64, but Debian and distros derived from Debian use > /usr/local/lib instead. That's how it was the last time I checked, > anyway. I don't know what Gentoo and all the others do. Grrr, that's annoying. Do you know any resource which has clear explanation on that ? (reading LSB documents do not appeal much to

me,

...

and I would only do that as a last resort).

The Debian folks will tell you that they did it right and everyone else screwed up horribly O_o.

Well, as arrogant as debian folks may be, they are almost always right :) I will take a look at the debian maintainer guide, then.

...

Anyway, it looks like they put the 32 bit stuff in lib32, so if a distro has a lib32, then you can figure the 64 bit stuff is in lib. Those are the only two variations I know of. What 64 bit windows does, I have no idea. Windows and 64 bits, here is something which scares me. I also have to see how solaris does it. But anyway, since vmware can run 64 bits OS on my 32 bits linux, things should be easier.

ISTR that Solaris does it the Debian way. I wouldn't put money on it without checking, though. Chuck