Mailman 3 Re: [Numpy-discussion] glibc memory corruption when running numpy.test() - NumPy-Discussion

Re: [Numpy-discussion] glibc memory corruption when running numpy.test()

Hoyt Koepke

20 Nov 2008 20 Nov '08

8:26 a.m.

Hi, Sorry; my first message wasn't under 40 KB with the attachments, so here's the same message but with the log files at http://www.stat.washington.edu/~hoytak/logs.tar.bz2.

...

Which ones ?

Sorry; ATLAS = 3.9.4 and lapack=3.2. I'll give 3.8.2 a shot per your advice.

...

You should not do that, it won't work as you would expect. It is a good rule to assume that you should never set the *FLAGS variable unless you really know what you are doing.

Fair enough. In my case I was having some issues with 32 bit and 64 bit mismatches (I think that fftw defaulted to 32 bit), so I set the flags variables. I also wanted to get the extra few percent of performance by using the tuning flags. I'll back up a bit now before playing with them now, though.

...

First, can you try without any blas/lapack (Do BLAS=None LAPACK=None ATLAS=None python setup.py ....) ?

This now works in the sense that it doesn't hang. I still get a number of test failures, however (build + test logs attached). Thanks a lot for the help! --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak@gmail.com ++++++++++++++++++++++++++++++++++++++++++

Show replies by date

David Cournapeau

20 Nov 20 Nov

8:45 a.m.

New subject: glibc memory corruption when running numpy.test()

On Thu, 2008-11-20 at 00:26 -0800, Hoyt Koepke wrote:

...

Hi,

Sorry; my first message wasn't under 40 KB with the attachments, so here's the same message but with the log files at http://www.stat.washington.edu/~hoytak/logs.tar.bz2.

...
Which ones ?

Sorry; ATLAS = 3.9.4 and lapack=3.2. I'll give 3.8.2 a shot per your advice.

Sorry, I meant which problems did you get when trying to build numpy with those ? Lapack 3.2 is really recent, and seems to use a new BLAS, which is likely not supported by ATLAS. But to be faire, that won't explain most failures you get.

...

...
You should not do that, it won't work as you would expect. It is a good rule to assume that you should never set the *FLAGS variable unless you really know what you are doing.

Fair enough. In my case I was having some issues with 32 bit and 64 bit mismatches (I think that fftw defaulted to 32 bit), so I set the flags variables. I also wanted to get the extra few percent of performance by using the tuning flags. I'll back up a bit now before playing with them now, though.

I honestly don't think those flags matter much in the case of numpy/scipy. In particular, using SSE and co automatically is simply impossible in numpy case, since the C code is very generic (non-aligned - non contiguous items) and the compiler has no way to know at compile time which cases are contiguous. FFTW support has been removed in recent scipy, so this won't be a problem anymore :)

...

This now works in the sense that it doesn't hang. I still get a number of test failures, however (build + test logs attached).

Those errors seem link to the flags you have been using. Some errors are really strange (4 vs 8 bytes types), but I don't see how it could be explained by a mismatch of 32 vs 64 bits machine code (to the best of my knowledge, you can't mix 32 and 64 bits machine code in one binary). Maybe a compiler bug when using -march flag. Please try building numpy wo BLAS/LAPACK and wo compiler flags first, to test that the bare configuration does work, and that the problems are not due to some bugs in your toolchain/OS/etc... The test suite should run without any failure in this case; then, we can work on the BLAS/LAPACK thing, cheers, David

Hoyt Koepke

9:14 a.m.

New subject: glibc memory corruption when running numpy.test()

Hi,

...

I honestly don't think those flags matter much in the case of numpy/scipy. In particular, using SSE and co automatically is simply impossible in numpy case, since the C code is very generic (non-aligned - non contiguous items) and the compiler has no way to know at compile time which cases are contiguous.

Good to know. I'll try to surpress my desire to optimize and not care about them :-).

...

Those errors seem link to the flags you have been using. Some errors are really strange (4 vs 8 bytes types), but I don't see how it could be explained by a mismatch of 32 vs 64 bits machine code (to the best of my knowledge, you can't mix 32 and 64 bits machine code in one binary). Maybe a compiler bug when using -march flag.

Please try building numpy wo BLAS/LAPACK and wo compiler flags first, to test that the bare configuration does work, and that the problems are not due to some bugs in your toolchain/OS/etc... The test suite should run without any failure in this case; then, we can work on the BLAS/LAPACK thing,

I believe the logs I attached (or rather linked to) don't involve atlas or lapack or any compiler flags. I agree that they are strange and I may have something weird floating around. It's getting late here, so I'll double check everything in the morning and may try to run gcc's test suite to verify that isn't the problem. Thanks again! --Hoyt

...

cheers,

David

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

-- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak@gmail.com ++++++++++++++++++++++++++++++++++++++++++

David Cournapeau

9:29 a.m.

New subject: glibc memory corruption when running numpy.test()

On Thu, Nov 20, 2008 at 6:14 PM, Hoyt Koepke wrote:

...

I believe the logs I attached (or rather linked to) don't involve atlas or lapack or any compiler flags.

Ah, yes, sorry, I missed the build.log one. The only thing which surprises me a bit is the size of long double (I have never seen it to be 16 bytes on Linux, but in theory, it should not matter as long as the detected size is correct; I don't have a 64 bits in handy ATM, will check at home). I must say I don't have any more ideas on what could cause this mess. Did you clean the install directory and the build directory before building ? David

Charles R Harris

9:40 a.m.

New subject: glibc memory corruption when running numpy.test()

On Thu, Nov 20, 2008 at 2:29 AM, David Cournapeau wrote:

...

On Thu, Nov 20, 2008 at 6:14 PM, Hoyt Koepke wrote:

...
I believe the logs I attached (or rather linked to) don't involve atlas or lapack or any compiler flags.

Ah, yes, sorry, I missed the build.log one. The only thing which surprises me a bit is the size of long double (I have never seen it to be 16 bytes on Linux, but in theory, it should not matter as long as

I believe that's normal on 64 bit machines - long doubles are filled out in the natural word size. Thus 80 goes to 64 + 32 on 32 bit machines, 64 + 64 on 64 bit machines. Chuck

5633

Age (days ago)

5633

Last active (days ago)

List overview

Download

4 comments

4 participants

participants (4)

Charles R Harris
David Cournapeau
David Cournapeau
Hoyt Koepke

Re: [Numpy-discussion] glibc memory corruption when running numpy.test()

Hoyt Koepke

David Cournapeau

Hoyt Koepke

David Cournapeau

Charles R Harris

tags

participants (4)