[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?

[Dan Yamins]

>
> Hey Dan.   Now, that you mention you are using OS X, I'm fairly
> confident that the problem is that you are using a 32-bit version of
> Python (i.e. you are not running in full 64-bit mode and so the 4GB
> limit applies).
>
> The most common Python on OS X is 32-bit python.  I think a few people
> in the SAGE project have successfully built Python in 64-bit mode on OSX
> (but I don't think they have released anything yet).  You would have to
> use a 64-bit version of Python to compile NumPy if you want to access
> large memory.
>
> -Travis
>
>
Travis, thanks for the message.   I think you're probably right -- I didn't
build python myself but instead downloaded the universal OSX binary from the
python download page -- and that surely wasn't built for 64-bit system.   So
I guess I'll have to figure out to do the 64-bit build.

Which leaves a question that I think Chuck brought up in a way:

> In [1]: s = numpy.random.binomial(1,.5,(20000,100))
> In [2]: inner(s,s)
> Out[2]:
> array([[45, 22, 17, ..., 20, 26, 23],
>      [22, 52, 26, ..., 23, 33, 24],
>       [17, 26, 52, ..., 27, 27, 19],
>       ...,
>       [20, 23, 27, ..., 46, 26, 22],
>       [26, 33, 27, ..., 26, 54, 25],
>       [23, 24, 19, ..., 22, 25, 44]])

>This on 32 bit fedora 8 with 2GiB of actual memory. It was slow and a
couple of hundred megs of something went into swap, but it >did complete. So
this looks to me like an OS X problem. Are there any limitations on the user
memory sizes? There might be some >system setting accounting for this.

Chuck, is this another way of asking:  why is my OS X system not paging
memory the way you'd expect a system to respond to the malloc command?    Is
python somehow overloaded the malloc command so that when the OS says a swap
would have to occur, somehow instead of just the swap, that an error message
involving mmap is somehow triggered?   (Sorry if this makes no sense.)   I
should add that the tried the same code on a 32-bit windows machine and got
the same error as on OS X.   Maybe the Linux python  builds manage this
stuff better.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080604/c46b5901/attachment.html>