large memory address space on Mac OS X (intel)
Dear list: I cannot seem to figure how to create arrays > 2 GB on a Mac Pro (using Intel chip and Tiger, 4.8). I have hand compiled both Python 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB. I also run out of space if I try and 3-6 several arrays of 1000 mb or so (the mem-alloc failure does not seem consistent, depends on whether I am creating them with a "numpy.ones()" call, or creating them on the fly by doing math with the other arrays "e.g., c = 4.3*a + 3.1*b"). Is this a numpy issue, or a Python 2.5 issue for the Mac? I have tried this on the SGI Altix, and this works fine. If there is a compile flag to turn on 64 bit support in the Mac compile, I would be glad to find out about it. Or do I have to wait for Leopard? Thanks. Lou Wicker ------------------------------------------------------------------------ ---- | Dr. Louis J. Wicker | NSSL/WRDD | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072-7323 | | E-mail: Louis.Wicker@noaa.gov | HTTP: www.nssl.noaa.gov/~lwicker | Phone: (405) 325-6340 | Fax: (405) 325-6780 | | "Programming is not just creating strings of instructions | for a computer to execute. It's also 'literary' in that you | are trying to communicate a program structure to | other humans reading the code." - Paul Rubin | |"Real efficiency comes from elegant solutions, not optimized programs. | Optimization is always just a few correctness-preserving transformations | away." - Jonathan Sobel ------------------------------------------------------------------------ ---- | | "The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." | ------------------------------------------------------------------------ ----
Louis Wicker wrote:
Dear list:
I cannot seem to figure how to create arrays > 2 GB on a Mac Pro (using Intel chip and Tiger, 4.8). I have hand compiled both Python 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB. I also run out of space if I try and 3-6 several arrays of 1000 mb or so (the mem-alloc failure does not seem consistent, depends on whether I am creating them with a "numpy.ones()" call, or creating them on the fly by doing math with the other arrays "e.g., c = 4.3*a + 3.1*b").
Is this a numpy issue, or a Python 2.5 issue for the Mac? I have tried this on the SGI Altix, and this works fine.
It must be a malloc issue. NumPy uses the system malloc to construct arrays. It just reports errors back to you if it can't. I don't think the Mac Pro uses a 64-bit chip, does it? -Travis
Travis Oliphant wrote:
Louis Wicker wrote:
Dear list:
I cannot seem to figure how to create arrays > 2 GB on a Mac Pro (using Intel chip and Tiger, 4.8). I have hand compiled both Python 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB. I also run out of space if I try and 3-6 several arrays of 1000 mb or so (the mem-alloc failure does not seem consistent, depends on whether I am creating them with a "numpy.ones()" call, or creating them on the fly by doing math with the other arrays "e.g., c = 4.3*a + 3.1*b").
Is this a numpy issue, or a Python 2.5 issue for the Mac? I have tried this on the SGI Altix, and this works fine.
It must be a malloc issue. NumPy uses the system malloc to construct arrays. It just reports errors back to you if it can't.
I don't think the Mac Pro uses a 64-bit chip, does it?
Intel Core 2 Duo's are 64-bit, but the OS and runtime libraries are not. None of the Python distributions for OS X are compiled for 64-bit support, either. When Leopard comes out, there will be better 64-bit support in the OS, and Python will follow. Until then, Python on OS X is 32-bit. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Travis: yes it does. Its the Woodcrest server chip which supports 32 and 64 bit operations. For example the new Intel Fortran compiler can grab more than 2 GB of memory (its a beta10 version). I think gcc 4.x can as well. However, Tiger (OS X 10.4.x) is not completely 64 bit compliant - Leopard is supposed to be pretty darn close. Is there a numpy flag I could try for compilation.... Lou On Feb 1, 2007, at 1:41 PM, Travis Oliphant wrote:
Louis Wicker wrote:
Dear list:
I cannot seem to figure how to create arrays > 2 GB on a Mac Pro (using Intel chip and Tiger, 4.8). I have hand compiled both Python 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB. I also run out of space if I try and 3-6 several arrays of 1000 mb or so (the mem-alloc failure does not seem consistent, depends on whether I am creating them with a "numpy.ones()" call, or creating them on the fly by doing math with the other arrays "e.g., c = 4.3*a + 3.1*b").
Is this a numpy issue, or a Python 2.5 issue for the Mac? I have tried this on the SGI Altix, and this works fine.
It must be a malloc issue. NumPy uses the system malloc to construct arrays. It just reports errors back to you if it can't.
I don't think the Mac Pro uses a 64-bit chip, does it?
-Travis
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
------------------------------------------------------------------------ ---- | Dr. Louis J. Wicker | NSSL/WRDD | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072-7323 | | E-mail: Louis.Wicker@noaa.gov | HTTP: www.nssl.noaa.gov/~lwicker | Phone: (405) 325-6340 | Fax: (405) 325-6780 | | "Programming is not just creating strings of instructions | for a computer to execute. It's also 'literary' in that you | are trying to communicate a program structure to | other humans reading the code." - Paul Rubin | |"Real efficiency comes from elegant solutions, not optimized programs. | Optimization is always just a few correctness-preserving transformations | away." - Jonathan Sobel ------------------------------------------------------------------------ ---- | | "The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." | ------------------------------------------------------------------------ ----
Travis: quick follow up: Mac Pro's currently have the dual-core 5100 Xeon (two processors, two cores each), the 5300 Xeon's (quad-core) are coming in a few weeks, we think. Lou On Feb 1, 2007, at 1:41 PM, Travis Oliphant wrote:
Louis Wicker wrote:
Dear list:
I cannot seem to figure how to create arrays > 2 GB on a Mac Pro (using Intel chip and Tiger, 4.8). I have hand compiled both Python 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB. I also run out of space if I try and 3-6 several arrays of 1000 mb or so (the mem-alloc failure does not seem consistent, depends on whether I am creating them with a "numpy.ones()" call, or creating them on the fly by doing math with the other arrays "e.g., c = 4.3*a + 3.1*b").
Is this a numpy issue, or a Python 2.5 issue for the Mac? I have tried this on the SGI Altix, and this works fine.
It must be a malloc issue. NumPy uses the system malloc to construct arrays. It just reports errors back to you if it can't.
I don't think the Mac Pro uses a 64-bit chip, does it?
-Travis
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
------------------------------------------------------------------------ ---- | Dr. Louis J. Wicker | NSSL/WRDD | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072-7323 | | E-mail: Louis.Wicker@noaa.gov | HTTP: www.nssl.noaa.gov/~lwicker | Phone: (405) 325-6340 | Fax: (405) 325-6780 | | "Programming is not just creating strings of instructions | for a computer to execute. It's also 'literary' in that you | are trying to communicate a program structure to | other humans reading the code." - Paul Rubin | |"Real efficiency comes from elegant solutions, not optimized programs. | Optimization is always just a few correctness-preserving transformations | away." - Jonathan Sobel ------------------------------------------------------------------------ ---- | | "The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." | ------------------------------------------------------------------------ ----
Thanks Robert thats kinda what I thought. Since Leopard is not far off, then by summer things will be fine, I hope... L On Feb 1, 2007, at 1:48 PM, Robert Kern wrote:
Travis Oliphant wrote:
Louis Wicker wrote:
Dear list:
I cannot seem to figure how to create arrays > 2 GB on a Mac Pro (using Intel chip and Tiger, 4.8). I have hand compiled both Python 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB. I also run out of space if I try and 3-6 several arrays of 1000 mb or so (the mem-alloc failure does not seem consistent, depends on whether I am creating them with a "numpy.ones()" call, or creating them on the fly by doing math with the other arrays "e.g., c = 4.3*a + 3.1*b").
Is this a numpy issue, or a Python 2.5 issue for the Mac? I have tried this on the SGI Altix, and this works fine.
It must be a malloc issue. NumPy uses the system malloc to construct arrays. It just reports errors back to you if it can't.
I don't think the Mac Pro uses a 64-bit chip, does it?
Intel Core 2 Duo's are 64-bit, but the OS and runtime libraries are not. None of the Python distributions for OS X are compiled for 64-bit support, either.
When Leopard comes out, there will be better 64-bit support in the OS, and Python will follow. Until then, Python on OS X is 32-bit.
-- Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
------------------------------------------------------------------------ ---- | Dr. Louis J. Wicker | NSSL/WRDD | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072-7323 | | E-mail: Louis.Wicker@noaa.gov | HTTP: www.nssl.noaa.gov/~lwicker | Phone: (405) 325-6340 | Fax: (405) 325-6780 | | "Programming is not just creating strings of instructions | for a computer to execute. It's also 'literary' in that you | are trying to communicate a program structure to | other humans reading the code." - Paul Rubin | |"Real efficiency comes from elegant solutions, not optimized programs. | Optimization is always just a few correctness-preserving transformations | away." - Jonathan Sobel ------------------------------------------------------------------------ ---- | | "The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." | ------------------------------------------------------------------------ ----
Louis Wicker wrote:
Travis:
yes it does. Its the Woodcrest server chip http://www.intel.com/business/xeon/?cid=cim:ggl%7Cxeon_us_woodcrest%7Ck6913%... which supports 32 and 64 bit operations. For example the new Intel Fortran compiler can grab more than 2 GB of memory (its a beta10 version). I think gcc 4.x can as well.
Nice. I didn't know this.
However, Tiger (OS X 10.4.x) is not completely 64 bit compliant - Leopard is supposed to be pretty darn close.
Is there a numpy flag I could try for compilation....
It's entirely compiler and system dependent. NumPy just uses the system malloc. If you can compile it so that the system malloc supports 64-bit then O.K. (but you will probably run into trouble unless Python is also compiled as a 64-bit application). From Robert's answer, I guess it is impossible under Tiger to compile with 64-bit support. -Travis
Here is a small c program that we used more than a year ago to confirm
that tiger is really doing a 64-bit malloc (on G5).
Louis Wicker wrote:
Travis:
yes it does. Its the Woodcrest server chip http://www.intel.com/business/xeon/?cid=cim:ggl%7Cxeon_us_woodcrest%7Ck6913%... which supports 32 and 64 bit operations. For example the new Intel Fortran compiler can grab more than 2 GB of memory (its a beta10 version). I think gcc 4.x can as well.
Nice. I didn't know this.
However, Tiger (OS X 10.4.x) is not completely 64 bit compliant - Leopard is supposed to be pretty darn close.
Is there a numpy flag I could try for compilation....
It's entirely compiler and system dependent. NumPy just uses the system malloc. If you can compile it so that the system malloc supports 64-bit then O.K. (but you will probably run into trouble unless Python is also compiled as a 64-bit application). From Robert's answer, I guess it is impossible under Tiger to compile with 64-bit support.
-Travis
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Here is a small c program that we used more than a year ago to confirm
that tiger is really doing a 64-bit malloc (on G5).
Louis Wicker wrote:
Travis:
yes it does. Its the Woodcrest server chip http://www.intel.com/business/xeon/?cid=cim:ggl%7Cxeon_us_woodcrest%7Ck6913%... which supports 32 and 64 bit operations. For example the new Intel Fortran compiler can grab more than 2 GB of memory (its a beta10 version). I think gcc 4.x can as well.
Nice. I didn't know this.
However, Tiger (OS X 10.4.x) is not completely 64 bit compliant - Leopard is supposed to be pretty darn close.
Is there a numpy flag I could try for compilation....
It's entirely compiler and system dependent. NumPy just uses the system malloc. If you can compile it so that the system malloc supports 64-bit then O.K. (but you will probably run into trouble unless Python is also compiled as a 64-bit application). From Robert's answer, I guess it is impossible under Tiger to compile with 64-bit support.
-Travis
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Sebastian: that code helps a lot. A standard gcc (no flags) of that code breaks, but if you compile it with gcc -m64, you can address large memory spaces. So I will try and compile numpy with -m64.... Lou On Feb 1, 2007, at 2:01 PM, Sebastian Haase wrote:
#include
#include int main() { size_t n; void *p; double gb; for(gb=10;gb>.3;gb-=.5) { n= 1024L * 1024L * 1024L * gb; p = malloc( n ); printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p); free(p); } return 0; }
------------------------------------------------------------------------ ---- | Dr. Louis J. Wicker | NSSL/WRDD | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072-7323 | | E-mail: Louis.Wicker@noaa.gov | HTTP: www.nssl.noaa.gov/~lwicker | Phone: (405) 325-6340 | Fax: (405) 325-6780 | | "Programming is not just creating strings of instructions | for a computer to execute. It's also 'literary' in that you | are trying to communicate a program structure to | other humans reading the code." - Paul Rubin | |"Real efficiency comes from elegant solutions, not optimized programs. | Optimization is always just a few correctness-preserving transformations | away." - Jonathan Sobel ------------------------------------------------------------------------ ---- | | "The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." | ------------------------------------------------------------------------ ----
Louis Wicker wrote:
Sebastian:
that code helps a lot. A standard gcc (no flags) of that code breaks, but if you compile it with gcc -m64, you can address large memory spaces.
So I will try and compile numpy with -m64....
It won't work. Your Python is not compiled as a 64-bit program. The whole stack down to the runtime libraries needs to be built as a 64-bit program. That's easy enough to do with a single-file C program, but in Tiger the 64-bit runtime provides very few services, not enough to build Python. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert: thanks - I appreciate the advice, and hopefully a) Leopard will get here in a few months, and b) that will fix this. cheers! Lou Wicker On Feb 1, 2007, at 3:11 PM, Robert Kern wrote:
Louis Wicker wrote:
Sebastian:
that code helps a lot. A standard gcc (no flags) of that code breaks, but if you compile it with gcc -m64, you can address large memory spaces.
So I will try and compile numpy with -m64....
It won't work. Your Python is not compiled as a 64-bit program. The whole stack down to the runtime libraries needs to be built as a 64-bit program. That's easy enough to do with a single-file C program, but in Tiger the 64-bit runtime provides very few services, not enough to build Python.
-- Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
------------------------------------------------------------------------ ---- | Dr. Louis J. Wicker | NSSL/WRDD | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072-7323 | | E-mail: Louis.Wicker@noaa.gov | HTTP: www.nssl.noaa.gov/~lwicker | Phone: (405) 325-6340 | Fax: (405) 325-6780 | | "Programming is not just creating strings of instructions | for a computer to execute. It's also 'literary' in that you | are trying to communicate a program structure to | other humans reading the code." - Paul Rubin | |"Real efficiency comes from elegant solutions, not optimized programs. | Optimization is always just a few correctness-preserving transformations | away." - Jonathan Sobel ------------------------------------------------------------------------ ---- | | "The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." | ------------------------------------------------------------------------ ----
On Thu, Feb 01, 2007 at 01:33:23PM -0600, Louis Wicker wrote:
Dear list: Hi,
may I suggest you to read this? http://orange.blender.org/blog/stupid-memory-problems It worth a read. David
I cannot seem to figure how to create arrays > 2 GB on a Mac Pro (using Intel chip and Tiger, 4.8). I have hand compiled both Python 2.5 and numpy 1.0.1, and cannot make arrays bigger than 2 GB. I also run out of space if I try and 3-6 several arrays of 1000 mb or so (the mem-alloc failure does not seem consistent, depends on whether I am creating them with a "numpy.ones()" call, or creating them on the fly by doing math with the other arrays "e.g., c = 4.3*a + 3.1*b").
Is this a numpy issue, or a Python 2.5 issue for the Mac? I have tried this on the SGI Altix, and this works fine.
If there is a compile flag to turn on 64 bit support in the Mac compile, I would be glad to find out about it. Or do I have to wait for Leopard?
Thanks.
Lou Wicker
------------------------------------------------------------------------ ---- | Dr. Louis J. Wicker | NSSL/WRDD | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072-7323 | | E-mail: Louis.Wicker@noaa.gov | HTTP: www.nssl.noaa.gov/~lwicker | Phone: (405) 325-6340 | Fax: (405) 325-6780 | | "Programming is not just creating strings of instructions | for a computer to execute. It's also 'literary' in that you | are trying to communicate a program structure to | other humans reading the code." - Paul Rubin | |"Real efficiency comes from elegant solutions, not optimized programs. | Optimization is always just a few correctness-preserving transformations | away." - Jonathan Sobel ------------------------------------------------------------------------ ---- | | "The contents of this message are mine personally and | do not reflect any position of the Government or NOAA." | ------------------------------------------------------------------------ ----
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
-- David Douard LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian : http://www.logilab.fr/formations Développement logiciel sur mesure : http://www.logilab.fr/services Informatique scientifique : http://www.logilab.fr/science
participants (6)
-
David Douard -
Louis Wicker -
Robert Kern -
Sebastian Haase -
Sebastian Haase -
Travis Oliphant