NumPy on 64-bit Linux (Opteron)
Is anyone here using NumPy on Opteron machines running Linux in 64 bit mode? I am running into problems such as: Python 2.4 (#4, Jan 18 2005, 18:06:45) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from Numeric import * a = arange(4) a array([0, 1, 2, 3]) a.shape = (2, 2) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: total size of new array must be unchanged
This is with Python 2.4 and Numeric 23.7. Before looking into this myself, I thought I'd ask here, perhaps someone has already found a fix. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ------------------------------------------------------------------------ -------
On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen@laposte.net wrote:
Is anyone here using NumPy on Opteron machines running Linux in 64 bit mode? I am running into problems such as:
Python 2.4 (#4, Jan 18 2005, 18:06:45) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from Numeric import * a = arange(4) a array([0, 1, 2, 3]) a.shape = (2, 2) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: total size of new array must be unchanged
This is with Python 2.4 and Numeric 23.7. Before looking into this myself, I thought I'd ask here, perhaps someone has already found a fix.
I don't have a fix, but, for the record, I can reproduce this on my alphastation. a.typecode() says that the contents of a are Int64. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org
konrad.hinsen@laposte.net writes:
Is anyone here using NumPy on Opteron machines running Linux in 64 bit mode? I am running into problems such as:
Python 2.4 (#4, Jan 18 2005, 18:06:45) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from Numeric import * a = arange(4) a array([0, 1, 2, 3]) a.shape = (2, 2) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: total size of new array must be unchanged
This is with Python 2.4 and Numeric 23.7. Before looking into this myself, I thought I'd ask here, perhaps someone has already found a fix.
I do have currently access to an Opteron machine. OS is SuSE SLES9, with a lot of Python packages from SuSE Professional 9.1. I get: ~> python Python 2.3.3 (#1, Nov 29 2004, 22:35:24) [GCC 3.3.3 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import Numeric as N a = N.arange(4) a array([0, 1, 2, 3]) a.shape = (2,2) N.__file__ '/usr/lib64/python2.3/site-packages/Numeric/Numeric.pyc' N.__version__ '23.1'
Kind regards Berthold -- Germanischer Lloyd AG CAE Development Vorsetzen 35 20459 Hamburg Phone: +49(0)40 36149-7374 Fax: +49(0)40 36149-7320 e-mail: hoel@gl-group.com Internet: http://www.gl-group.com This e-mail contains confidential information for the exclusive attention of the intended addressee. Any access of third parties to this e-mail is unauthorised. Any use of this e-mail by unintended recipients such as copying, distribution, disclosure etc. is prohibited and may be unlawful. When addressed to our clients the content of this e-mail is subject to the General Terms and Conditions of GL's Group of Companies applicable at the date of this e-mail. GL's Group of Companies does not warrant and/or guarantee that this message at the moment of receipt is authentic, correct and its communication free of errors, interruption etc.
On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen@laposte.net wrote:
Is anyone here using NumPy on Opteron machines running Linux in 64 bit mode? I am running into problems such as:
Python 2.4 (#4, Jan 18 2005, 18:06:45) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from Numeric import * a = arange(4) a array([0, 1, 2, 3]) a.shape = (2, 2) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: total size of new array must be unchanged
This is with Python 2.4 and Numeric 23.7. Before looking into this myself, I thought I'd ask here, perhaps someone has already found a fix.
Yes; I submitted a patch (#1108739) to the Numeric patch tracker last week. Basically, PyArray_Reshape takes an array of dimensions. That has to be a PyArray_INT (C ints) instead of the PyArray_LONG (C longs) that it has in the code, because that array is passed to PyArray_FromDimsAndDataAndDescr. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
(RE: numarray) Hi, On an Opteron (Suse9) I was happy to see that I could allocate the memory for an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError'). I remember that Todd always said that python itself wasn't 64bit-ready yet. But why is it needed to support >4GB python-lists if one is only interested in large numarrays (or numeric) ? Just for sake of completeness here is one traceback I got:
import numarray a = numarray.array(shape=(1024,1024,1024), type=numarray.Int) a.type() Int32 a.shape (1024, 1024, 1024) len(a) 1024 a Traceback (most recent call last): File "<stdin>", line 1, in ? File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ numarraycore.py", line 896, in __repr__ return array_repr(self) File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ numarraycore.py", line 1542, in array_repr lst = arrayprint.array2string( File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ arrayprint.py", line 188, in array2string separator, prefix) File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ arrayprint.py", line 137, in _array2string data = _leading_trailing(a) File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ arrayprint.py", line 111, in _leading_trailing l = [_leading_trailing(a[i]) for i in range( File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ arrayprint.py", line 111, in _leading_trailing l = [_leading_trailing(a[i]) for i in range( File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ arrayprint.py", line 105, in _leading_trailing b = _gen.concatenate((a[:_summaryEdgeItems], File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ generic.py", line 1061, in concatenate return _concat(arrs) File "/home/haase/myCVS/numarray/build/lib.linux-x86_64-2.3/numarray/ generic.py", line 1051, in _concat dest[ix:ix+_shape0(a)]._copyFrom(a) numarray.libnumarray.error: copy4bytes: access beyond buffer. offset=11 buffersize=0
Thanks Sebastian Haase On Tuesday 08 February 2005 08:44 am, David M. Cooke wrote:
On Tue, Feb 08, 2005 at 12:44:18PM +0100, konrad.hinsen@laposte.net wrote:
Is anyone here using NumPy on Opteron machines running Linux in 64 bit mode? I am running into problems such as:
Python 2.4 (#4, Jan 18 2005, 18:06:45) [GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from Numeric import * a = arange(4) a
array([0, 1, 2, 3])
a.shape = (2, 2)
Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: total size of new array must be unchanged
This is with Python 2.4 and Numeric 23.7. Before looking into this myself, I thought I'd ask here, perhaps someone has already found a fix.
Yes; I submitted a patch (#1108739) to the Numeric patch tracker last week.
Basically, PyArray_Reshape takes an array of dimensions. That has to be a PyArray_INT (C ints) instead of the PyArray_LONG (C longs) that it has in the code, because that array is passed to PyArray_FromDimsAndDataAndDescr.
Sebastian Haase wrote:
(RE: numarray) Hi, On an Opteron (Suse9) I was happy to see that I could allocate the memory for an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError'). I remember that Todd always said that python itself wasn't 64bit-ready yet. But why is it needed to support >4GB python-lists if one is only interested in large numarrays (or numeric) ?
In short, to support the Python sequence protocol (arrays can be used anywhere Python code expects sequences transparently). The good news is that work is well underway in Python to change the limit on sequence indices. I think it will take a number of months for these changes to make it out (in a released version of Python as well as the necessary updates to numarray). As soon as the changes are committed to Python CVS, we can start working on updates to numarray to support them. Perry Greenfield
On Wednesday 09 February 2005 11:24 am, Perry Greenfield wrote:
Sebastian Haase wrote:
(RE: numarray) Hi, On an Opteron (Suse9) I was happy to see that I could allocate the memory for an Int32-1024x1024x1024 cube. (On a P4 a get 'MemoryError'). I remember that Todd always said that python itself wasn't 64bit-ready yet. But why is it needed to support >4GB python-lists if one is only interested in large numarrays (or numeric) ?
In short, to support the Python sequence protocol (arrays can be used anywhere Python code expects sequences transparently).
Perry, Thanks for the answer - that's about what I remembered. Is is conceivable to have a compiler flag (#define) to turn all the sequence protocol support off ? (How many places in the numarray code would that be?) I think the wish to just allocate large arrays and (even if in a very limited way!) work with them is clearly shared by many people. I always felt having a Py-List longer than billons of elements would have a rather low priority in comparison. Thanks again, Sebastian
The good news is that work is well underway in Python to change the limit on sequence indices. I think it will take a number of months for these changes to make it out (in a released version of Python as well as the necessary updates to numarray). As soon as the changes are committed to Python CVS, we can start working on updates to numarray to support them.
Perry Greenfield
Sebastian Haase wrote:
Perry, Thanks for the answer - that's about what I remembered. Is is conceivable to have a compiler flag (#define) to turn all the sequence protocol support off ? (How many places in the numarray code would that be?)
Offhand, I'm not sure (Todd probably has a much better quick answer for this question than I do). It is perhaps something that would allow us to test 64-bit arrays before the Python changes are ready if it isn't too hard to optionally disable the dependence on Python sequences. It's worth a quick look at after 1.2 is released.
I think the wish to just allocate large arrays and (even if in a very limited way!) work with them is clearly shared by many people. I always felt having a Py-List longer than billons of elements would have a rather low priority in comparison.
Just to note, we more and more local users who are begining to have a need for such large arrays too. Perry
On 08.02.2005, at 17:44, David M. Cooke wrote:
Yes; I submitted a patch (#1108739) to the Numeric patch tracker last week.
Thanks, that patch works fine for me as well. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ------------------------------------------------------------------------ -------
participants (6)
-
Alexandre -
Berthold Höllmann -
David M. Cooke -
konrad.hinsen＠laposte.net -
Perry Greenfield -
Sebastian Haase