Question about flags of fancy indexed array
Hello all Consider the following example: In [43]: x = N.zeros((3,2)) In [44]: x.flags Out[44]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [45]: x[:,[1,0]].flags Out[45]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy indexed x? I'm running NumPy 1.0.3.dev3792 here. Cheers, Albert
On 23/05/07, Albert Strasheim <fullung@gmail.com> wrote:
Consider the following example:
First a comment: almost nobody needs to care how the data is stored internally. Try to avoid looking at the flags unless you're interfacing with a C library. The nice feature of numpy is that it hides all that junk - strides, contiguous storage, iteration, what have you - so that you don't have to deal with it.
Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy indexed x? I'm running NumPy 1.0.3.dev3792 here.
Numpy arrays are always stored in contiguous blocks of memory with uniform strides. The "CONTIGUOUS" flag actually means something totally different, which is unfortunate, but in any case, "fancy indexing" can't be done as a simple reindexing operation. It must make a copy of the array. So what you're seeing is the flags of a fresh new array, created from scratch (and numpy always creates arrays in C order internally, though that is an implementation detail you should not rely on). Anne
Hello all On Wed, 23 May 2007, Anne Archibald wrote:
On 23/05/07, Albert Strasheim <fullung@gmail.com> wrote:
Consider the following example:
First a comment: almost nobody needs to care how the data is stored internally. Try to avoid looking at the flags unless you're interfacing with a C library. The nice feature of numpy is that it hides all that junk - strides, contiguous storage, iteration, what have you - so that you don't have to deal with it.
As luck would have it, I am interfacing with a C library.
Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy indexed x? I'm running NumPy 1.0.3.dev3792 here.
Numpy arrays are always stored in contiguous blocks of memory with uniform strides. The "CONTIGUOUS" flag actually means something totally different, which is unfortunate, but in any case, "fancy indexing" can't be done as a simple reindexing operation. It must make a copy of the array. So what you're seeing is the flags of a fresh new array, created from scratch (and numpy always creates arrays in C order internally, though that is an implementation detail you should not rely on).
If you are correct that this is in fact a fresh new array, I really don't understand where the values of these flags. To recap: In [19]: x = N.zeros((3,2)) In [20]: x.flags Out[20]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [21]: x[:,[1,0]].flags Out[21]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True. Thanks. Cheers, Albert
On 23/05/07, Albert Strasheim <fullung@gmail.com> wrote:
If you are correct that this is in fact a fresh new array, I really don't understand where the values of these flags. To recap:
In [19]: x = N.zeros((3,2))
In [20]: x.flags Out[20]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
In [21]: x[:,[1,0]].flags Out[21]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.
It looks like x[:,[1,0]] is done by fancy indexing on the first index and then transposing. I haven't looked at the implementation, though. If you need the result to be C-contiguous without further copying, you can do: x.transpose()[[1,0],:].transpose().flags which is horrible. I wouldn't rely on it, though, I'd use asconguousarray afterward. Perhaps Travis could comment on whether it is true that numpy does transpose like this when fancy indexing (at least, this flavour of fancy indexing). Anne
On 5/23/07, Anne Archibald <peridot.faceted@gmail.com> wrote:
On 23/05/07, Albert Strasheim <fullung@gmail.com> wrote:
If you are correct that this is in fact a fresh new array, I really don't understand where the values of these flags. To recap:
In [19]: x = N.zeros((3,2))
In [20]: x.flags Out[20]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
In [21]: x[:,[1,0]].flags Out[21]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.
It looks like x[:,[1,0]] is done by fancy indexing on the first index and then transposing. I haven't looked at the implementation, though. If you need the result to be C-contiguous without further copying, you can do: x.transpose()[[1,0],:].transpose().flags which is horrible. I wouldn't rely on it, though, I'd use asconguousarray afterward.
Both copy and ascontiguousarray do the job: In [39]: ascontiguousarray(x[:,[1,0]]).flags Out[39]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [40]: x[:,[1,0]].copy().flags Out[40]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False The extra copy is unfortunate but I don't see how it is to be avoided. Maintaining order and contiguity in memory is not the goal of numpy, convenience and flexibility is. So interfacing with external C and FORTRAN is always going to need some care. Chuck
I'm not sure why the data ends up F_CONTIGUOUS, but it appears that you can get things to end up as C_CONTIGUOUS by transposing before the indexing and then transposing back. I don't think that this results in extra copies, but I'm not certain of that.
a = np.arange(6).reshape(3,2) a array([[0, 1], [2, 3], [4, 5]]) b1 = a[:,[1,0]] a.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False b1.flags C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False b2 = a.transpose()[[(1,0)]].transpose() b2.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False b1 == b2 array([[True, True], [True, True], [True, True]], dtype=bool)
-- //=][=\\ tim.hochberg@ieee.org
On 5/23/07, Albert Strasheim <fullung@gmail.com> wrote:
Hello all
On Wed, 23 May 2007, Anne Archibald wrote:
On 23/05/07, Albert Strasheim <fullung@gmail.com> wrote:
Consider the following example:
First a comment: almost nobody needs to care how the data is stored internally. Try to avoid looking at the flags unless you're interfacing with a C library. The nice feature of numpy is that it hides all that junk - strides, contiguous storage, iteration, what have you - so that you don't have to deal with it.
As luck would have it, I am interfacing with a C library.
Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy indexed x? I'm running NumPy 1.0.3.dev3792 here.
Numpy arrays are always stored in contiguous blocks of memory with uniform strides. The "CONTIGUOUS" flag actually means something totally different, which is unfortunate, but in any case, "fancy indexing" can't be done as a simple reindexing operation. It must make a copy of the array. So what you're seeing is the flags of a fresh new array, created from scratch (and numpy always creates arrays in C order internally, though that is an implementation detail you should not rely on).
If you are correct that this is in fact a fresh new array, I really don't understand where the values of these flags. To recap:
In [19]: x = N.zeros((3,2))
In [20]: x.flags Out[20]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
In [21]: x[:,[1,0]].flags Out[21]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.
The contiguous refers to how stuff is layed out in memory. In this case it appears that fancy indexing creates the new array by first copying column 1, then column 2, so that the new array is indeed F_CONTIGUOUS. Assuming I correctly understand the behaviour of the tostring argument, which is debatable, that is indeed what happens. In [28]: x = arange(6, dtype=int8).reshape(3,2) In [29]: x Out[29]: array([[0, 1], [2, 3], [4, 5]], dtype=int8) In [30]: y = x[:,[1,0]] In [31]: y Out[31]: array([[1, 0], [3, 2], [5, 4]], dtype=int8) In [32]: x.tostring('A') Out[32]: '\x00\x01\x02\x03\x04\x05' In [33]: y.tostring('A') Out[33]: '\x01\x03\x05\x00\x02\x04' Chuck
On 5/23/07, Charles R Harris <charlesr.harris@gmail.com> wrote:
On 5/23/07, Albert Strasheim <fullung@gmail.com> wrote:
Hello all
On Wed, 23 May 2007, Anne Archibald wrote:
On 23/05/07, Albert Strasheim <fullung@gmail.com> wrote:
Consider the following example:
First a comment: almost nobody needs to care how the data is stored internally. Try to avoid looking at the flags unless you're interfacing with a C library. The nice feature of numpy is that it hides all that junk - strides, contiguous storage, iteration, what have you - so that you don't have to deal with it.
As luck would have it, I am interfacing with a C library.
Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy indexed x? I'm running NumPy 1.0.3.dev3792 here.
Numpy arrays are always stored in contiguous blocks of memory with uniform strides. The "CONTIGUOUS" flag actually means something totally different, which is unfortunate, but in any case, "fancy indexing" can't be done as a simple reindexing operation. It must make a copy of the array. So what you're seeing is the flags of a fresh new
array, created from scratch (and numpy always creates arrays in C order internally, though that is an implementation detail you should not rely on).
If you are correct that this is in fact a fresh new array, I really don't understand where the values of these flags. To recap:
In [19]: x = N.zeros((3,2))
In [20]: x.flags Out[20]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
In [21]: x[:,[1,0]].flags Out[21]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.
The contiguous refers to how stuff is layed out in memory. In this case it appears that fancy indexing creates the new array by first copying column 1, then column 2, so that the new array is indeed F_CONTIGUOUS. Assuming I correctly understand the behaviour of the tostring argument, which is debatable, that is indeed what happens.
Make that The contiguous flags refer to how stuff is layed out in memory. In this case it appears that fancy indexing creates the new array by first copying column 1, then column 0, so that the new array is indeed F_CONTIGUOUS. Assuming I correctly understand the behaviour of the tostring argument, which is debatable, that is indeed what happens. Real programmers don't do writing. Chuck
Charles R Harris wrote:
On 5/23/07, *Albert Strasheim* <fullung@gmail.com <mailto:fullung@gmail.com>> wrote:
Hello all
On Wed, 23 May 2007, Anne Archibald wrote:
> On 23/05/07, Albert Strasheim <fullung@gmail.com <mailto:fullung@gmail.com>> wrote: > > > Consider the following example: > > First a comment: almost nobody needs to care how the data is stored > internally. Try to avoid looking at the flags unless you're > interfacing with a C library. The nice feature of numpy is that it > hides all that junk - strides, contiguous storage, iteration, what > have you - so that you don't have to deal with it.
As luck would have it, I am interfacing with a C library.
> > Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy > > indexed x? I'm running NumPy 1.0.3.dev3792 here. > > Numpy arrays are always stored in contiguous blocks of memory with > uniform strides. The "CONTIGUOUS" flag actually means something > totally different, which is unfortunate, but in any case, "fancy > indexing" can't be done as a simple reindexing operation. It must make > a copy of the array. So what you're seeing is the flags of a fresh new > array, created from scratch (and numpy always creates arrays in C > order internally, though that is an implementation detail you should > not rely on).
If you are correct that this is in fact a fresh new array, I really don't understand where the values of these flags. To recap:
In [19]: x = N.zeros((3,2))
In [20]: x.flags Out[20]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
In [21]: x[:,[1,0]].flags Out[21]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.
The contiguous refers to how stuff is layed out in memory. In this case it appears that fancy indexing creates the new array by first copying column 1, then column 2, so that the new array is indeed F_CONTIGUOUS. Assuming I correctly understand the behaviour of the tostring argument, which is debatable, that is indeed what happens.
Yes, you are using the 'A' argument correctly. For contiguous arrays, you can always get a look at the actual buffer, using x.data[:] or y.data[:] You can't get a buffer for discontiguous arrays. -Travis
On Wed, May 23, 2007 at 09:49:08AM -0400, Anne Archibald wrote:
On 23/05/07, Albert Strasheim <fullung@gmail.com> wrote:
Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy indexed x? I'm running NumPy 1.0.3.dev3792 here.
Numpy arrays are always stored in contiguous blocks of memory with uniform strides. The "CONTIGUOUS" flag actually means something totally different, which is unfortunate, but in any case, "fancy indexing" can't be done as a simple reindexing operation. It must make a copy of the array. So what you're seeing is the flags of a fresh new array, created from scratch (and numpy always creates arrays in C order internally, though that is an implementation detail you should not rely on).
That still doesn't explain In [41]: N.zeros((3,2))[:,[0,1]].flags Out[41]: C_CONTIGUOUS : False F_CONTIGUOUS : True <<<<<<<<<<< OWNDATA : False <<<<<<<<<<< WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False vs. In [40]: N.zeros((3,2),order='F')[:,[0,1]].flags Out[40]: C_CONTIGUOUS : True F_CONTIGUOUS : False <<<<<<<<<< OWNDATA : False <<<<<<<<<< WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False Maybe the Fortran-ordering quiz at http://mentat.za.net/numpy/quiz needs an update! :) Cheers Stéfan
Albert Strasheim wrote:
Hello all
Consider the following example:
In [43]: x = N.zeros((3,2))
In [44]: x.flags Out[44]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
In [45]: x[:,[1,0]].flags Out[45]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy indexed x? I'm running NumPy 1.0.3.dev3792 here.
In this case, yes. When you use fancy-indexing with standard slicing (an extension that NumPy added), the implementation uses transposes under the covers quite often. So, you can't rely on the output of fancy indexing being a C-contiguous array (even though it won't be referring to the same data as the original array). So, you are exposing an implementation detail here. To interface to code that requires C-contiguous or F-contiguous data, you have to check the flags and make an appropriate copy. -Travis
participants (6)
-
Albert Strasheim -
Anne Archibald -
Charles R Harris -
Stefan van der Walt -
Timothy Hochberg -
Travis Oliphant