Large data arrays?

[John Machin]

On Apr 24, 5:17 pm, Ole Streicher <ole-usenet-s... at gmx.net> wrote:
> Hi John,
>
> John Machin <sjmac... at lexicon.net> writes:
> > The Morton layout wastes space if the matrix is not square. Your 100K
> > x 4K is very non-square. Looks like you might want to use e.g. 25
> > Morton arrays, each 4K x 4K.
>
> What I found was that Morton layout shall be usable, if the shape is
> rectangular and both dimensions are powers of two. But, all examples
> were done with equal dimensions, so I am a bit confused here.

Yes, you are right, it can be done in one hit with a rectangular
array. How it is done: in your case you have a 2**17 x 2**12 array, so
the Morton index corresponding to (i, j) would have the top 5 bits of
i followed by the remaining 12 bits of i interleaved with the 12 bits
of j -- scarcely distinguishable from my original suggestion of 25
4Kx4K arrays, once you've ignored the trailing approx 2**17 - 1000000
elements that you don't need to allocate space for ;-)

>
> From my access pattern, it would be probably better to combine 25 rows
> into one slice and have one matrix where every cell contains 25 rows.
>
> Are there any objections about that?

Can't object, because I'm not sure what you mean ... how many elements
in a "cell"?

Cheers,
John