[Numpy-discussion] Bytes vs. Unicode in Python3

[Pauli Virtanen]

Thu, 03 Dec 2009 14:03:13 +0100, Dag Sverre Seljebotn wrote:
[clip]
> Great! Are you storing the format string in the dtype types as well? (So
> that no release is needed and acquisitions are cheap...)

I regenerate it on each buffer acquisition. It's simple low-level C code, 
and I suspect it will always be fast enough. Of course, we could *cache* 
the result in the dtype. (If dtypes are immutable, which I don't remember 
right now.)

Do you have a case in mind where the speed of format string generation 
would be a bottleneck?

>> Some questions:
>>
>> How hard do we want to try supplying a buffer? Eg. if the consumer does
>> not specify strided but specifies suboffsets, should we try to compute
>> suitable suboffsets? Should we try making contiguous copies of the data
>> (I guess this would break buffer semantics?)?
>>   
> Actually per the PEP, suboffsets imply strided:
> 
> #define PyBUF_INDIRECT (0x0100 | PyBUF_STRIDES)
> 
> :-) So there's no real way for a consumer to specify only suboffsets,
> 0x0100 is not a possible flag I think. Suboffsets can't really work
> without the strides anyway IIUC, and in the case of NumPy the field can
> always be left at 0.

Ok, great!

> IMO one should very much stay clear of making contiguous copies,
> especially considering the existance of PyBuffer_ToContiguous, which
> makes it trivial for client code to get a pointer to a contiguous buffer
> anyway. The intention of the PEP seems to be to export the buffer in as
> raw form as possible.

This is what I thought, too.

> Do keep in mind that IS_C_CONTIGUOUS and IS_F_CONTIGUOUS go be too
> conservative with NumPy arrays. If a contiguous buffer is requested,
> then  looping through the strides and checking that the strides are
> monotonically decreasing/increasing could eventually save copying in
> some cases. I think that could be worth it -- I actually have my own
> code for IS_F_CONTIGUOUS rather than relying on the flags personally
> because of this issue, so it does come up in practice.

Are you sure?

Assume monotonically increasing or decreasing strides with inner stride 
of itemsize. Now, if the strides are not C or F-contiguous, doesn't this 
imply that part of the data in the memory block is *not* pointed to by a 
set of indices? [For example, strides = {itemsize, 3*itemsize}; dims = 
{2, 2}. Now, there is unused memory between items (1,0) and (0,1).]

This probably boils down to what exactly was meant in the PEP and Python 
docs by "contiguous". I'd believe it was meant to be the same as in Numpy 
-- that you can send the array data e.g. to Fortran as-is. If so, there 
should not be gaps in the data, if the client explicitly requested that 
the buffer be contiguous.

Maybe you meant that the Numpy array flags (which the macros check) are 
not always up-to-date wrt. the stride information?

-- 
Pauli Virtanen