On Mar 5, 2013 7:53 PM, "Nathaniel Smith" <njs@pobox.com> wrote:
>
> On 4 Mar 2013 23:21, "Jaime Fernández del Río" <jaime.frio@gmail.com> wrote:
> >
> > On Mon, Mar 4, 2013 at 2:29 PM, Todd <toddrjen@gmail.com> wrote:
> >>
> >>
> >> 5. Currently dtypes are limited to a set of fixed types, or combinations of these types.  You can't have, say, a 48 bit float or a 1-bit bool.  This project would be to allow users to create entirely new, non-standard dtypes based on simple rules, such as specifying the length of the sign, length of the exponent, and length of the mantissa for a custom floating-point number.  Hopefully this would mostly be used for reading in non-standard data and not used that often, but for some situations it could be useful for storing data too (such as large amounts of boolean data, or genetic code which can be stored in 2 bits and is often very large).
> >
> >
> > I second this general idea. Simply having a pair of packbits/unpackbits functions that could work with 2 and 4 bit uints would make my life easier. If it were possible to have an array of dtype 'uint4' that used half the space of a 'uint8', but could have ufuncs an the like ran on it, it would be pure bliss. Not that I'm complaining, but a man can dream...
>
> This would be quite difficult, since it would require reworking the guts of the ndarray data structure to store strides and buffer offsets in bits rather than bytes, and probably with endianness handling too. Indexing is all done at the ndarray buffer-of-bytes layer, without any involvement of the dtype.
>
> Consider:
>
> a = zeros(10, dtype=uint4)
> b = a[1::3]
>
> Now b is a view onto a discontiguous set of half-bytes within a...
>
> You could have a dtype that represented several uint4s that together added up to an integral number of bytes, sort of like a structured dtype. Or packbits()/unpackbits(), like you say.
>
> -n

Then perhaps such a project could be a four-stage thing.

1. Allow for the creation of int, unit, float, bool, and complex dtypes with an arbitrary number of bytes

2. Allow for the creation of dtypes which are integer fractions of a byte, (1, 2, or 4 bits), and must be padded to a whole byte.

3. Have an optional internal value in an array that tells it to exclude the last n bits of the last byte.  This would be used to hide the padding from step 2.  This should be abstracted into a general-purpose method for excluding bits from the byte-to-dtype conversion so it can be used in step 4.

4. Allow for the creation of dtypes that are non-integer fractions of a byte or non-integer multiples of a byte (3, 5, 6, 7, 9, 10, 11, 12, etc, bits). Each element in the array would be stored as a certain number of bytes, with the method from 3 used to cut it down to the right number of bits.  So a 3 bit dtype would have two elements per byte with 2 bits excluded. A 5 bit dtype would have 1 element per byte with 3 bits excluded. A 12 bit dtype would have one element in two bytes with with 4 bits excluded from the second byte.

This approach would allow for arbitrary numbers of bits without breaking the internal representation, would have each stage building off the previous stage, and we would still have something useful even if not all the stages are completed.