On Tue, Feb 26, 2013 at 11:17 AM, Todd wrote:

Is numpy planning to participate in GSOC this year, either on their own or as a part of another group?

If we participate, it should be under the PSF organization. I suspect participation for NumPy (and SciPy) largely depends on mentors being available.

If so, should we start trying to get some project suggestions together?

That can't hurt - good project descriptions will be useful not just for GSOC but also for people new to the project looking for ways to contribute. I suggest to use the wiki on Github for that. Ralf

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Mon, Mar 4, 2013 at 2:29 PM, Todd wrote:

5. Currently dtypes are limited to a set of fixed types, or combinations of these types. You can't have, say, a 48 bit float or a 1-bit bool. This project would be to allow users to create entirely new, non-standard dtypes based on simple rules, such as specifying the length of the sign, length of the exponent, and length of the mantissa for a custom floating-point number. Hopefully this would mostly be used for reading in non-standard data and not used that often, but for some situations it could be useful for storing data too (such as large amounts of boolean data, or genetic code which can be stored in 2 bits and is often very large).

I second this general idea. Simply having a pair of packbits/unpackbits functions that could work with 2 and 4 bit uints would make my life easier. If it were possible to have an array of dtype 'uint4' that used half the space of a 'uint8', but could have ufuncs an the like ran on it, it would be pure bliss. Not that I'm complaining, but a man can dream... Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

On Mon, Mar 4, 2013 at 4:29 PM, Todd wrote:

3. Structured arrays are accessed in a manner similar to python dictionaries, using a key. However, they don't support the normal python dictionary methods like keys, values, items, iterkeys, itervalues, iteritems, etc. This project would be to implement as much of the dictionary (and ordereddict) API as possible in structured arrays (making sure that the resulting API presented to the user takes into account whether python 2 or python 3 is being used).

Along these lines: what about implementing the new "memory friendly" dictionary [0] with a NumPy structured array backend for the dense array portion, and allowing any specified column of the array to be the dictionary keys? This would merge the strengths of NumPy structured arrays with Python dictionaries. Some thought would have to be given to mutability / immutability issues, but these are surmountable. Further enhancements would be to allow for multiple key columns -- analogous to multiple indexes into a database. [0] http://mail.python.org/pipermail/python-dev/2012-December/123028.html

4. The numpy ndarray class stores data in a regular manner in memory. This makes many linear algebra operations easier, but makes changing the number of elements in an array nearly impossible in practice unless you are very careful. There are other data structures that make adding and removing elements easier, but are not as efficient at linear algebra operations. The purpose of this project would be to create such a class in numpy, one that is duck type compatible with ndarray but makes resizing feasible. This would obviously come at a performance penalty for linear algebra related functions. They would still have consistent dtypes and could not be nested, unlike python lists. This could either be based on a new c-based type or be a subclass of list under the hood.

This made me think of a serious performance limitation of structured dtypes: a structured dtype is always "packed", which may lead to terrible byte alignment for common types. For instance, `dtype([('a', 'u1'), ('b', 'u8')]).itemsize == 9`, meaning that the 8-byte integer is not aligned as an equivalent C-struct's would be, leading to all sorts of horrors at the cache and register level. Python's ctypes does the right thing here, and can be mined for ideas. For instance, the equivalent ctypes Structure adds pad bytes so the 8-byte integer is on the correct boundary: class Aligned(ctypes.Structure): _fields_ = [('a', ctypes.c_uint8), ('b', ctypes.c_uint64)] print ctypes.sizeof(Aligned()) # --> 16 I'd be surprised if someone hasn't already proposed fixing this, although perhaps this would be outside the scope of a GSOC project. I'm willing to wager that the performance improvements would be easily measureable. Just some more thoughts. Kurt

On 2013/03/04 9:01 PM, Nicolas Rougier wrote:

...

...

This made me think of a serious performance limitation of structured dtypes: a structured dtype is always "packed", which may lead to terrible byte alignment for common types. For instance, `dtype([('a', 'u1'), ('b', 'u8')]).itemsize == 9`, meaning that the 8-byte integer is not aligned as an equivalent C-struct's would be, leading to all sorts of horrors at the cache and register level.

Doesn't the "align" kwarg of np.dtype do what you want? In [2]: dt = np.dtype(dict(names=['a', 'b'], formats=['u1', 'u8']), align=True) In [3]: dt.itemsize Out[3]: 16 Eric

On Tue, Mar 5, 2013 at 1:45 AM, Eric Firing wrote:

On 2013/03/04 9:01 PM, Nicolas Rougier wrote:

...

...

...

This made me think of a serious performance limitation of structured dtypes: a structured dtype is always "packed", which may lead to terrible byte alignment for common types. For instance, `dtype([('a', 'u1'), ('b', 'u8')]).itemsize == 9`, meaning that the 8-byte integer is not aligned as an equivalent C-struct's would be, leading to all sorts of horrors at the cache and register level.

Doesn't the "align" kwarg of np.dtype do what you want?

In [2]: dt = np.dtype(dict(names=['a', 'b'], formats=['u1', 'u8']), align=True)

In [3]: dt.itemsize Out[3]: 16

Thanks! That's what I get for not checking before posting. Consider this my vote to make `aligned=True` the default.

Eric _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Mar 5, 2013 7:53 PM, "Nathaniel Smith" wrote:

On 4 Mar 2013 23:21, "Jaime Fernández del Río"

...

On Mon, Mar 4, 2013 at 2:29 PM, Todd wrote:

...

5. Currently dtypes are limited to a set of fixed types, or

combinations of these types. You can't have, say, a 48 bit float or a 1-bit bool. This project would be to allow users to create entirely new, non-standard dtypes based on simple rules, such as specifying the length of

...

I second this general idea. Simply having a pair of packbits/unpackbits

functions that could work with 2 and 4 bit uints would make my life easier. If it were possible to have an array of dtype 'uint4' that used half the space of a 'uint8', but could have ufuncs an the like ran on it, it would be pure bliss. Not that I'm complaining, but a man can dream...

This would be quite difficult, since it would require reworking the guts of the ndarray data structure to store strides and buffer offsets in bits rather than bytes, and probably with endianness handling too. Indexing is all done at the ndarray buffer-of-bytes layer, without any involvement of

wrote: the sign, length of the exponent, and length of the mantissa for a custom floating-point number. Hopefully this would mostly be used for reading in non-standard data and not used that often, but for some situations it could be useful for storing data too (such as large amounts of boolean data, or genetic code which can be stored in 2 bits and is often very large). the dtype.

Consider:

a = zeros(10, dtype=uint4) b = a[1::3]

Now b is a view onto a discontiguous set of half-bytes within a...

You could have a dtype that represented several uint4s that together

added up to an integral number of bytes, sort of like a structured dtype. Or packbits()/unpackbits(), like you say.

-n

Then perhaps such a project could be a four-stage thing. 1. Allow for the creation of int, unit, float, bool, and complex dtypes with an arbitrary number of bytes 2. Allow for the creation of dtypes which are integer fractions of a byte, (1, 2, or 4 bits), and must be padded to a whole byte. 3. Have an optional internal value in an array that tells it to exclude the last n bits of the last byte. This would be used to hide the padding from step 2. This should be abstracted into a general-purpose method for excluding bits from the byte-to-dtype conversion so it can be used in step 4. 4. Allow for the creation of dtypes that are non-integer fractions of a byte or non-integer multiples of a byte (3, 5, 6, 7, 9, 10, 11, 12, etc, bits). Each element in the array would be stored as a certain number of bytes, with the method from 3 used to cut it down to the right number of bits. So a 3 bit dtype would have two elements per byte with 2 bits excluded. A 5 bit dtype would have 1 element per byte with 3 bits excluded. A 12 bit dtype would have one element in two bytes with with 4 bits excluded from the second byte. This approach would allow for arbitrary numbers of bits without breaking the internal representation, would have each stage building off the previous stage, and we would still have something useful even if not all the stages are completed.

On Wed, Mar 6, 2013 at 4:29 AM, Francesc Alted wrote:

On 3/5/13 7:14 PM, Kurt Smith wrote:

...

On Tue, Mar 5, 2013 at 1:45 AM, Eric Firing wrote:

...

On 2013/03/04 9:01 PM, Nicolas Rougier wrote:

...

...

...

This made me think of a serious performance limitation of structured dtypes: a structured dtype is always "packed", which may lead to terrible byte alignment for common types. For instance, `dtype([('a', 'u1'), ('b', 'u8')]).itemsize == 9`, meaning that the 8-byte integer is not aligned as an equivalent C-struct's would be, leading to all sorts of horrors at the cache and register level. Doesn't the "align" kwarg of np.dtype do what you want?

In [2]: dt = np.dtype(dict(names=['a', 'b'], formats=['u1', 'u8']), align=True)

In [3]: dt.itemsize Out[3]: 16 Thanks! That's what I get for not checking before posting.

Consider this my vote to make `aligned=True` the default.

I would not run too much. The example above takes 9 bytes to host the structure, while a `aligned=True` will take 16 bytes. I'd rather let the default as it is, and in case performance is critical, you can always copy the unaligned field to a new (homogeneous) array.

Yes, I can absolutely see the case you're making here, and I made my "vote" with the understanding that `aligned=False` will almost certainly stay the default. Adding 'aligned=True' is simple for me to do, so no harm done. My case is based on what's the least surprising behavior: C structs / all C compilers, the builtin `struct` module, and ctypes `Structure` subclasses all use padding to ensure aligned fields by default. You can turn this off to get packed structures, but the default behavior in these other places is alignment, which is why I was surprised when I first saw that NumPy structured dtypes are packed by default.

-- Francesc Alted

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion