Hi Wes, On Mon, Mar 12, 2012 at 9:33 AM, Wes McKinney <wesmckinn@gmail.com> wrote:

Now, one problem with this is that you want the mapping + dtype to be invertible (otherwise you're left doing some type inference). The way that I implement the mapping is to restrict the labeling to be from 0 to N - 1 which makes things easier.

If we decide that having an explicit value mapping (...?)

You might want to finish whatever thought that was :)

On 03/13/2012 06:44 PM, Mark Wiebe wrote:

On Fri, Mar 9, 2012 at 8:55 AM, Bryan Van de Ven <bryanv@continuum.io <mailto:bryanv@continuum.io>> wrote:

Hi all,

I have started working on a NEP for adding an enumerated type to NumPy. It is on my GitHub:

https://github.com/bryevdv/numpy/blob/enum/doc/neps/enum.rst

It is still very rough, and incomplete in places. But I would like to get feedback sooner rather than later in order to refine it. In particular there are a few questions inline in the document that I would like input on. Any comments, suggestions, questions, concerns, etc. are very welcome.

This looks like a great start to me.

I think the open/closed enum distinction will need to be explored a little bit more, because it interacts with dtype immutability/hashability. Do you know if there are any examples of Python objects in the wild that dynamically convert from not being hashable (i.e. raising an exception if used as a dict key) to become hashable?

In Sage, the matrix objects are mutable when constructed, and you can set_immutable to make them immutable. The way I look at that though is that it is part of the construction phase of the object, you'd typically construct, fill it in, then set_immutable (to finish construction), then use it. set/frozenset is an example of the opposite, and a design I personally like better (i.e., "frozen_dtype" :-)). Dag

It might be worth adding a section which briefly compares and contrasts the proposed functionality with enums in various programming languages. Here are two links I found to try and get an idea:

MS on C# enum usage: http://msdn.microsoft.com/en-us/library/cc138362.aspx Wikipedia on C++ enum class: http://en.wikipedia.org/wiki/C%2B%2B11#Strongly_typed_enumerations

For example, the C# enum has a way to enable a "flags" mode, which will create successive powers of 2. This may not be a feature NumPy needs, but if people are finding it useful in C#, maybe it would be useful here too.

Cheers, Mark

Thanks,

Bryan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On Wed, Mar 14, 2012 at 1:44 AM, Mark Wiebe <mwwiebe@gmail.com> wrote:

...

On Fri, Mar 9, 2012 at 8:55 AM, Bryan Van de Ven <bryanv@continuum.io> wrote:

...

Hi all,

I have started working on a NEP for adding an enumerated type to NumPy. It is on my GitHub:

https://github.com/bryevdv/numpy/blob/enum/doc/neps/enum.rst

It is still very rough, and incomplete in places. But I would like to get feedback sooner rather than later in order to refine it. In particular there are a few questions inline in the document that I would like input on. Any comments, suggestions, questions, concerns, etc. are very welcome.

This looks like a great start to me.

I think the open/closed enum distinction will need to be explored a

...

bit more, because it interacts with dtype immutability/hashability. Do you know if there are any examples of Python objects in the wild that dynamically convert from not being hashable (i.e. raising an exception if used as a dict key) to become hashable?

I haven't run into any...

Thinking about it, I'm not sure I have any use case for this type being mutable. Maybe someone else can think of one? The first case that came to mind was in reading a large text file, where you want to (1) auto-create an enum, (2) use a pre-allocated array, and (3) don't know ahead of time what the levels are:

a = np.empty(lines_in_file, dtype=np.dtype(Enum())) for i, line in enumerate(f): field = line.split()[0] a.dtype.add_level(field) a[i] = field a.dtype.seal()

But really this is just can be done just as easily and efficiently without a mutable dtype:

a = np.empty(lines_in_file, dtype=np.int32) intern_table = {} next_level = 0 for i, line in enumerate(f): field = line.split()[0] val = intern_table.setdefault(field, next_level) if val == next_level: next_level += 1 a[i] = val a = a.view(dtype=np.dtype(Enum(map=intern_table)))

I notice that the HDF5 C library has a concept of open versus closed enums, but I can't tell from the documentation at hand why this is; it looks like it might just be a limitation of the implementation. (Like, a workaround for C's lack of a standard mapping type, which makes it inconvenient to pass in all the mappings in to a single API call.)

...

It might be worth adding a section which briefly compares and contrasts

On Thursday, March 15, 2012, Nathaniel Smith <njs@pobox.com> wrote: little the

...

proposed functionality with enums in various programming languages. Here are two links I found to try and get an idea:

MS on C# enum usage: http://msdn.microsoft.com/en-us/library/cc138362.aspx Wikipedia on C++ enum class: http://en.wikipedia.org/wiki/C%2B%2B11#Strongly_typed_enumerations

For example, the C# enum has a way to enable a "flags" mode, which will create successive powers of 2. This may not be a feature NumPy needs, but if people are finding it useful in C#, maybe it would be useful here too.

There's also a long, ongoing debate about how to do enums in Python -- e.g.: http://www.python.org/dev/peps/pep-0354/ http://pypi.python.org/pypi/enum/ http://pypi.python.org/pypi/enum_meta/ http://pypi.python.org/pypi/flufl.enum/ http://pypi.python.org/pypi/lazr.enum/ http://pypi.python.org/pypi/pyutilib.enum/ http://pypi.python.org/pypi/coding/

http://stackoverflow.com/questions/36932/whats-the-best-way-to-implement-an-...

I guess Guido likes flufl.enum: http://mail.python.org/pipermail/python-ideas/2011-July/010909.html

BUT, I'm not sure any of this is relevant at all. "Enums" are a programming language feature that are, first and foremost, about injecting names into your code's namespace. What I'm hoping to see is a dtype for holding categorical data, similar to an R "factor" http://stat.ethz.ch/R-manual/R-devel/library/base/html/factor.html https://svn.r-project.org/R/trunk/src/library/base/R/factor.R (NB: This is GPL code if anyone is paranoid about contamination, but also the most complete API description available) or an HDF5 "enum" http://www.hdfgroup.org/HDF5/doc/H5.user/Datatypes.html#Datatypes_Enum I believe pandas has some functionality along these lines too, though I can't find it in the online docs -- hopefully Wes will fill us in.

These are basically objects that act for most purposes like string arrays, but in which all strings are required to come from a finite, specified list. This list acts like some metadata attached to the array; it's order may or may not be significant. And they're implemented internally as integer arrays.

I'm not sure what it would even mean to treat this kind of data as "flags", since you can't take the bitwise-or of two strings...

-- Nathaniel

I guess my problem is that this isn't _quite_ like an enum that I am familiar with (but not quite unlike it either). Should we call it "factor", to avoid confusion or are there going to be too many that won't know what that is, but would be drawn in by a name of "enum"? Just a thought. Ben Root

On Mar 16, 2012 1:02 AM, "Stéfan van der Walt" <stefan <stefan@sun.ac.za>@<stefan@sun.ac.za> sun.ac.za <stefan@sun.ac.za>> wrote:

On Thu, Mar 15, 2012 at 4:02 PM, Nathaniel Smith <njs <njs@pobox.com>@<njs@pobox.com>

...

I'm not sure what it would even mean to treat this kind of data as "flags", since you can't take the bitwise-or of two strings...

This makes a more sense outside of ndarrays, where you would do something

pobox.com <njs@pobox.com>> wrote: like:

enum FLAG0 = 1, FLAG1 = 2, FLAG2 = 4 do_something(data, mode=FLAG0 & FLAG2)

The enum is therefore just a handle for its numerical value. While it may not be that useful in an array, I think Mark was just pointing out that there may be other similar use cases, such as enumerating from 0 to N-1, or in reverse from N-1 down to 0, or in steps of 2, or in powers of 2, etc.

Right, there may be. But are there? That's the question :-) It looks like R doesn't support anything except 1, ..., N numbering. There's really no reason it would either, since in their design the underlying integer values are almost entirely hidden from users. You could get at them if you wanted, but I bet less than 1% of users are even aware that factors and integers have anything to do with each other. Factors are just documented to be a way to store an array of strings drawn from a limited ordered list. (The ordering is important for things like polynomial coding and treatment versus baseline coding.) HDF5 supports arbitrary symbol<->integer mappings. 0, ..., N-1 coding makes the common problem of creating an indicator matrix very convenient: ind = np.zeros((enum_a.length, len(enum_.dtype.levels)), dtype=bool) ind[:, enum_a.view(dtype=np.int32)] = True But we can't restrict ourselves to only this coding if we want compatibility with HDF5 or R (because R is 1-based). So I guess supporting arbitrary mappings is worth it - though I doubt this flexibility will be used much. I'm curious if anyone can think of a reason they'd use it besides interoperability. Cheers, - Nathaniel

Hi all,

I have spent some time thinking about things, and discussing them with folks nearby. I actually got to wondering whether we really need new dtypes for this. It seems like enumerated values or factor levels could be cast as an annotation or metadata that could be attached to any existing integral dtypes. It spells differently enough that I have put up an alternate version that reflects this notion. I'd like to see what folks think of this direction:

https://github.com/bryevdv/numpy/blob/enum/doc/neps/enum_alt.rst

So this would require adding machinery to existing dtypes to behave

On Fri, Mar 16, 2012 at 4:26 PM, Bryan Van de Ven <bryanv@continuum.io> wrote: properly

when there is factor metadata present. Perhaps that is not an acceptable trade-off, but it seems worth discussing.

I took a look at this, but I think something was lost in the translation from your head to text :-). Your description here makes it sound like what's different about this proposal is that there's very different underlying mechanics, but the enum_alt file just seems to describe an alternative and more-or-less equivalent user-level API. Unless you told me, I would have assumed that it just created a new dtype, rather than modified existing ones. What mechanism are you thinking of? Or did I miss something?

I think a very similar approach could be used to add categorical ranges to any numerical or string types (I think they are called "shingles" in R?)

A 'shingle' is a way of mapping (floating point) numbers into categories. However, they generally allow a single number to fall into multiple categories. So for example, you might take these data points: 1 2 3 4 5 6 7 8 9 10 11 And divide them into categories A, B, C like this: 1 2 3 4 5 6 7 8 9 10 11 AAAAAAAAAAAAA BBBBBBBBBBBBB CCCCCCCCCCCCCCC Which is why they're called "shingles" :-) http://www.floridadisaster.org/hrg/images/roofs/shingle_loose_tab_large.jpg This can be a very convenient data structure for various sorts of visualizations, but I'm not sure how it would make sense to integrate it into basic numerical types. R has a more basic function called 'cut' which takes a numerical array plus some specified breakpoints, and returns a factor array. But that's a simple utility function that doesn't need any special features in the underlying representation. -- Nathaniel