[Numpy-discussion] Non-numerical info associated with sub-arrays

[Tim Churches]

On Fri, 2002-12-27 at 11:29, Magnus Lie Hetland wrote:
> I'm working on some two-dimensional tables of data, where some data
> are numerical, while other aren't. I'd like to use numarray's
> numerical capabilities with the numerical parts (columns) while
> keeping the data in each row together. (I'm sure this generalizes to
> more dimensions, and to sub-arrays in general, not just rows.)
> 
> It's not a hard problem, really, but the obvious solution--to keep
> the other rows in separate arrays/lists and just juggle things
> around--seems a bit clunky. I was just wondering if anyone had other
> ideas (would it be practical to include all the data in a single array
> somehow--I seem to recall that Numeric could have arbitrary object
> arrays, but I'm not sure whether numarray supports this?) or perhaps
> some hints on how to organize code around this? I wrote a small class
> that wraps things up and works a bit lik R/S-plus's data frames; is
> there some other more standard code out there for this sort of thing?
> (It's a problem that crops up often in data processing of various
> kinds...)

Have a look at the discussion on RecordArrays in this overview of
Numarray: http://stsdas.stsci.edu/numarray/DesignOverview.html

However, in the meantime, as you note, its not too hard to write a class
which emulates R/S-Plus data frames. Just store each column in its own
Numeric array of the appropriate type (which might be the PyObject
types, which can hold any Python object type), and have the wrapper
class implement __getitem__ etc to collect the relevant "rows" from each
column and return them as a complete row as a dict or a sequence. Not
that fast, but not slow either. You can implement a generator to allow
cursor-like traversal of the all the rows if you like. Happy to
collaborate on furthering this idea. By memory-mapping disc-based
versions of the  Numeric arrays, and using the BsdDb3 record number
database format for the string columns, you can even make a disc-based
"record array" which can be larger than available RAM+swap. I hope to
release code written under contract by Dave Cole (see
http://www.object-craft.com.au ) which illustrates this idea in the next
month or so (but I've been saying that to myself for a year or more...).

Tim C