On 7/8/07, Vincent Nijs <v-nijs@kellogg.northwestern.edu> wrote:

Thanks for looking into this Torgil! I agree that this is a much more
complicated setup. I'll check if there is anything I can do on the data end.
Otherwise I'll go with Timothy's suggestion and read in numbers as floats
and convert to int later as needed.

Here is a strategy that should allow auto detection without too much in the way of inefficiency. The basic idea is to convert till you run into a problem, store that data away, and continue the conversion with a new dtype. At the end you assemble all the chunks of data you've accumulated into one large array. It should be reasonably efficient in terms of both memory and speed.

The implementation is a little rough, but it should get the idea across.