Erin Sheldon wrote:

On 3/15/06, Perry Greenfield wrote:

...

You are right that this is messy. We would like to change this sometime. But we'd like to complete the transition to numpy first before doing that so it may be some months before we can (and it may not look quite like what you suggest). But your point is very valid.

Thanks, Perry

OK, fair enough.

Incidentally, I realized that this attribute _coldefs is not part of recarray anyway, but something added by pyfits. I see now that the names and the formats with a greater than sign concatenated on the front can be extracted from dtype:

In [247]: t.dtype Out[247]: [('x', '>f4'), ('y', '>i4')]

I could write my own function to extract what I need, but I thought I would ask: is there already a simpler way? And is there a function to compare this '>f4' stuff to the named types such as Float32 ('f')?

The dtype object does contain what you want. In fact. It's the fields attribute of the dtype object that is a dictionary accessed by field name. Thus, to see if a field is a valid field itdentifier, if name in t.dtype.fields: would work (well there is a slight problem in that -1 is a special key to the dictionary that returns a list of field names ordered by offset and so would work also), but if you now that name is already a string, then no problem. -Travis

Fernando Perez wrote:

Mmh, just curious: I wonder about the wisdom of that overloading of a 'magic' key (-1). It will make thinks like

for name in t.dtype.fields:

No real wisdom. More organic growth. Intially I didn't have an ordered list of fields, but as more complicated data-type descriptors were supported, this became an important thing to have. I should have probably added a n additional element to the PyArray_Descr structure. Remember, it was growing out of the old PyArray_Descr already and I was a little concerned about changing it too much (there are ramifications of changing this structure in several places). So, instead of adding a "ordered_names" tuple to the dtype object, I just stuck it in the fields dictionary. I agree it's kind of odd sitting there. It probably deserves a re-factoring and pulling that out into a new attribute of the dtype object. This would mean that the PyArray_Descr structure would need a new object (names perhaps), and it would need to be tracked. Not a huge change and probably worth it before the next release. -Travis

On 3/15/06, Travis Oliphant wrote:

The dtype object does contain what you want. In fact. It's the fields attribute of the dtype object that is a dictionary accessed by field name. Thus, to see if a field is a valid field itdentifier,

if name in t.dtype.fields:

would work (well there is a slight problem in that -1 is a special key to the dictionary that returns a list of field names ordered by offset and so would work also), but if you now that name is already a string, then no problem.

Yes, I see, but I think you meant if name in t.dtype.fields.keys(): which contains the -1 as a key. In [275]: t.dtype.fields.keys: Out[275]: ('tag1','tag2',-1) So, since the last key points to the names, one can also do: In [279]: t.dtype.fields[-1] Out[279]: ('tag1', 'tag2') which is not transparent, but does what you need. For now, I'll probably just write a simple function to wrap this. Thanks, Erin

Erin Sheldon wrote:

Yes, I see, but I think you meant

if name in t.dtype.fields.keys():

No, he really meant: if name in t.dtype.fields: dictionaries are iterators, so you don't need to construct the list of keys separately. It's just a redundant waste of time and memory in most cases, unless you intend to modify the dict in your loop, case in which the iterator approach won't work and you /do/ need the explicit keys() call. In addition if name in t.dtype.fields is faster than: if name in t.dtype.fields.keys() While both are O(N) operations, the first requires a single call to the hash function on 'name' and then a C lookup in the dict's internal key table as a hash table, while the second is a direct walkthrough of a list with python-level equality testing. In [15]: nkeys = 1000000 In [16]: dct = dict(zip(keys,[None]*len(keys))) In [17]: time bool(-1 in keys) CPU times: user 0.01 s, sys: 0.00 s, total: 0.01 s Wall time: 0.01 Out[17]: False In [18]: time bool(-1 in dct) CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 Out[18]: False In realistic cases for your original question you are not likely to see the difference, but it's always a good idea to be aware of the performance characteristics of various approaches. For a different problem, there may well be a real difference. Cheers, f

Nice. Python decides to compare with the keys and not the values. The possibilities for obfuscation are endless. On 3/15/06, Fernando Perez wrote:

Erin Sheldon wrote:

...

Yes, I see, but I think you meant

if name in t.dtype.fields.keys():

No, he really meant:

if name in t.dtype.fields:

dictionaries are iterators, so you don't need to construct the list of keys separately. It's just a redundant waste of time and memory in most cases, unless you intend to modify the dict in your loop, case in which the iterator approach won't work and you /do/ need the explicit keys() call.

In addition

if name in t.dtype.fields

is faster than:

if name in t.dtype.fields.keys()

While both are O(N) operations, the first requires a single call to the hash function on 'name' and then a C lookup in the dict's internal key table as a hash table, while the second is a direct walkthrough of a list with python-level equality testing.

In [15]: nkeys = 1000000

In [16]: dct = dict(zip(keys,[None]*len(keys)))

In [17]: time bool(-1 in keys) CPU times: user 0.01 s, sys: 0.00 s, total: 0.01 s Wall time: 0.01 Out[17]: False

In [18]: time bool(-1 in dct) CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s Wall time: 0.00 Out[18]: False

In realistic cases for your original question you are not likely to see the difference, but it's always a good idea to be aware of the performance characteristics of various approaches. For a different problem, there may well be a real difference.

Cheers,

f

On Mar 15, 2006, at 6:41 PM, Erin Sheldon wrote:

On 3/15/06, Perry Greenfield wrote:

...

You are right that this is messy. We would like to change this sometime. But we'd like to complete the transition to numpy first before doing that so it may be some months before we can (and it may not look quite like what you suggest). But your point is very valid.

Thanks, Perry

Erin P.S. If it is man power that is preventing some of the simple things like this from being implemented, I could volunteer some time.

Ultimately, yes. But there's a little more to it. We've felt there are a number of improvements that can be made and that these might benefit from thinking more globally about how to do them rather than add a few convenience functions. Then there is the issue of whether these changes are added to the numarray version, etc. But help would be welcomed. I'd say wait a little bit until we put out an alpha and then start some discussion on how it should be improved. The astropy mailing list is a better forum for that than this one though. Thanks, Perry