So in this case you've written the field to behave one way during field detection and another way during the real field evaluation. This is confusing the field machinery.

I think you're running into a bug in how the artio frontend handles some of its particle fields, see this pull request I just opened:

https://github.com/yt-project/yt/pull/1802

After applying that patch I think you'll be able to avoid the check to see if you're in field detection.

On Sun, May 27, 2018 at 1:20 PM, Nick Gnedin <ngnedin@gmail.com> wrote:

Nathan,

I made progress with the real code, but then I hit another error that leaves me dumbfounded:

----------------------------
import yt

def StellarLuminosity(field,data):
    if(not 'FieldDetector' in str(type(data))):
        ts = data[('STAR','age')].in_units("yr").v
        zs = (data[('STAR','metallicity_snii')]+data[('STAR','metallicity_snia')]).v
        return data[('STAR','mass')].in_units("Msun")
    else:
        return data[('STAR','age')]


d = yt.load("rei10_a0.1001/rei10_a0.1001.art")


d.add_field(("STAR","luminosity"),function=StellarLuminosity,units="Msun",sampling_type="particle")
d.add_deposited_particle_field(("STAR","luminosity"),"nearest",weight_field='particle_ones')

print(d.derived_field_list)

print(d.all_data()[('STAR','metallicity_snii')])
print(d.all_data()[("deposit","STAR_nn_luminosity")])
--------------------------

Below is the error I get. ('STAR','metallicity_snii') certainly exists, I can print it, but the deposited field seems to be unable to find it.

yt : [INFO     ] 2018-05-27 13:15:16,205 Created 256 chunks for ARTIO
[  5.39916630e-30   2.21650149e-22   2.95487393e-06 ...,   1.23813297e-05
   1.28902484e-05   1.42594863e-05] dimensionless
yt : [INFO     ] 2018-05-27 13:15:16,396 Created 256 chunks for ARTIO
Traceback (most recent call last):
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 335, in _generate_fluid_field
    finfo.check_available(gen_obj)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/fields/derived_field.py", line 199, in check_available
    validator(data)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/fields/derived_field.py", line 418, in __call__
    raise NeedsGridType(self.ghost_zones,self.fields)
yt.fields.field_exceptions.NeedsGridType: (0, None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "debug.py", line 23, in <module>
    print(d.all_data()[("deposit","STAR_nn_luminosity")])
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 282, in __getitem__
    self.get_data(f)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 1337, in get_data
    self._generate_fields(fields_to_generate)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 1357, in _generate_fields
    fd = self._generate_field(field)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 319, in _generate_field
    tr = self._generate_fluid_field(field)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 337, in _generate_fluid_field
    rv = self._generate_spatial_fluid(field, ngt_exception.ghost_zones)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 357, in _generate_spatial_fluid
    ind += o.select(self.selector, self[field], rv, ind)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 282, in __getitem__
    self.get_data(f)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 1337, in get_data
    self._generate_fields(fields_to_generate)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 1357, in _generate_fields
    fd = self._generate_field(field)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 319, in _generate_field
    tr = self._generate_fluid_field(field)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 339, in _generate_fluid_field
    rv = finfo(gen_obj)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/fields/derived_field.py", line 237, in __call__
    dd = self._function(self, data)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/static_output.py", line 1274, in _deposit_field
    fields = [data[ptype, deposit_field]]
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/octree_subset.py", line 75, in __getitem__
    tr = super(OctreeSubset, self).__getitem__(key)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 282, in __getitem__
    self.get_data(f)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 1337, in get_data
    self._generate_fields(fields_to_generate)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 1357, in _generate_fields
    fd = self._generate_field(field)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 317, in _generate_field
    tr = self._generate_particle_field(field)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/data_objects/data_containers.py", line 402, in _generate_particle_field
    rv = self.ds._get_field_info(*field)(gen_obj)
  File "/data/gnedin/soft/anaconda3/lib/python3.6/site-packages/yt/fields/derived_field.py", line 235, in __call__
    "for %s" % (self.name,))
RuntimeError: Something has gone terribly wrong, _function is NullFunc for ('STAR', 'METALLICITY_SNII')


n





On 5/27/2018 12:08 PM, Nathan Goldbaum wrote:
Hi Nick,

What you're seeing in those print statements is how yt's field detection system works. Before we actually select and return data, we pass your field function fake data to make sure it doesn't raise errors or it doesn't select fields that aren't available for your dataset.

Here's a script I made based on yours that helps explain what's going on:

https://gist.github.com/ngoldbaum/4d27987bda8ecbce5c9bd9b98bbbbf49

And here's the output of that script:

https://gist.github.com/ngoldbaum/475fcd827a53e4d02e76cb25711ff5ea

I made a couple modifications compared to your script. First, I added a line at the end that selects your new field from a data object, which causes real data to be fed into the field function after field detection completes. I also specified "units='auto'" in the call to add_field. The dimensions keyword is only used when "units='auto'" is set.

You can see that your field function is called twice. First with a `FieldDetector` instance and then the second time with a real yt data object. In general the field function might be called several times before real data is supplied to it. One way I commonly deal with this is the following hack to only see output when the field definition is called with real data:

     def NewField(field, data):
         q = data[('STAR', 'age')]
         if 'FieldDetector' in str(type(data)):
             print("BBB",data)
         return q

Although more often I use python's debugger via "import pdb; pdb.set_trace()" instead of print statements.

Hope that helps,

Nathan

On Sun, May 27, 2018 at 10:51 AM, Nick Gnedin <ngnedin@gmail.com <mailto:ngnedin@gmail.com>> wrote:


    Guys,

    I am trying to add a new particle field, and apparently I am doing
    something wrong, but I could find the documentation for this:

    import yt
    def NewField(field,data):
         q = data[('STAR','age')]
         print("BBB",data)
         return q
    d = yt.load("rei10_a0.1001/rei10_a0.1001.art")
    print('AAA',d.all_data()[("STAR","age")])
    d.add_field(("STAR","new-field"),function=NewField,dimensions="time",sampling_type="particle")

    The result is here:

    AAA [  1.35689106e+08   1.29578798e+08   1.15241829e+08 ...,
    2.97175878e+07
        2.04648908e+07   1.35365980e+07] yr
    BBB defaultdict(<function FieldDetector.__init__.<locals>.<lambda>
    at 0x2b9e698e3598>, {('STAR', 'BIRTH_TIME'): YTArray([ 1.])
    (dimensionless), ('STAR', 'creation_time'): YTArray([ 1.])
    (dimensionless), ('STAR', 'age'): YTArray([ 1.]) (dimensionless)})

    It appears that the data object sent to NewField(field,data) has all
    its fields set to dimensionless.

    I tried omitting sampling_type="particle", but it does not help,
    just generates an incomprehensible warning.

    n
    _______________________________________________
    yt-users mailing list -- yt-users@python.org
    <mailto:yt-users@python.org>
    To unsubscribe send an email to yt-users-leave@python.org
    <mailto:yt-users-leave@python.org>