On Apr 11, 2014 7:36 PM, "B.W. Keller" <kellerbw@mcmaster.ca> wrote:
>
> Alright everyone, I've fixed the problem. It looks like when we added support for pkdGrav's double precision positions, it was implemented in a rather slow way, the whole file needed to be loaded and processed (more than once). I've implemented all the filetype checking in a more efficient way. It now only reads ~30 bytes to check if a file is valid. Here's the PR:
>
> https://bitbucket.org/yt_analysis/yt/pull-request/811/quickly-validate-tipsy-files/diff
>
> BenWell, my face is red. Thanks to Ben for covering for me, but sounds like this was my fault! Sorry everyone.
Relatedly, answer tests are just about back online.
Matt
>
>
> On Fri, Apr 11, 2014 at 2:19 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
>>
>> I have implemented the suggested patch in this commit: https://bitbucket.org/yt_analysis/yt/commits/cc6b873a4471e049b6fef33146e92615c72f1243
>>
>>
>> On Fri, Apr 11, 2014 at 10:58 AM, B.W. Keller <kellerbw@mcmaster.ca> wrote:
>>>
>>> Yep, that's the quickest fix. I'll send a fix up later today. Sorry about this folks.
>>>
>>>
>>> On Fri, Apr 11, 2014 at 1:18 PM, Sam Skillman <samskillman@gmail.com> wrote:
>>>>
>>>> +1 to John's implementation. In the interim, tipsy can then still be loaded explicitly.
>>>>
>>>>
>>>> On Fri, Apr 11, 2014 at 10:13 AM, John ZuHone <jzuhone@gmail.com> wrote:
>>>>>
>>>>> Sorry, being a bit more descriptive... by that I mean:
>>>>>
>>>>> @classmethod
>>>>> def _is_valid(self, *args, **kwargs):
>>>>> return False
>>>>> ...
>>>>>
>>>>> Which is a bit hacky, but the least invasive.
>>>>>
>>>>> On Apr 11, 2014, at 1:07 PM, John ZuHone <jzuhone@gmail.com> wrote:
>>>>>
>>>>>> +1, except it seems like the easiest thing to do is to just make sure that Tipsy's _is_valid always returns False for now.
>>>>>>
>>>>>> On Apr 11, 2014, at 12:41 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> While Ben works on his fix, would anyone object to temporarily reverting the merged pull request that triggered this behavior?
>>>>>>>
>>>>>>> I worry about FLASH users who do not read yt-dev. I guess it *is* the bleeding edge, experimental version so bugs should be expected but still, reverting seems like an easy temporary fix that takes some pressure off Ben to quickly develop a true fix to the underlying issue.
>>>>>>>
>>>>>>> Nathan
>>>>>>>
>>>>>>> On Thursday, April 10, 2014, B.W. Keller <kellerbw@mcmaster.ca> wrote:
>>>>>>>>
>>>>>>>> Good idea Mike! I'll do that too.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Apr 10, 2014 at 7:39 PM, Mike Warren <mswarren@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> You could detect that it is not a tipsy file quickly, by using the
>>>>>>>>> constraint in the header that nbodies=nsph+ndark+nstar and ndim is
>>>>>>>>> presumably 1,2 or 3.
>>>>>>>>>
>>>>>>>>> struct tipsy_dump {
>>>>>>>>> double time;
>>>>>>>>> int nbodies;
>>>>>>>>> int ndim;
>>>>>>>>> int nsph;
>>>>>>>>> int ndark;
>>>>>>>>> int nstar;
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Apr 10, 2014 at 5:32 PM, B.W. Keller <kellerbw@mcmaster.ca> wrote:
>>>>>>>>> > Oh my. Sorry that I have introduced this, unfortunately there is no way to
>>>>>>>>> > detect tipsy files other than actually reading the entire file from disk.
>>>>>>>>> > Perhaps the way to fix this would be to drop the priority of tipsy datasets
>>>>>>>>> > to the bottom, so that a valid FLASH dataset will be detected prior to the
>>>>>>>>> > Tipsy check?
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Thu, Apr 10, 2014 at 7:15 PM, Nathan Goldbaum <nathan12343@gmail.com>
>>>>>>>>> > wrote:
>>>>>>>>> >>
>>>>>>>>> >> Hi all,
>>>>>>>>> >>
>>>>>>>>> >> We just had a bug report from Aaron Smith at UT Austin. The symptom is
>>>>>>>>> >> that the "load" comman was taking 30 seconds to complete on his FLASH
>>>>>>>>> >> dataset, which should never happen for FLASH.
>>>>>>>>> >>
>>>>>>>>> >> After asking him to profile the code, he produced the following profile:
>>>>>>>>> >>
>>>>>>>>> >> http://ngoldbaum.net/yt-load/
>>>>>>>>> >>
>>>>>>>>> >> It seems that the recent changes to the Tipsy frontend which allow it to
>>>>>>>>> >> autodetect binary outputs have made it so in some cases non-tipsy data is
>>>>>>>>> >> loaded off disk.
>>>>>>>>> >>
>>>>>>>>> >> I'm not sure about the best way to handle this, which is why I'm writing
>>>>>>>>> >> to the list rather than issuing a PR.
>>>>>>>>> >>
>>>>>>>>> >> -Nathan
>>>>>>>>> >>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > yt-dev mailing list
>>>>>>>>> > yt-dev@lists.spacepope.org
>>>>>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>> >
>>>>>>>>> _______________________________________________
>>>>>>>>> yt-dev mailing list
>>>>>>>>> yt-dev@lists.spacepope.org
>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> yt-dev mailing list
>>>>>>> yt-dev@lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev@lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev@lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev@lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev@lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev@lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
_______________________________________________
yt-dev mailing list
yt-dev@lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org