Hi all,
Some problems Sam was having with the fortan kdtree has alerted me to some necessary changes due to changes in Forthon. Two of them are simple. One is to remove a --with-numpy flag in the Makefile. The second is that the most recent version of Forthon (0.8.9) builds the fKDpy.so object in it's own build/lib.arch.../ directory, not in the same directory as the Makefile. This is easily fixed by a change to the setup.py file in yt/utilities. I will issue a PR for this depending on the discussion of the following issue.
The not-simple change is that the newest Forthon setup.py is hard-coded to install the Forthon package for the system Python, regardless of which actual Python stack you're using, which is not convenient for our custom-curated yt stack. The fix is to simply remove the hard-coding. But I want to ask y'all what the best course forward from here is: A) ask the author to change it, B) make the change in the Forthon linked to in the install_script, or C) stick to the old version for now?
Thanks!
Hi Stephen.
I like option A the best. That's definitely a bug. Maybe stick to the old version until that's fixed?
Just curious, which parts of yt depend on Forthon?
Best, Casey
On Tue, Aug 7, 2012 at 1:03 PM, Stephen Skory s@skory.us wrote:
Hi all,
Some problems Sam was having with the fortan kdtree has alerted me to some necessary changes due to changes in Forthon. Two of them are simple. One is to remove a --with-numpy flag in the Makefile. The second is that the most recent version of Forthon (0.8.9) builds the fKDpy.so object in it's own build/lib.arch.../ directory, not in the same directory as the Makefile. This is easily fixed by a change to the setup.py file in yt/utilities. I will issue a PR for this depending on the discussion of the following issue.
The not-simple change is that the newest Forthon setup.py is hard-coded to install the Forthon package for the system Python, regardless of which actual Python stack you're using, which is not convenient for our custom-curated yt stack. The fix is to simply remove the hard-coding. But I want to ask y'all what the best course forward from here is: A) ask the author to change it, B) make the change in the Forthon linked to in the install_script, or C) stick to the old version for now?
Thanks!
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
I've been resorting to 0.8.7 for the past couple of months after 0.8.8 and 0.8.9 were released because I run into strange errors, probably because of the not-simlpe-fix issue Stephen noted. The portion of the code where I encountered problems were doing parallel HOP halo finding.
From G.S.
On Tue, Aug 7, 2012 at 1:36 PM, Casey W. Stark caseywstark@gmail.comwrote:
Hi Stephen.
I like option A the best. That's definitely a bug. Maybe stick to the old version until that's fixed?
Just curious, which parts of yt depend on Forthon?
Best, Casey
On Tue, Aug 7, 2012 at 1:03 PM, Stephen Skory s@skory.us wrote:
Hi all,
Some problems Sam was having with the fortan kdtree has alerted me to some necessary changes due to changes in Forthon. Two of them are simple. One is to remove a --with-numpy flag in the Makefile. The second is that the most recent version of Forthon (0.8.9) builds the fKDpy.so object in it's own build/lib.arch.../ directory, not in the same directory as the Makefile. This is easily fixed by a change to the setup.py file in yt/utilities. I will issue a PR for this depending on the discussion of the following issue.
The not-simple change is that the newest Forthon setup.py is hard-coded to install the Forthon package for the system Python, regardless of which actual Python stack you're using, which is not convenient for our custom-curated yt stack. The fix is to simply remove the hard-coding. But I want to ask y'all what the best course forward from here is: A) ask the author to change it, B) make the change in the Forthon linked to in the install_script, or C) stick to the old version for now?
Thanks!
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
to be specific (I must have accidentally deleted the language because I thought I had it in my original message) the Forthon install bug only happens in Darwin.
I'm going to email the author, suggesting that he add a check to see if the system Python is being used before installing in the system site-packages. I'll let y'all know.
Hi all,
I have corresponded with the author, David Grote, and he has updated Forthon to 0.8.10 and included a fix that checks to see if the system Python is being used, and if so, uses the hard-coding, and if not, it doesn't. In my tests it seems to work fine.
http://hifweb.lbl.gov/Forthon/Forthon-0.8.10.tgz
I will shortly issue a PR for the needed fixes for Forthon stuff in yt.
Hi Stephen,
Thanks for taking care of that -- this sounds like the best possible fix.
Casey was asking about which components use Forthon. My understanding is that it's only an indirect hard requirement, and that the things that use it are the merger tree you wrote (not the one that works on Enzo-FOF output), the two point functions, and the pHOP halo finder. Is that right?
-Matt
On Tue, Aug 7, 2012 at 6:58 PM, Stephen Skory s@skory.us wrote:
Hi all,
I have corresponded with the author, David Grote, and he has updated Forthon to 0.8.10 and included a fix that checks to see if the system Python is being used, and if so, uses the hard-coding, and if not, it doesn't. In my tests it seems to work fine.
http://hifweb.lbl.gov/Forthon/Forthon-0.8.10.tgz
I will shortly issue a PR for the needed fixes for Forthon stuff in yt.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Matt,
Casey was asking about which components use Forthon. My understanding is that it's only an indirect hard requirement, and that the things that use it are the merger tree you wrote (not the one that works on Enzo-FOF output), the two point functions, and the pHOP halo finder. Is that right?
Ah, yes, I forgot to answer that. You're correct, Matt, those are the places that the Fortran tree is used. In all honesty, the scipy.spatial kDTree is good enough (in terms of speed) for all but the halo finding, so that can someday be swapped out for the Fortran one. Maybe I'll issue myself a ticket...
Hey Stephen and Matt.
Thanks for letting me know. Yeah, too bad the scipy version isn't up to the performance we need.
- Casey
On Tue, Aug 7, 2012 at 4:10 PM, Stephen Skory s@skory.us wrote:
Hi Matt,
Casey was asking about which components use Forthon. My understanding is that it's only an indirect hard requirement, and that the things that use it are the merger tree you wrote (not the one that works on Enzo-FOF output), the two point functions, and the pHOP halo finder. Is that right?
Ah, yes, I forgot to answer that. You're correct, Matt, those are the places that the Fortran tree is used. In all honesty, the scipy.spatial kDTree is good enough (in terms of speed) for all but the halo finding, so that can someday be swapped out for the Fortran one. Maybe I'll issue myself a ticket...
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Stephen,
On Tue, Aug 7, 2012 at 4:03 PM, Stephen Skory s@skory.us wrote:
Hi all,
Some problems Sam was having with the fortan kdtree has alerted me to some necessary changes due to changes in Forthon. Two of them are simple. One is to remove a --with-numpy flag in the Makefile. The second is that the most recent version of Forthon (0.8.9) builds the fKDpy.so object in it's own build/lib.arch.../ directory, not in the same directory as the Makefile. This is easily fixed by a change to the setup.py file in yt/utilities. I will issue a PR for this depending on the discussion of the following issue.
While I initially thought this would work, I'm now seeing issues with how this interacts with using yt in develop mode, which is by far the most common. I don't believe it is importable anymore from a "develop" environment. One option I could see is adding a new target to the Makefile, or change the Makefile to use --builddir=. . What do you think?
-Matt
-Matt
The not-simple change is that the newest Forthon setup.py is hard-coded to install the Forthon package for the system Python, regardless of which actual Python stack you're using, which is not convenient for our custom-curated yt stack. The fix is to simply remove the hard-coding. But I want to ask y'all what the best course forward from here is: A) ask the author to change it, B) make the change in the Forthon linked to in the install_script, or C) stick to the old version for now?
Thanks!
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Matt,
While I initially thought this would work, I'm now seeing issues with how this interacts with using yt in develop mode, which is by far the most common. I don't believe it is importable anymore from a "develop" environment. One option I could see is adding a new target to the Makefile, or change the Makefile to use --builddir=. . What do you think?
The --builddir= option doesn't seem to put the fKDpy.so where --builddir specifies.
How about a line in the Makefile that moves the .so file back where we have always had it after compilation, and return setup.py to how it was before my PR? I've just tried this and it appears to work.
On Tue, Aug 7, 2012 at 11:11 PM, Stephen Skory s@skory.us wrote:
Hi Matt,
While I initially thought this would work, I'm now seeing issues with how this interacts with using yt in develop mode, which is by far the most common. I don't believe it is importable anymore from a "develop" environment. One option I could see is adding a new target to the Makefile, or change the Makefile to use --builddir=. . What do you think?
The --builddir= option doesn't seem to put the fKDpy.so where --builddir specifies.
How about a line in the Makefile that moves the .so file back where we have always had it after compilation, and return setup.py to how it was before my PR? I've just tried this and it appears to work.
Sounds great to me. I think the ideal solution would be integrating Forthon into setup.py, but sicne that's probably tricky and difficult, I think this is a good way for now.
-- Stephen Skory s@skory.us http://stephenskory.com/ 510.621.3687 (google voice) _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (4)
-
Casey W. Stark -
Geoffrey So -
Matthew Turk -
Stephen Skory