Running yt in parallel on a subset of available processes
Is there a way to restrict yt to only use a subset of the available processes when it's running in parallel? I have a set of ipython engines running, and I'm trying to launch a yt script on a subset of them, but it just hangs. It works fine if I launch the same script on all the processes. Printing out MPI.COMM_WORLD.Get_size() from the script gives the total number of engines, not the size of the subset I'm trying to use. I haven't done a lot of parallel programming, but I would guess that the running processes are waiting on one that's not running a yt script, since they can see the total number of processes available, but not the size of the subset of processes that are actually running a yt script. Does anyone have any thoughts on how to do this? Thanks, Dan
Hi Dan,
I've done this before by modifying how the global communicator is
created. The way they work in yt is through pushing new communicators
onto a stack, which can vary based on which processors you're on. I
don't think I ever wrote it up in a PR, but I think it could be done
without too much work.
On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn
Is there a way to restrict yt to only use a subset of the available processes when it's running in parallel? I have a set of ipython engines running, and I'm trying to launch a yt script on a subset of them, but it just hangs. It works fine if I launch the same script on all the processes.
Printing out MPI.COMM_WORLD.Get_size() from the script gives the total number of engines, not the size of the subset I'm trying to use.
I haven't done a lot of parallel programming, but I would guess that the running processes are waiting on one that's not running a yt script, since they can see the total number of processes available, but not the size of the subset of processes that are actually running a yt script.
Does anyone have any thoughts on how to do this?
Thanks,
Dan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Matt,
I was able to get my MPI problem fixed by using subcommunicators. I
modified enable_parallelism() to take an optional argument that specifies
the MPI communicator yt should use. If none is specified, it defaults to
COMM_WORLD. Seems to be working pretty well so far.
Thanks for the advice,
Dan
2014-12-03 19:16 GMT-05:00 Matthew Turk
Hi Dan,
I've done this before by modifying how the global communicator is created. The way they work in yt is through pushing new communicators onto a stack, which can vary based on which processors you're on. I don't think I ever wrote it up in a PR, but I think it could be done without too much work.
Is there a way to restrict yt to only use a subset of the available processes when it's running in parallel? I have a set of ipython engines running, and I'm trying to launch a yt script on a subset of them, but it just hangs. It works fine if I launch the same script on all the
On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn
wrote: processes. Printing out MPI.COMM_WORLD.Get_size() from the script gives the total number of engines, not the size of the subset I'm trying to use.
I haven't done a lot of parallel programming, but I would guess that the running processes are waiting on one that's not running a yt script,
since
they can see the total number of processes available, but not the size of the subset of processes that are actually running a yt script.
Does anyone have any thoughts on how to do this?
Thanks,
Dan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hey Daniel,
This is great that you got this working, and it sounds like something from
which others could benefit! Would you mind submitting this change to the
main codebase in a pull request?
Cameron
On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn
Hi Matt,
I was able to get my MPI problem fixed by using subcommunicators. I modified enable_parallelism() to take an optional argument that specifies the MPI communicator yt should use. If none is specified, it defaults to COMM_WORLD. Seems to be working pretty well so far.
Thanks for the advice,
Dan
2014-12-03 19:16 GMT-05:00 Matthew Turk
: Hi Dan,
I've done this before by modifying how the global communicator is created. The way they work in yt is through pushing new communicators onto a stack, which can vary based on which processors you're on. I don't think I ever wrote it up in a PR, but I think it could be done without too much work.
Is there a way to restrict yt to only use a subset of the available processes when it's running in parallel? I have a set of ipython engines running, and I'm trying to launch a yt script on a subset of them, but it just hangs. It works fine if I launch the same script on all the
On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn
wrote: processes. Printing out MPI.COMM_WORLD.Get_size() from the script gives the total number of engines, not the size of the subset I'm trying to use.
I haven't done a lot of parallel programming, but I would guess that the running processes are waiting on one that's not running a yt script,
since
they can see the total number of processes available, but not the size of the subset of processes that are actually running a yt script.
Does anyone have any thoughts on how to do this?
Thanks,
Dan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
Hi Cameron,
Yeah, no problem. I'm not exactly sure how to do that. Do I have to fork yt
first, or can I use my existing installation since it's already using
Mercurial? I'm familiar with subversion, since that's what we use in our
research group, but I'm a bit hazy on how the whole thing is set up with
yt. When I installed yt using the installation script, it created a yt-hg
directory. Is that a checked-out version of the code that I can commit?
Thanks!
Dan
2014-12-03 20:59 GMT-05:00 Cameron Hummels
Hey Daniel,
This is great that you got this working, and it sounds like something from which others could benefit! Would you mind submitting this change to the main codebase in a pull request?
Cameron
On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn
wrote: Hi Matt,
I was able to get my MPI problem fixed by using subcommunicators. I modified enable_parallelism() to take an optional argument that specifies the MPI communicator yt should use. If none is specified, it defaults to COMM_WORLD. Seems to be working pretty well so far.
Thanks for the advice,
Dan
2014-12-03 19:16 GMT-05:00 Matthew Turk
: Hi Dan,
I've done this before by modifying how the global communicator is created. The way they work in yt is through pushing new communicators onto a stack, which can vary based on which processors you're on. I don't think I ever wrote it up in a PR, but I think it could be done without too much work.
Is there a way to restrict yt to only use a subset of the available processes when it's running in parallel? I have a set of ipython engines running, and I'm trying to launch a yt script on a subset of them, but it just hangs. It works fine if I launch the same script on all the
On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn
wrote: processes. Printing out MPI.COMM_WORLD.Get_size() from the script gives the total number of engines, not the size of the subset I'm trying to use.
I haven't done a lot of parallel programming, but I would guess that
the
running processes are waiting on one that's not running a yt script, since they can see the total number of processes available, but not the size of the subset of processes that are actually running a yt script.
Does anyone have any thoughts on how to do this?
Thanks,
Dan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
Hi Dan,
Yup, you can commit in that directory, and push to a fork. Create a
fork on bitbucket, then do your changes and commit, and issue this
command:
hg push https://bitbucket.org/yourusername/yt
Then you can issue a pull request via their web interface. You can
also edit your hgrc to add an alias, but that can happen later.
-Matt
On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn
Hi Cameron,
Yeah, no problem. I'm not exactly sure how to do that. Do I have to fork yt first, or can I use my existing installation since it's already using Mercurial? I'm familiar with subversion, since that's what we use in our research group, but I'm a bit hazy on how the whole thing is set up with yt. When I installed yt using the installation script, it created a yt-hg directory. Is that a checked-out version of the code that I can commit?
Thanks!
Dan
2014-12-03 20:59 GMT-05:00 Cameron Hummels
: Hey Daniel,
This is great that you got this working, and it sounds like something from which others could benefit! Would you mind submitting this change to the main codebase in a pull request?
Cameron
On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn
wrote: Hi Matt,
I was able to get my MPI problem fixed by using subcommunicators. I modified enable_parallelism() to take an optional argument that specifies the MPI communicator yt should use. If none is specified, it defaults to COMM_WORLD. Seems to be working pretty well so far.
Thanks for the advice,
Dan
2014-12-03 19:16 GMT-05:00 Matthew Turk
: Hi Dan,
I've done this before by modifying how the global communicator is created. The way they work in yt is through pushing new communicators onto a stack, which can vary based on which processors you're on. I don't think I ever wrote it up in a PR, but I think it could be done without too much work.
On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn
wrote: Is there a way to restrict yt to only use a subset of the available processes when it's running in parallel? I have a set of ipython engines running, and I'm trying to launch a yt script on a subset of them, but it just hangs. It works fine if I launch the same script on all the processes.
Printing out MPI.COMM_WORLD.Get_size() from the script gives the total number of engines, not the size of the subset I'm trying to use.
I haven't done a lot of parallel programming, but I would guess that the running processes are waiting on one that's not running a yt script, since they can see the total number of processes available, but not the size of the subset of processes that are actually running a yt script.
Does anyone have any thoughts on how to do this?
Thanks,
Dan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
There is also some discussion on how to do it in the developer
documentation:
http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing...
Thanks for your hard work on this, and don't hesitate to ask for more help!
Cameron
On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk
Hi Dan,
Yup, you can commit in that directory, and push to a fork. Create a fork on bitbucket, then do your changes and commit, and issue this command:
hg push https://bitbucket.org/yourusername/yt
Then you can issue a pull request via their web interface. You can also edit your hgrc to add an alias, but that can happen later.
-Matt
On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn
wrote: Hi Cameron,
Yeah, no problem. I'm not exactly sure how to do that. Do I have to fork yt first, or can I use my existing installation since it's already using Mercurial? I'm familiar with subversion, since that's what we use in our research group, but I'm a bit hazy on how the whole thing is set up with yt. When I installed yt using the installation script, it created a yt-hg directory. Is that a checked-out version of the code that I can commit?
Thanks!
Dan
2014-12-03 20:59 GMT-05:00 Cameron Hummels
: Hey Daniel,
This is great that you got this working, and it sounds like something from which others could benefit! Would you mind submitting this change to the main codebase in a pull request?
Cameron
On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn
wrote: Hi Matt,
I was able to get my MPI problem fixed by using subcommunicators. I modified enable_parallelism() to take an optional argument that
specifies
the MPI communicator yt should use. If none is specified, it defaults to COMM_WORLD. Seems to be working pretty well so far.
Thanks for the advice,
Dan
2014-12-03 19:16 GMT-05:00 Matthew Turk
: Hi Dan,
I've done this before by modifying how the global communicator is created. The way they work in yt is through pushing new communicators onto a stack, which can vary based on which processors you're on. I don't think I ever wrote it up in a PR, but I think it could be done without too much work.
On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn
wrote: Is there a way to restrict yt to only use a subset of the available processes when it's running in parallel? I have a set of ipython engines running, and I'm trying to launch a yt script on a subset of them, but it just hangs. It works fine if I launch the same script on all the processes.
Printing out MPI.COMM_WORLD.Get_size() from the script gives the total number of engines, not the size of the subset I'm trying to use.
I haven't done a lot of parallel programming, but I would guess that the running processes are waiting on one that's not running a yt script, since they can see the total number of processes available, but not the size of the subset of processes that are actually running a yt script.
Does anyone have any thoughts on how to do this?
Thanks,
Dan
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
Ok, thanks for the help--that ended up being pretty easy. Something else
I'm confused about is why the same files appear in two different
directories. There's the hg-yt directory in YT_DIR/src, and then there's
YT_DIR/yt, which seems to be the exact same thing. What's the difference
between them?
Another question--will the fork that I created be updated with future
changes to the code, or do I have to manually update it, and, if so, how?
Thanks for your help,
Dan
2014-12-04 11:23 GMT-05:00 Cameron Hummels
There is also some discussion on how to do it in the developer documentation:
http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing...
Thanks for your hard work on this, and don't hesitate to ask for more help!
Cameron
On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk
wrote: Hi Dan,
Yup, you can commit in that directory, and push to a fork. Create a fork on bitbucket, then do your changes and commit, and issue this command:
hg push https://bitbucket.org/yourusername/yt
Then you can issue a pull request via their web interface. You can also edit your hgrc to add an alias, but that can happen later.
-Matt
Hi Cameron,
Yeah, no problem. I'm not exactly sure how to do that. Do I have to fork yt first, or can I use my existing installation since it's already using Mercurial? I'm familiar with subversion, since that's what we use in our research group, but I'm a bit hazy on how the whole thing is set up with yt. When I installed yt using the installation script, it created a yt-hg directory. Is that a checked-out version of the code that I can commit?
Thanks!
Dan
2014-12-03 20:59 GMT-05:00 Cameron Hummels
: Hey Daniel,
This is great that you got this working, and it sounds like something from which others could benefit! Would you mind submitting this change to
main codebase in a pull request?
Cameron
On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn
wrote: Hi Matt,
I was able to get my MPI problem fixed by using subcommunicators. I modified enable_parallelism() to take an optional argument that
specifies
the MPI communicator yt should use. If none is specified, it defaults to COMM_WORLD. Seems to be working pretty well so far.
Thanks for the advice,
Dan
2014-12-03 19:16 GMT-05:00 Matthew Turk
: Hi Dan,
I've done this before by modifying how the global communicator is created. The way they work in yt is through pushing new communicators onto a stack, which can vary based on which processors you're on. I don't think I ever wrote it up in a PR, but I think it could be done without too much work.
On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn
wrote: > Is there a way to restrict yt to only use a subset of the available > processes when it's running in parallel? I have a set of ipython > engines > running, and I'm trying to launch a yt script on a subset of them, but > it > just hangs. It works fine if I launch the same script on all the > processes. > > Printing out MPI.COMM_WORLD.Get_size() from the script gives the total > number of engines, not the size of the subset I'm trying to use. > > I haven't done a lot of parallel programming, but I would guess > the > running processes are waiting on one that's not running a yt
On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn
wrote: the that script, > since > they can see the total number of processes available, but not the size > of > the subset of processes that are actually running a yt script. > > Does anyone have any thoughts on how to do this? > > Thanks, > > Dan > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > _______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Thu Dec 04 2014 at 4:23:06 PM Daniel Fenn
Ok, thanks for the help--that ended up being pretty easy. Something else I'm confused about is why the same files appear in two different directories. There's the hg-yt directory in YT_DIR/src,
The yt-hg directory is a clone of the yt mercurial repository. It gets created by the install script.
and then there's YT_DIR/yt, which seems to be the exact same thing. What's the difference between them?
I don't have this directory in my install script build. Is there a chance you created this directory at some point?
Another question--will the fork that I created be updated with future changes to the code, or do I have to manually update it, and, if so, how?
You will need to manually keep it up to date. You can do this by clicking the "sync now" link on the overview page for your fork on the bitbucket web interface, or by manually pushing changes to your fork using "hg push". You can configure mercurial so that your default push location is your fork, see "hg help paths". That said, I don't think it's important to keep your fork up to date. I generally have my paths setup so I pull directly from the main yt_analysis/yt repository and then push to my fork only when I want to issue pull requests. If your fork is out of date, you when you push the changes for your pull request, you will also push the changes that will bring it up to date.
Thanks for your help,
Dan
2014-12-04 11:23 GMT-05:00 Cameron Hummels
: There is also some discussion on how to do it in the developer
documentation:
http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing...
Thanks for your hard work on this, and don't hesitate to ask for more help!
Cameron
On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk
wrote: Hi Dan,
Yup, you can commit in that directory, and push to a fork. Create a fork on bitbucket, then do your changes and commit, and issue this command:
hg push https://bitbucket.org/yourusername/yt
Then you can issue a pull request via their web interface. You can also edit your hgrc to add an alias, but that can happen later.
-Matt
Hi Cameron,
Yeah, no problem. I'm not exactly sure how to do that. Do I have to fork yt first, or can I use my existing installation since it's already using Mercurial? I'm familiar with subversion, since that's what we use in our research group, but I'm a bit hazy on how the whole thing is set up with yt. When I installed yt using the installation script, it created a yt-hg directory. Is that a checked-out version of the code that I can commit?
Thanks!
Dan
2014-12-03 20:59 GMT-05:00 Cameron Hummels
: Hey Daniel,
This is great that you got this working, and it sounds like something from which others could benefit! Would you mind submitting this change to
main codebase in a pull request?
Cameron
On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn
wrote: Hi Matt,
I was able to get my MPI problem fixed by using subcommunicators. I modified enable_parallelism() to take an optional argument that
specifies
the MPI communicator yt should use. If none is specified, it defaults to COMM_WORLD. Seems to be working pretty well so far.
Thanks for the advice,
Dan
2014-12-03 19:16 GMT-05:00 Matthew Turk
: > Hi Dan, > > I've done this before by modifying how the global communicator is > created. The way they work in yt is through pushing new communicators > onto a stack, which can vary based on which processors you're on. I > don't think I ever wrote it up in a PR, but I think it could be done > without too much work. > > On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn
wrote: > > Is there a way to restrict yt to only use a subset of the available > > processes when it's running in parallel? I have a set of ipython > > engines > > running, and I'm trying to launch a yt script on a subset of > > it > > just hangs. It works fine if I launch the same script on all the > > processes. > > > > Printing out MPI.COMM_WORLD.Get_size() from the script gives the total > > number of engines, not the size of the subset I'm trying to use. > > > > I haven't done a lot of parallel programming, but I would guess
> > the > > running processes are waiting on one that's not running a yt
On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn
wrote: the them, but that script, > > since > > they can see the total number of processes available, but not the size > > of > > the subset of processes that are actually running a yt script. > > > > Does anyone have any thoughts on how to do this? > > > > Thanks, > > > > Dan > > > > _______________________________________________ > > yt-users mailing list > > yt-users@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
There's a good chance I created that directory at some point. That cuts
down on my confusion a good deal. One final question--I'm trying to test
out some of the recent changes that were made--specifically pull requests
#1328 and #1331. I thought I would just have to do hg update in the hg-yt
directory.
The update worked fine, and it now says I'm at version 3.1-dev, but when I
check the log, I don't see any mention of those pull requests. Did I miss
them (very possible), or have they not been added to the main repository? I
notice that they both still say "open". If that's the case, is there a way
that I can download the changed files?
Thanks for taking the time to answer all my questions.
Dan
2014-12-04 19:30 GMT-05:00 Nathan Goldbaum
On Thu Dec 04 2014 at 4:23:06 PM Daniel Fenn
wrote: Ok, thanks for the help--that ended up being pretty easy. Something else I'm confused about is why the same files appear in two different directories. There's the hg-yt directory in YT_DIR/src,
The yt-hg directory is a clone of the yt mercurial repository. It gets created by the install script.
and then there's YT_DIR/yt, which seems to be the exact same thing. What's the difference between them?
I don't have this directory in my install script build. Is there a chance you created this directory at some point?
Another question--will the fork that I created be updated with future changes to the code, or do I have to manually update it, and, if so, how?
You will need to manually keep it up to date. You can do this by clicking the "sync now" link on the overview page for your fork on the bitbucket web interface, or by manually pushing changes to your fork using "hg push". You can configure mercurial so that your default push location is your fork, see "hg help paths".
That said, I don't think it's important to keep your fork up to date. I generally have my paths setup so I pull directly from the main yt_analysis/yt repository and then push to my fork only when I want to issue pull requests. If your fork is out of date, you when you push the changes for your pull request, you will also push the changes that will bring it up to date.
Thanks for your help,
Dan
2014-12-04 11:23 GMT-05:00 Cameron Hummels
: There is also some discussion on how to do it in the developer
documentation:
http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing...
Thanks for your hard work on this, and don't hesitate to ask for more help!
Cameron
On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk
wrote: Hi Dan,
Yup, you can commit in that directory, and push to a fork. Create a fork on bitbucket, then do your changes and commit, and issue this command:
hg push https://bitbucket.org/yourusername/yt
Then you can issue a pull request via their web interface. You can also edit your hgrc to add an alias, but that can happen later.
-Matt
Hi Cameron,
Yeah, no problem. I'm not exactly sure how to do that. Do I have to fork yt first, or can I use my existing installation since it's already using Mercurial? I'm familiar with subversion, since that's what we use in our research group, but I'm a bit hazy on how the whole thing is set up with yt. When I installed yt using the installation script, it created a yt-hg directory. Is that a checked-out version of the code that I can commit?
Thanks!
Dan
2014-12-03 20:59 GMT-05:00 Cameron Hummels
: Hey Daniel,
This is great that you got this working, and it sounds like something from which others could benefit! Would you mind submitting this change to
main codebase in a pull request?
Cameron
On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn
wrote: > > Hi Matt, > > I was able to get my MPI problem fixed by using subcommunicators. I > modified enable_parallelism() to take an optional argument that specifies > the MPI communicator yt should use. If none is specified, it defaults to > COMM_WORLD. Seems to be working pretty well so far. > > Thanks for the advice, > > Dan > > 2014-12-03 19:16 GMT-05:00 Matthew Turk : > >> Hi Dan, >> >> I've done this before by modifying how the global communicator is >> created. The way they work in yt is through pushing new communicators >> onto a stack, which can vary based on which processors you're on. I >> don't think I ever wrote it up in a PR, but I think it could be done >> without too much work. >> >> On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn wrote: >> > Is there a way to restrict yt to only use a subset of the available >> > processes when it's running in parallel? I have a set of ipython >> > engines >> > running, and I'm trying to launch a yt script on a subset of >> > it >> > just hangs. It works fine if I launch the same script on all the >> > processes. >> > >> > Printing out MPI.COMM_WORLD.Get_size() from the script gives the total >> > number of engines, not the size of the subset I'm trying to use. >> > >> > I haven't done a lot of parallel programming, but I would guess
>> > the >> > running processes are waiting on one that's not running a yt
>> > since >> > they can see the total number of processes available, but not
On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn
wrote: the them, but that script, the size >> > of >> > the subset of processes that are actually running a yt script. >> > >> > Does anyone have any thoughts on how to do this? >> > >> > Thanks, >> > >> > Dan >> > >> > _______________________________________________ >> > yt-users mailing list >> > yt-users@lists.spacepope.org >> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org > > > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
On Thu Dec 04 2014 at 5:08:18 PM Daniel Fenn
There's a good chance I created that directory at some point. That cuts down on my confusion a good deal. One final question--I'm trying to test out some of the recent changes that were made--specifically pull requests #1328 and #1331. I thought I would just have to do hg update in the hg-yt directory.
The update worked fine, and it now says I'm at version 3.1-dev, but when I check the log, I don't see any mention of those pull requests. Did I miss them (very possible), or have they not been added to the main repository? I notice that they both still say "open". If that's the case, is there a way that I can download the changed files?
To test a PR, you're going to need to pull it in locally from the issuer's
fork of yt, then update to the tip of their PR. I'm looking at our
documentation, and I don't think we cover this specific workflow, so this
is something that should probably be added.
Here what I do. Go to the pull request page on bitbucket, and click over
to the commits tab. I.e. this page for PR 1328:
https://bitbucket.org/yt_analysis/yt/pull-request/1328/delay-running-callbac...
That lists all of the changesets that are part of the pull request. You're
going to need to copy/paste the changeset hash for the tip of the pull
request, in this case it's c62d7d0.
Next, in your local clone of yt, do:
$ hg pull -r c62d7d0 https://bitbucket.org/ngoldbaum/yt
Note that I'm pulling from the PR issuer's repository. Pull requests are
not part of the main repository until they have been merged.
Next, just in case any compiled code changes (i.e. a cython .pyx file), run:
$ python setup.py develop
It's good to get into the habit of doing this, since you will run into
weird behavior if any compiled files changed and they didn't get recompiled.
Now the changeset in the tip of the pull request is the "active" change.
You can test the pull request.
To do back to "mainline" yt, you can issue the following command:
hg update
Thanks for taking the time to answer all my questions.
Dan
2014-12-04 19:30 GMT-05:00 Nathan Goldbaum
: On Thu Dec 04 2014 at 4:23:06 PM Daniel Fenn
wrote: Ok, thanks for the help--that ended up being pretty easy. Something else I'm confused about is why the same files appear in two different directories. There's the hg-yt directory in YT_DIR/src,
The yt-hg directory is a clone of the yt mercurial repository. It gets created by the install script.
and then there's YT_DIR/yt, which seems to be the exact same thing. What's the difference between them?
I don't have this directory in my install script build. Is there a chance you created this directory at some point?
Another question--will the fork that I created be updated with future changes to the code, or do I have to manually update it, and, if so, how?
You will need to manually keep it up to date. You can do this by clicking the "sync now" link on the overview page for your fork on the bitbucket web interface, or by manually pushing changes to your fork using "hg push". You can configure mercurial so that your default push location is your fork, see "hg help paths".
That said, I don't think it's important to keep your fork up to date. I generally have my paths setup so I pull directly from the main yt_analysis/yt repository and then push to my fork only when I want to issue pull requests. If your fork is out of date, you when you push the changes for your pull request, you will also push the changes that will bring it up to date.
Thanks for your help,
Dan
2014-12-04 11:23 GMT-05:00 Cameron Hummels
: There is also some discussion on how to do it in the developer
documentation:
http://yt-project.org/docs/dev/developing/developing.html#making-and-sharing...
Thanks for your hard work on this, and don't hesitate to ask for more help!
Cameron
On Thu, Dec 4, 2014 at 8:05 AM, Matthew Turk
wrote: Hi Dan,
Yup, you can commit in that directory, and push to a fork. Create a fork on bitbucket, then do your changes and commit, and issue this command:
hg push https://bitbucket.org/yourusername/yt
Then you can issue a pull request via their web interface. You can also edit your hgrc to add an alias, but that can happen later.
-Matt
Hi Cameron,
Yeah, no problem. I'm not exactly sure how to do that. Do I have to fork yt first, or can I use my existing installation since it's already using Mercurial? I'm familiar with subversion, since that's what we use in our research group, but I'm a bit hazy on how the whole thing is set up with yt. When I installed yt using the installation script, it created a yt-hg directory. Is that a checked-out version of the code that I can commit?
Thanks!
Dan
2014-12-03 20:59 GMT-05:00 Cameron Hummels
: > Hey Daniel, > > This is great that you got this working, and it sounds like something from > which others could benefit! Would you mind submitting this change to the > main codebase in a pull request? > > Cameron > > On Wed, Dec 3, 2014 at 6:46 PM, Daniel Fenn
wrote: >> >> Hi Matt, >> >> I was able to get my MPI problem fixed by using subcommunicators. I >> modified enable_parallelism() to take an optional argument that specifies >> the MPI communicator yt should use. If none is specified, it defaults to >> COMM_WORLD. Seems to be working pretty well so far. >> >> Thanks for the advice, >> >> Dan >> >> 2014-12-03 19:16 GMT-05:00 Matthew Turk : >> >>> Hi Dan, >>> >>> I've done this before by modifying how the global communicator is >>> created. The way they work in yt is through pushing new communicators >>> onto a stack, which can vary based on which processors you're on. I >>> don't think I ever wrote it up in a PR, but I think it could be done >>> without too much work. >>> >>> On Wed, Dec 3, 2014 at 11:47 AM, Daniel Fenn wrote: >>> > Is there a way to restrict yt to only use a subset of the available >>> > processes when it's running in parallel? I have a set of ipython >>> > engines >>> > running, and I'm trying to launch a yt script on a subset of >>> > it >>> > just hangs. It works fine if I launch the same script on all the >>> > processes. >>> > >>> > Printing out MPI.COMM_WORLD.Get_size() from the script gives
>>> > number of engines, not the size of the subset I'm trying to use. >>> > >>> > I haven't done a lot of parallel programming, but I would guess
>>> > the >>> > running processes are waiting on one that's not running a yt
>>> > since >>> > they can see the total number of processes available, but not
On Thu, Dec 4, 2014 at 8:08 AM, Daniel Fenn
wrote: them, but the total that script, the size >>> > of >>> > the subset of processes that are actually running a yt script. >>> > >>> > Does anyone have any thoughts on how to do this? >>> > >>> > Thanks, >>> > >>> > Dan >>> > >>> > _______________________________________________ >>> > yt-users mailing list >>> > yt-users@lists.spacepope.org >>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >>> > >>> _______________________________________________ >>> yt-users mailing list >>> yt-users@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> >> >> >> _______________________________________________ >> yt-users mailing list >> yt-users@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >> > > > > -- > Cameron Hummels > Postdoctoral Researcher > Steward Observatory > University of Arizona > http://chummels.org > > _______________________________________________ > yt-users mailing list > yt-users@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org >
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-- Cameron Hummels Postdoctoral Researcher Steward Observatory University of Arizona http://chummels.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________ yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
_______________________________________________
yt-users mailing list yt-users@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
participants (4)
-
Cameron Hummels
-
Daniel Fenn
-
Matthew Turk
-
Nathan Goldbaum