Hi Dave, The key thing here is the sys.argv access. On Fri, Dec 2, 2011 at 11:53 AM, david collins wrote:

Hey, Matt--

I'm a little slow, so let me make sure I understand what your suggestions are, or at least put it into context that I use.

Say I have a dumb script that does

<dumb> import sys pf = load(sys.argv[1]) pc = PlotCollection(pf,center=[0.5]*3) pc.add_projection("ClownDensity",axis=2) pc.save("Clown2PerCm2") </dumb>

I would typically do something like "mpirun -np 4013 python2.7 dumb.py path --parallel"

For option 1, use of this would only work if I called "mpirun -np 4013 yt run dumb.py path --parallel" instead

Yes.

For option 2, this wouldn't work at all?

Accessing sys.args wouldn't work in the way you currently have it. However, we could add a "leftovers" or something variable to the yt.mods namespace that includes all positional arguments not already parsed. I just tested and this does work.

For option 3, this would work as "normal",

Yes, but if you mistyped "--paralleeel" it would not fail (nor does it fail now.)

and option 4 I'd have to put "path" inside of dumb.py?

Yes. Talking this through leads me to think we might be able to have a combination of 4 and 2. I just tested and what we *can* do is actually provide the parser arguments as above, but allow for a "leftovers" section for all *positional* arguments to the script. This would mean for --something and -s arguments you would need to add those to the parser object, but for things like what you do above, you wouldn't. Would that work for your use case? -Matt

Additionally, if I use optparse, similar outcomes?

d.

On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk wrote:

...

Hi all,

I have prepared a Pull Request to change how yt processes arguments to scripts.  I just issued it, but I am emailing because I think discussion of what it does warrants a bit more public hashing out. The PR is not done yet, for the reasons I outline below, so please don't anybody accept it yet.

https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-...

This will directly affect you if you have:

1) Ever written "from yt.config import ytcfg; ytcfg[...." 2) Ever put your *own* command-line parser into yt. 3) Gotten annoyed with configuration files.

What I've done is create a new file, startup_tasks.py, that gets imported whenever yt.mods gets imported, and only the first time that happens.  It sets up an argument parser (using argparse, which is Python 2.7 only) that parses looking for:

--parallel --paste --paste-detailed --detailed --rpdb --parallel

One of the things this does is that it also provides --help, so you can see what is available.  Furthermore, I've added a --config option, so that from the command line you can set configuration options.  For instance:

--config serialize=False

and so on.  This is pretty cool I think and will go a long way toward making things nicer.  However, the way this works is still up for a few more problems.  There are basically two ways this can work:

 * Parse the entirety of sys.args and accept all arguments that yt finds, rejecting and throwing an error on unrecognized ones (i.e., typos or things you might pass in to a script your write on the command line).  This will be an exclusive operation.  * Parse *non-exclusively*, allowing unrecognized arguments to pass through.  However, the old arguments will still be there: so any script that has issues with things like --parallel and whatnot will now see there, whereas it did not before because yt (totally un-cool!) stripped them out of the sys.args variable.  I don't want to do this anymore.

The way I have implemented this for the yt command line tool is to set a flag that says, "We're also inside the command line, so don't parse anything, we'll handle adding new options to the parser and then we'll parse everything at the end."  This way you can pass both --parallel and whatever option the yt command line utility wants.  This works because startup_tasks creates a "parser" object, adds arguments to that parser object, then delays actually conducting the parsing until all the arguments from teh command line tool have been added.

There are four ways this can work.  I have presented them in order of my increasing preference.  (Coincidentally, on the astropy mailing list they discussed this this week, as I was thinking about my feelings on it as well, and they are moving away from parsing args in the library; I think that works for them because AstroPy is designed to be used much more inside larger frameworks, whereas yt is somewhat more insular.) 1) Don't do any argument parsing if not called through a yt-specific script runner.  This means if you want to pass --parallel, you have to run with something like "yt run my_script.py --parallel".  Same for --config and so on. 2) Parse all arguments any time yt.mods is imported, do not allow for additional arguments.  This breaks scripts that have their own parsing. 3) Parse *some* of the arguments, but not all.  All typos would succeed and this could lead to confusion for the user. 4) Provide a yt-specific mechanism for adding new arguments.  So if you want to add new arguments, you do it at the top of your script, rather than the bottom, and at the bottom inside the construction "if __name__ == '__main__'" you'd inspect the values.

Anyway, I'm inclined to go for #4, simply because it would be the simplest mechanism for ensuring an explicit method of getting arguments into user-written scripts.

Thoughts?

-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Hi Dave, On Fri, Dec 2, 2011 at 1:39 PM, david collins wrote:

...

Talking this through leads me to think we might be able to have a combination of 4 and 2.  I just tested and what we *can* do is actually provide the parser arguments as above, but allow for a "leftovers" section for all *positional* arguments to the script. This would mean for --something and -s arguments you would need to add those to the parser object, but for things like what you do above, you wouldn't.

Would that work for your use case?

So is option 4 something like

<myscript> import yt if __name__ == "__main__":   yt_option_parser.add_option("--clown_hair_size_threshold")   do some stuff using the clow hair size </myscript>

like a modified/subclassed optparse?  Then my command line is like mpirun -np 16127 python myscript --clown_hair 14 --parallel ?

Precisely.

In your "leftovers" version of Option 2, could that just modify sys.argv in place, pop-ing out the bits that yt wants or wants to raise an error on?  Not strictly necessary, but it would mean a little less modification of old scripts on my part.  (Though frankly, scripts where I use sys.argv directly should be thrown in the toilet, so forcing me to re-write those tools to use some less fragile option parser is probably good for me.)

What we could do is have it pop everything out, then replacing sys.argv with the contents of the leftovers. I don't really like that options. What I think I like more is providing this "leftovers" list (which can be looked at similarly to sys.args) to yt.mods.

Anyhow, as long as it still retains the ability for me to pass in non-yt options to the script, I'm happy.  And there are certainly bigger concerns here than my army of dumb scripts.  The first version of option 2 wouldn't really do that, but the 'leftovers' version would.  I think I like your new 2 + 4.

Okay. Then I'll aim for that. Does anyone else have any feelings on this? -Matt

d.

...

-Matt

...

Additionally, if I use optparse, similar outcomes?

d.

On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk wrote:

...

Hi all,

I have prepared a Pull Request to change how yt processes arguments to scripts.  I just issued it, but I am emailing because I think discussion of what it does warrants a bit more public hashing out. The PR is not done yet, for the reasons I outline below, so please don't anybody accept it yet.

https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-...

This will directly affect you if you have:

1) Ever written "from yt.config import ytcfg; ytcfg[...." 2) Ever put your *own* command-line parser into yt. 3) Gotten annoyed with configuration files.

What I've done is create a new file, startup_tasks.py, that gets imported whenever yt.mods gets imported, and only the first time that happens.  It sets up an argument parser (using argparse, which is Python 2.7 only) that parses looking for:

--parallel --paste --paste-detailed --detailed --rpdb --parallel

One of the things this does is that it also provides --help, so you can see what is available.  Furthermore, I've added a --config option, so that from the command line you can set configuration options.  For instance:

--config serialize=False

and so on.  This is pretty cool I think and will go a long way toward making things nicer.  However, the way this works is still up for a few more problems.  There are basically two ways this can work:

 * Parse the entirety of sys.args and accept all arguments that yt finds, rejecting and throwing an error on unrecognized ones (i.e., typos or things you might pass in to a script your write on the command line).  This will be an exclusive operation.  * Parse *non-exclusively*, allowing unrecognized arguments to pass through.  However, the old arguments will still be there: so any script that has issues with things like --parallel and whatnot will now see there, whereas it did not before because yt (totally un-cool!) stripped them out of the sys.args variable.  I don't want to do this anymore.

The way I have implemented this for the yt command line tool is to set a flag that says, "We're also inside the command line, so don't parse anything, we'll handle adding new options to the parser and then we'll parse everything at the end."  This way you can pass both --parallel and whatever option the yt command line utility wants.  This works because startup_tasks creates a "parser" object, adds arguments to that parser object, then delays actually conducting the parsing until all the arguments from teh command line tool have been added.

There are four ways this can work.  I have presented them in order of my increasing preference.  (Coincidentally, on the astropy mailing list they discussed this this week, as I was thinking about my feelings on it as well, and they are moving away from parsing args in the library; I think that works for them because AstroPy is designed to be used much more inside larger frameworks, whereas yt is somewhat more insular.) 1) Don't do any argument parsing if not called through a yt-specific script runner.  This means if you want to pass --parallel, you have to run with something like "yt run my_script.py --parallel".  Same for --config and so on. 2) Parse all arguments any time yt.mods is imported, do not allow for additional arguments.  This breaks scripts that have their own parsing. 3) Parse *some* of the arguments, but not all.  All typos would succeed and this could lead to confusion for the user. 4) Provide a yt-specific mechanism for adding new arguments.  So if you want to add new arguments, you do it at the top of your script, rather than the bottom, and at the bottom inside the construction "if __name__ == '__main__'" you'd inspect the values.

Anyway, I'm inclined to go for #4, simply because it would be the simplest mechanism for ensuring an explicit method of getting arguments into user-written scripts.

Thoughts?

-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

That's the big problem, yes. I would be fine with yt taking over things like "--help" and whatnot, which are reserved. But I would also note that we're actually *changing* the functionality, not *removing* it if we go with 2+4. It says, if you want your script to be able to run in parallel yt, or to use these yt commands, here's how you interface with the yt argument parsing. On Fri, Dec 2, 2011 at 2:51 PM, Britton Smith wrote:

I think I still prefer option 3.  I'm in favor of making the system of parsing yt args cleaner, but options 1, 2, and 4 feel to me like we're saying in order to use yt, you have to trade in some python functionality that you've been using.  Is the only reason we wouldn't go with option 3 that it will not catch typos in flags that we intended to be for yt?

Britton

On Fri, Dec 2, 2011 at 1:39 PM, david collins wrote:

...

...

Talking this through leads me to think we might be able to have a combination of 4 and 2.  I just tested and what we *can* do is actually provide the parser arguments as above, but allow for a "leftovers" section for all *positional* arguments to the script. This would mean for --something and -s arguments you would need to add those to the parser object, but for things like what you do above, you wouldn't.

Would that work for your use case?

So is option 4 something like

<myscript> import yt if __name__ == "__main__":   yt_option_parser.add_option("--clown_hair_size_threshold")   do some stuff using the clow hair size </myscript>

like a modified/subclassed optparse?  Then my command line is like mpirun -np 16127 python myscript --clown_hair 14 --parallel ?

In your "leftovers" version of Option 2, could that just modify sys.argv in place, pop-ing out the bits that yt wants or wants to raise an error on?  Not strictly necessary, but it would mean a little less modification of old scripts on my part.  (Though frankly, scripts where I use sys.argv directly should be thrown in the toilet, so forcing me to re-write those tools to use some less fragile option parser is probably good for me.)

Anyhow, as long as it still retains the ability for me to pass in non-yt options to the script, I'm happy.  And there are certainly bigger concerns here than my army of dumb scripts.  The first version of option 2 wouldn't really do that, but the 'leftovers' version would.  I think I like your new 2 + 4.

d.

...

-Matt

...

Additionally, if I use optparse, similar outcomes?

d.

On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk wrote:

...

Hi all,

I have prepared a Pull Request to change how yt processes arguments to scripts.  I just issued it, but I am emailing because I think discussion of what it does warrants a bit more public hashing out. The PR is not done yet, for the reasons I outline below, so please don't anybody accept it yet.

https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-...

This will directly affect you if you have:

1) Ever written "from yt.config import ytcfg; ytcfg[...." 2) Ever put your *own* command-line parser into yt. 3) Gotten annoyed with configuration files.

What I've done is create a new file, startup_tasks.py, that gets imported whenever yt.mods gets imported, and only the first time that happens.  It sets up an argument parser (using argparse, which is Python 2.7 only) that parses looking for:

--parallel --paste --paste-detailed --detailed --rpdb --parallel

One of the things this does is that it also provides --help, so you can see what is available.  Furthermore, I've added a --config option, so that from the command line you can set configuration options.  For instance:

--config serialize=False

and so on.  This is pretty cool I think and will go a long way toward making things nicer.  However, the way this works is still up for a few more problems.  There are basically two ways this can work:

 * Parse the entirety of sys.args and accept all arguments that yt finds, rejecting and throwing an error on unrecognized ones (i.e., typos or things you might pass in to a script your write on the command line).  This will be an exclusive operation.  * Parse *non-exclusively*, allowing unrecognized arguments to pass through.  However, the old arguments will still be there: so any script that has issues with things like --parallel and whatnot will now see there, whereas it did not before because yt (totally un-cool!) stripped them out of the sys.args variable.  I don't want to do this anymore.

The way I have implemented this for the yt command line tool is to set a flag that says, "We're also inside the command line, so don't parse anything, we'll handle adding new options to the parser and then we'll parse everything at the end."  This way you can pass both --parallel and whatever option the yt command line utility wants.  This works because startup_tasks creates a "parser" object, adds arguments to that parser object, then delays actually conducting the parsing until all the arguments from teh command line tool have been added.

There are four ways this can work.  I have presented them in order of my increasing preference.  (Coincidentally, on the astropy mailing list they discussed this this week, as I was thinking about my feelings on it as well, and they are moving away from parsing args in the library; I think that works for them because AstroPy is designed to be used much more inside larger frameworks, whereas yt is somewhat more insular.) 1) Don't do any argument parsing if not called through a yt-specific script runner.  This means if you want to pass --parallel, you have to run with something like "yt run my_script.py --parallel".  Same for --config and so on. 2) Parse all arguments any time yt.mods is imported, do not allow for additional arguments.  This breaks scripts that have their own parsing. 3) Parse *some* of the arguments, but not all.  All typos would succeed and this could lead to confusion for the user. 4) Provide a yt-specific mechanism for adding new arguments.  So if you want to add new arguments, you do it at the top of your script, rather than the bottom, and at the bottom inside the construction "if __name__ == '__main__'" you'd inspect the values.

Anyway, I'm inclined to go for #4, simply because it would be the simplest mechanism for ensuring an explicit method of getting arguments into user-written scripts.

Thoughts?

-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

something like yt.argv.  That way, anyone who has been pull arguments from the command line can simply replace sys.argv with yt.argv, or something like that.  As long as the substitution is intuitive, people won't have to go parsing through source or documentation when they go back to an old script and find it doesn't work anymore.

If we go with 2+4 and "leftovers," what would be the actual behavior of from yt.mods import * import sys.argv print sys.argv ? Would that still give me the same thing as before, and additionally create yt.optparse and yt.leftovers, with yt getting its info from these features? Or will sys.argv get replaced with something else? d.

Britton

On Fri, Dec 2, 2011 at 2:57 PM, Matthew Turk wrote:

...

That's the big problem, yes.  I would be fine with yt taking over things like "--help" and whatnot, which are reserved.  But I would also note that we're actually *changing* the functionality, not *removing* it if we go with 2+4.  It says, if you want your script to be able to run in parallel yt, or to use these yt commands, here's how you interface with the yt argument parsing.

On Fri, Dec 2, 2011 at 2:51 PM, Britton Smith wrote:

...

I think I still prefer option 3.  I'm in favor of making the system of parsing yt args cleaner, but options 1, 2, and 4 feel to me like we're saying in order to use yt, you have to trade in some python functionality that you've been using.  Is the only reason we wouldn't go with option 3 that it will not catch typos in flags that we intended to be for yt?

Britton

On Fri, Dec 2, 2011 at 1:39 PM, david collins wrote:

...

...

Talking this through leads me to think we might be able to have a combination of 4 and 2.  I just tested and what we *can* do is actually provide the parser arguments as above, but allow for a "leftovers" section for all *positional* arguments to the script. This would mean for --something and -s arguments you would need to add those to the parser object, but for things like what you do above, you wouldn't.

Would that work for your use case?

So is option 4 something like

<myscript> import yt if __name__ == "__main__":   yt_option_parser.add_option("--clown_hair_size_threshold")   do some stuff using the clow hair size </myscript>

like a modified/subclassed optparse?  Then my command line is like mpirun -np 16127 python myscript --clown_hair 14 --parallel ?

In your "leftovers" version of Option 2, could that just modify sys.argv in place, pop-ing out the bits that yt wants or wants to raise an error on?  Not strictly necessary, but it would mean a little less modification of old scripts on my part.  (Though frankly, scripts where I use sys.argv directly should be thrown in the toilet, so forcing me to re-write those tools to use some less fragile option parser is probably good for me.)

Anyhow, as long as it still retains the ability for me to pass in non-yt options to the script, I'm happy.  And there are certainly bigger concerns here than my army of dumb scripts.  The first version of option 2 wouldn't really do that, but the 'leftovers' version would.  I think I like your new 2 + 4.

d.

...

-Matt

...

Additionally, if I use optparse, similar outcomes?

d.

On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk wrote: > Hi all, > > I have prepared a Pull Request to change how yt processes arguments > to > scripts.  I just issued it, but I am emailing because I think > discussion of what it does warrants a bit more public hashing out. > The PR is not done yet, for the reasons I outline below, so please > don't anybody accept it yet. > > > > https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-... > > This will directly affect you if you have: > > 1) Ever written "from yt.config import ytcfg; ytcfg[...." > 2) Ever put your *own* command-line parser into yt. > 3) Gotten annoyed with configuration files. > > What I've done is create a new file, startup_tasks.py, that gets > imported whenever yt.mods gets imported, and only the first time > that > happens.  It sets up an argument parser (using argparse, which is > Python 2.7 only) that parses looking for: > > --parallel > --paste > --paste-detailed > --detailed > --rpdb > --parallel > > One of the things this does is that it also provides --help, so you > can see what is available.  Furthermore, I've added a --config > option, > so that from the command line you can set configuration options. >  For > instance: > > --config serialize=False > > and so on.  This is pretty cool I think and will go a long way > toward > making things nicer.  However, the way this works is still up for a > few more problems.  There are basically two ways this can work: > >  * Parse the entirety of sys.args and accept all arguments that yt > finds, rejecting and throwing an error on unrecognized ones (i.e., > typos or things you might pass in to a script your write on the > command line).  This will be an exclusive operation. >  * Parse *non-exclusively*, allowing unrecognized arguments to pass > through.  However, the old arguments will still be there: so any > script that has issues with things like --parallel and whatnot will > now see there, whereas it did not before because yt (totally > un-cool!) > stripped them out of the sys.args variable.  I don't want to do > this > anymore. > > The way I have implemented this for the yt command line tool is to > set > a flag that says, "We're also inside the command line, so don't > parse > anything, we'll handle adding new options to the parser and then > we'll > parse everything at the end."  This way you can pass both > --parallel > and whatever option the yt command line utility wants.  This works > because startup_tasks creates a "parser" object, adds arguments to > that parser object, then delays actually conducting the parsing > until > all the arguments from teh command line tool have been added. > > There are four ways this can work.  I have presented them in order > of > my increasing preference.  (Coincidentally, on the astropy mailing > list they discussed this this week, as I was thinking about my > feelings on it as well, and they are moving away from parsing args > in > the library; I think that works for them because AstroPy is > designed > to be used much more inside larger frameworks, whereas yt is > somewhat > more insular.) > 1) Don't do any argument parsing if not called through a > yt-specific > script runner.  This means if you want to pass --parallel, you have > to > run with something like "yt run my_script.py --parallel".  Same for > --config and so on. > 2) Parse all arguments any time yt.mods is imported, do not allow > for > additional arguments.  This breaks scripts that have their own > parsing. > 3) Parse *some* of the arguments, but not all.  All typos would > succeed and this could lead to confusion for the user. > 4) Provide a yt-specific mechanism for adding new arguments.  So if > you want to add new arguments, you do it at the top of your script, > rather than the bottom, and at the bottom inside the construction > "if > __name__ == '__main__'" you'd inspect the values. > > Anyway, I'm inclined to go for #4, simply because it would be the > simplest mechanism for ensuring an explicit method of getting > arguments into user-written scripts. > > Thoughts? > > -Matt > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer.

It'll have everything in it, including options that have been parsed by the option parser.

Perfect. I'm on board with all that you've proposed. d.

...

d.

...

Britton

On Fri, Dec 2, 2011 at 2:57 PM, Matthew Turk wrote:

...

That's the big problem, yes.  I would be fine with yt taking over things like "--help" and whatnot, which are reserved.  But I would also note that we're actually *changing* the functionality, not *removing* it if we go with 2+4.  It says, if you want your script to be able to run in parallel yt, or to use these yt commands, here's how you interface with the yt argument parsing.

On Fri, Dec 2, 2011 at 2:51 PM, Britton Smith wrote:

...

I think I still prefer option 3.  I'm in favor of making the system of parsing yt args cleaner, but options 1, 2, and 4 feel to me like we're saying in order to use yt, you have to trade in some python functionality that you've been using.  Is the only reason we wouldn't go with option 3 that it will not catch typos in flags that we intended to be for yt?

Britton

On Fri, Dec 2, 2011 at 1:39 PM, david collins wrote:

...

> Talking this through leads me to think we might be able to have a > combination of 4 and 2.  I just tested and what we *can* do is > actually provide the parser arguments as above, but allow for a > "leftovers" section for all *positional* arguments to the script. > This would mean for --something and -s arguments you would need to > add > those to the parser object, but for things like what you do above, > you > wouldn't. > > Would that work for your use case?

So is option 4 something like

<myscript> import yt if __name__ == "__main__":   yt_option_parser.add_option("--clown_hair_size_threshold")   do some stuff using the clow hair size </myscript>

like a modified/subclassed optparse?  Then my command line is like mpirun -np 16127 python myscript --clown_hair 14 --parallel ?

In your "leftovers" version of Option 2, could that just modify sys.argv in place, pop-ing out the bits that yt wants or wants to raise an error on?  Not strictly necessary, but it would mean a little less modification of old scripts on my part.  (Though frankly, scripts where I use sys.argv directly should be thrown in the toilet, so forcing me to re-write those tools to use some less fragile option parser is probably good for me.)

Anyhow, as long as it still retains the ability for me to pass in non-yt options to the script, I'm happy.  And there are certainly bigger concerns here than my army of dumb scripts.  The first version of option 2 wouldn't really do that, but the 'leftovers' version would.  I think I like your new 2 + 4.

d.

> > -Matt > >> >> Additionally, if I use optparse, similar outcomes? >> >> d. >> >> >> On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk >> wrote: >>> Hi all, >>> >>> I have prepared a Pull Request to change how yt processes arguments >>> to >>> scripts.  I just issued it, but I am emailing because I think >>> discussion of what it does warrants a bit more public hashing out. >>> The PR is not done yet, for the reasons I outline below, so please >>> don't anybody accept it yet. >>> >>> >>> >>> https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-... >>> >>> This will directly affect you if you have: >>> >>> 1) Ever written "from yt.config import ytcfg; ytcfg[...." >>> 2) Ever put your *own* command-line parser into yt. >>> 3) Gotten annoyed with configuration files. >>> >>> What I've done is create a new file, startup_tasks.py, that gets >>> imported whenever yt.mods gets imported, and only the first time >>> that >>> happens.  It sets up an argument parser (using argparse, which is >>> Python 2.7 only) that parses looking for: >>> >>> --parallel >>> --paste >>> --paste-detailed >>> --detailed >>> --rpdb >>> --parallel >>> >>> One of the things this does is that it also provides --help, so you >>> can see what is available.  Furthermore, I've added a --config >>> option, >>> so that from the command line you can set configuration options. >>>  For >>> instance: >>> >>> --config serialize=False >>> >>> and so on.  This is pretty cool I think and will go a long way >>> toward >>> making things nicer.  However, the way this works is still up for a >>> few more problems.  There are basically two ways this can work: >>> >>>  * Parse the entirety of sys.args and accept all arguments that yt >>> finds, rejecting and throwing an error on unrecognized ones (i.e., >>> typos or things you might pass in to a script your write on the >>> command line).  This will be an exclusive operation. >>>  * Parse *non-exclusively*, allowing unrecognized arguments to pass >>> through.  However, the old arguments will still be there: so any >>> script that has issues with things like --parallel and whatnot will >>> now see there, whereas it did not before because yt (totally >>> un-cool!) >>> stripped them out of the sys.args variable.  I don't want to do >>> this >>> anymore. >>> >>> The way I have implemented this for the yt command line tool is to >>> set >>> a flag that says, "We're also inside the command line, so don't >>> parse >>> anything, we'll handle adding new options to the parser and then >>> we'll >>> parse everything at the end."  This way you can pass both >>> --parallel >>> and whatever option the yt command line utility wants.  This works >>> because startup_tasks creates a "parser" object, adds arguments to >>> that parser object, then delays actually conducting the parsing >>> until >>> all the arguments from teh command line tool have been added. >>> >>> There are four ways this can work.  I have presented them in order >>> of >>> my increasing preference.  (Coincidentally, on the astropy mailing >>> list they discussed this this week, as I was thinking about my >>> feelings on it as well, and they are moving away from parsing args >>> in >>> the library; I think that works for them because AstroPy is >>> designed >>> to be used much more inside larger frameworks, whereas yt is >>> somewhat >>> more insular.) >>> 1) Don't do any argument parsing if not called through a >>> yt-specific >>> script runner.  This means if you want to pass --parallel, you have >>> to >>> run with something like "yt run my_script.py --parallel".  Same for >>> --config and so on. >>> 2) Parse all arguments any time yt.mods is imported, do not allow >>> for >>> additional arguments.  This breaks scripts that have their own >>> parsing. >>> 3) Parse *some* of the arguments, but not all.  All typos would >>> succeed and this could lead to confusion for the user. >>> 4) Provide a yt-specific mechanism for adding new arguments.  So if >>> you want to add new arguments, you do it at the top of your script, >>> rather than the bottom, and at the bottom inside the construction >>> "if >>> __name__ == '__main__'" you'd inspect the values. >>> >>> Anyway, I'm inclined to go for #4, simply because it would be the >>> simplest mechanism for ensuring an explicit method of getting >>> arguments into user-written scripts. >>> >>> Thoughts? >>> >>> -Matt >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> >> -- >> Sent from my computer. >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer.

Hi all, I have revised my pull request: https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-... This now includes all the arguments in sys.argv, including the arguments parsed out by the internal yt system. The arguments remaining are all inside yt.startup_tasks.leftover_args. There's a line included that shows how to switch to the behavior where we modify sys.argv. I have come around a bit and I think this would be okay, since we have been doing it for some time. The new system is much easier to modify, and it also allows us to specify configuration options on teh command line using --config serialize=False for instance. If somebody wouldn't mind pulling this down and testing it out, that'd be great. Works for me okay though, and I like the new look of the system -- you can now actually get help on the arguments! -Matt On Fri, Dec 2, 2011 at 4:08 PM, j s oishi wrote:

Agreed. I am also on board.

j

On Fri, Dec 2, 2011 at 1:03 PM, david collins wrote:

...

...

It'll have everything in it, including options that have been parsed by the option parser.

Perfect.  I'm on board with all that you've proposed.

d.

...

...

d.

...

Britton

On Fri, Dec 2, 2011 at 2:57 PM, Matthew Turk wrote:

...

That's the big problem, yes.  I would be fine with yt taking over things like "--help" and whatnot, which are reserved.  But I would also note that we're actually *changing* the functionality, not *removing* it if we go with 2+4.  It says, if you want your script to be able to run in parallel yt, or to use these yt commands, here's how you interface with the yt argument parsing.

On Fri, Dec 2, 2011 at 2:51 PM, Britton Smith wrote: > I think I still prefer option 3.  I'm in favor of making the system of > parsing yt args cleaner, but options 1, 2, and 4 feel to me like we're > saying in order to use yt, you have to trade in some python > functionality > that you've been using.  Is the only reason we wouldn't go with option 3 > that it will not catch typos in flags that we intended to be for yt? > > Britton > > On Fri, Dec 2, 2011 at 1:39 PM, david collins > wrote: >> >> > Talking this through leads me to think we might be able to have a >> > combination of 4 and 2.  I just tested and what we *can* do is >> > actually provide the parser arguments as above, but allow for a >> > "leftovers" section for all *positional* arguments to the script. >> > This would mean for --something and -s arguments you would need to >> > add >> > those to the parser object, but for things like what you do above, >> > you >> > wouldn't. >> > >> > Would that work for your use case? >> >> So is option 4 something like >> >> <myscript> >> import yt >> if __name__ == "__main__": >>   yt_option_parser.add_option("--clown_hair_size_threshold") >>   do some stuff using the clow hair size >> </myscript> >> >> like a modified/subclassed optparse?  Then my command line is like >> mpirun -np 16127 python myscript --clown_hair 14 --parallel >> ? >> >> In your "leftovers" version of Option 2, could that just modify >> sys.argv in place, pop-ing out the bits that yt wants or wants to >> raise an error on?  Not strictly necessary, but it would mean a little >> less modification of old scripts on my part.  (Though frankly, scripts >> where I use sys.argv directly should be thrown in the toilet, so >> forcing me to re-write those tools to use some less fragile option >> parser is probably good for me.) >> >> Anyhow, as long as it still retains the ability for me to pass in >> non-yt options to the script, I'm happy.  And there are certainly >> bigger concerns here than my army of dumb scripts.  The first version >> of option >> 2 wouldn't really do that, but the 'leftovers' version would.  I think >> I like your new 2 + 4. >> >> d. >> >> >> > >> > -Matt >> > >> >> >> >> Additionally, if I use optparse, similar outcomes? >> >> >> >> d. >> >> >> >> >> >> On Fri, Dec 2, 2011 at 9:30 AM, Matthew Turk >> >> wrote: >> >>> Hi all, >> >>> >> >>> I have prepared a Pull Request to change how yt processes arguments >> >>> to >> >>> scripts.  I just issued it, but I am emailing because I think >> >>> discussion of what it does warrants a bit more public hashing out. >> >>> The PR is not done yet, for the reasons I outline below, so please >> >>> don't anybody accept it yet. >> >>> >> >>> >> >>> >> >>> https://bitbucket.org/yt_analysis/yt/pull-request/38/overhaul-configuration-... >> >>> >> >>> This will directly affect you if you have: >> >>> >> >>> 1) Ever written "from yt.config import ytcfg; ytcfg[...." >> >>> 2) Ever put your *own* command-line parser into yt. >> >>> 3) Gotten annoyed with configuration files. >> >>> >> >>> What I've done is create a new file, startup_tasks.py, that gets >> >>> imported whenever yt.mods gets imported, and only the first time >> >>> that >> >>> happens.  It sets up an argument parser (using argparse, which is >> >>> Python 2.7 only) that parses looking for: >> >>> >> >>> --parallel >> >>> --paste >> >>> --paste-detailed >> >>> --detailed >> >>> --rpdb >> >>> --parallel >> >>> >> >>> One of the things this does is that it also provides --help, so you >> >>> can see what is available.  Furthermore, I've added a --config >> >>> option, >> >>> so that from the command line you can set configuration options. >> >>>  For >> >>> instance: >> >>> >> >>> --config serialize=False >> >>> >> >>> and so on.  This is pretty cool I think and will go a long way >> >>> toward >> >>> making things nicer.  However, the way this works is still up for a >> >>> few more problems.  There are basically two ways this can work: >> >>> >> >>>  * Parse the entirety of sys.args and accept all arguments that yt >> >>> finds, rejecting and throwing an error on unrecognized ones (i.e., >> >>> typos or things you might pass in to a script your write on the >> >>> command line).  This will be an exclusive operation. >> >>>  * Parse *non-exclusively*, allowing unrecognized arguments to pass >> >>> through.  However, the old arguments will still be there: so any >> >>> script that has issues with things like --parallel and whatnot will >> >>> now see there, whereas it did not before because yt (totally >> >>> un-cool!) >> >>> stripped them out of the sys.args variable.  I don't want to do >> >>> this >> >>> anymore. >> >>> >> >>> The way I have implemented this for the yt command line tool is to >> >>> set >> >>> a flag that says, "We're also inside the command line, so don't >> >>> parse >> >>> anything, we'll handle adding new options to the parser and then >> >>> we'll >> >>> parse everything at the end."  This way you can pass both >> >>> --parallel >> >>> and whatever option the yt command line utility wants.  This works >> >>> because startup_tasks creates a "parser" object, adds arguments to >> >>> that parser object, then delays actually conducting the parsing >> >>> until >> >>> all the arguments from teh command line tool have been added. >> >>> >> >>> There are four ways this can work.  I have presented them in order >> >>> of >> >>> my increasing preference.  (Coincidentally, on the astropy mailing >> >>> list they discussed this this week, as I was thinking about my >> >>> feelings on it as well, and they are moving away from parsing args >> >>> in >> >>> the library; I think that works for them because AstroPy is >> >>> designed >> >>> to be used much more inside larger frameworks, whereas yt is >> >>> somewhat >> >>> more insular.) >> >>> 1) Don't do any argument parsing if not called through a >> >>> yt-specific >> >>> script runner.  This means if you want to pass --parallel, you have >> >>> to >> >>> run with something like "yt run my_script.py --parallel".  Same for >> >>> --config and so on. >> >>> 2) Parse all arguments any time yt.mods is imported, do not allow >> >>> for >> >>> additional arguments.  This breaks scripts that have their own >> >>> parsing. >> >>> 3) Parse *some* of the arguments, but not all.  All typos would >> >>> succeed and this could lead to confusion for the user. >> >>> 4) Provide a yt-specific mechanism for adding new arguments.  So if >> >>> you want to add new arguments, you do it at the top of your script, >> >>> rather than the bottom, and at the bottom inside the construction >> >>> "if >> >>> __name__ == '__main__'" you'd inspect the values. >> >>> >> >>> Anyway, I'm inclined to go for #4, simply because it would be the >> >>> simplest mechanism for ensuring an explicit method of getting >> >>> arguments into user-written scripts. >> >>> >> >>> Thoughts? >> >>> >> >>> -Matt >> >>> _______________________________________________ >> >>> yt-dev mailing list >> >>> yt-dev@lists.spacepope.org >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> >> >> >> >> >> -- >> >> Sent from my computer. >> >> _______________________________________________ >> >> yt-dev mailing list >> >> yt-dev@lists.spacepope.org >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > _______________________________________________ >> > yt-dev mailing list >> > yt-dev@lists.spacepope.org >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> >> -- >> Sent from my computer. >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-- Sent from my computer. _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

---

yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org