Options parsing in the Tornado Web Server
I remember seeing a bit of discussion about improving Python's default options parsing... Tornado's seems very impressive [1]:
- Andrey 1. http://github.com/facebook/tornado/blob/9a8bd2fb6fd6279be16d6f0a2e57e49fe1b9...
That's not particularly more expressive than what optparse gives you today, except for the use of a function with a side effect on a magic global (how the heck did options.port suddenly get a value?!) and the potential for confusing tornado.options with tornado.options.options. On Thu, Sep 10, 2009 at 12:50 PM, Andrey Fedorov <anfedorov@gmail.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Well, options.*port* corresponds to define("*port*", ...). Is it considered unPythonic to equate variable names and strings? I didn't think it was, since scopes are dictionaries... Good point about options.options. - Andrey On Thu, Sep 10, 2009 at 4:44 PM, Guido van Rossum <guido@python.org> wrote:
Andrey Fedorov wrote:
It's the fact that there is an options global in the module at all which can be surprising. Application global objects like that aren't necessarily bad, but they aren't necessarily good either. optparse uses independent parsers by default, leaving applications free to put the options information wherever they want (e.g. merging it with settings coming from system and per-user configuration files and storing the results in a myapp.settings module) An approach like the tornado example that provides its own global parser better also have its own mechanism for producing additional independent parsers if it ever hopes to match the features of optparse. Aside from the presence of that global parser, I'm not seeing a lot difference between options.define and parser.add_option though. Anyone wanting to replace/compete with optparse (particularly with goals for latter standard library inclusion) would do well to better articulate what they don't like about optparse though. I acknowledge that using optparse the first couple of times can have something of a learning curve, but that's because it is rather powerful. And if there are features that appear to be missing, then why not suggest those as optparse enhancements rather than trying to replace the module wholesale? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Note that argparse does provide that justification: http://argparse.googlecode.com/svn/trunk/doc/argparse-vs-optparse.html In particular: http://argparse.googlecode.com/svn/trunk/doc/argparse-vs-optparse.html#upgra... has a short blurb on why Steven didn't just extend optparse. He tried to, but decided it was easier if he didn't. Eric.
Just so people know, we started discussing over on the stdlib-sig the idea of trying to convince Steven Bethard to contribute argparse to the standard library as a way to improve the argument parsing situation in the standard library. On Thu, Sep 10, 2009 at 13:44, Guido van Rossum <guido@python.org> wrote:
I am Bret from FriendFeed, author of a lot of Tornado. We modeled it after the Google option parsing (http://code.google.com/ p/google-gflags/). The main distinction is that every module declares its own options, so your main() function doesn't need to be aware of all of the options used by the transitive closure of modules in your server. As anyone who has worked on large systems knows, passing around options and defaults becomes a big pain after your number of modules increases above 100 or so. That said, I think our options parsing works well for individual projects, but you would get lots of naming conflicts if it were adopted in any official capacity by Python given the options all have global scope, so I agree with Guido that optparse is probably better as an official module. It certainly was much more useful to us than optparse from an operational standpoint, though. Bret On Sep 10, 1:44 pm, Guido van Rossum <gu...@python.org> wrote:
Hey Bret! I still see your name in our code base a lot... :-) The Google flags code has a fundamentally different use case than the typical argument parsing -- thanks for pointing this out. In fact, the two use cases are so different that there is barely any overlap. (What Google does with flags is more typically done with environment variables, although I totally understand that that didn't work for you.) Maybe the discussion about flags parsing (which is apparently happening on a list I'm not on :-) is helped by clearly distinguishing the two styles. --Guido On Thu, Sep 10, 2009 at 2:29 PM, Bret Taylor <btaylor@gmail.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Thu, Sep 10, 2009 at 6:42 PM, Guido van Rossum <guido@python.org> wrote:
Other useful notes about the Google flag systems (most of which sounds like it applies to Tornado's system as well), for those who haven't used it: - Google's flag system is used primarily for configuring binaries, rather than command-line option parsing; it just happens to take the form of command-line options. - Accordingly, having a single global options dict is useful for configuring all the different libraries that get linked into a single binary. Most of the flags a given binary exposes come from these libraries, or libraries used by other libraries, etc. - When defining flags in libraries, you have to manually namespace them (mylibrary_rpc_deadline_secs, yourlibrary_rpc_deadline_secs, etc) to avoid collisions. - The flags system is designed to operate across languages: a Python application can define some flags, and that application may use a C++ extension module which defines more flags, and both are configured in the same place. Based on that, I'm not sure that a gflags-like system would be a good replacement for command-line parsing. In fact, gflags sometimes requires unexpected/unusual command-line ordering if you try to use it like a general option parser. Other people who've used gflags may have a different perspective. Collin Winter
That's not particularly more expressive than what optparse gives you today, except for the use of a function with a side effect on a magic global (how the heck did options.port suddenly get a value?!) and the potential for confusing tornado.options with tornado.options.options. On Thu, Sep 10, 2009 at 12:50 PM, Andrey Fedorov <anfedorov@gmail.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
Well, options.*port* corresponds to define("*port*", ...). Is it considered unPythonic to equate variable names and strings? I didn't think it was, since scopes are dictionaries... Good point about options.options. - Andrey On Thu, Sep 10, 2009 at 4:44 PM, Guido van Rossum <guido@python.org> wrote:
Andrey Fedorov wrote:
It's the fact that there is an options global in the module at all which can be surprising. Application global objects like that aren't necessarily bad, but they aren't necessarily good either. optparse uses independent parsers by default, leaving applications free to put the options information wherever they want (e.g. merging it with settings coming from system and per-user configuration files and storing the results in a myapp.settings module) An approach like the tornado example that provides its own global parser better also have its own mechanism for producing additional independent parsers if it ever hopes to match the features of optparse. Aside from the presence of that global parser, I'm not seeing a lot difference between options.define and parser.add_option though. Anyone wanting to replace/compete with optparse (particularly with goals for latter standard library inclusion) would do well to better articulate what they don't like about optparse though. I acknowledge that using optparse the first couple of times can have something of a learning curve, but that's because it is rather powerful. And if there are features that appear to be missing, then why not suggest those as optparse enhancements rather than trying to replace the module wholesale? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Note that argparse does provide that justification: http://argparse.googlecode.com/svn/trunk/doc/argparse-vs-optparse.html In particular: http://argparse.googlecode.com/svn/trunk/doc/argparse-vs-optparse.html#upgra... has a short blurb on why Steven didn't just extend optparse. He tried to, but decided it was easier if he didn't. Eric.
Just so people know, we started discussing over on the stdlib-sig the idea of trying to convince Steven Bethard to contribute argparse to the standard library as a way to improve the argument parsing situation in the standard library. On Thu, Sep 10, 2009 at 13:44, Guido van Rossum <guido@python.org> wrote:
I am Bret from FriendFeed, author of a lot of Tornado. We modeled it after the Google option parsing (http://code.google.com/ p/google-gflags/). The main distinction is that every module declares its own options, so your main() function doesn't need to be aware of all of the options used by the transitive closure of modules in your server. As anyone who has worked on large systems knows, passing around options and defaults becomes a big pain after your number of modules increases above 100 or so. That said, I think our options parsing works well for individual projects, but you would get lots of naming conflicts if it were adopted in any official capacity by Python given the options all have global scope, so I agree with Guido that optparse is probably better as an official module. It certainly was much more useful to us than optparse from an operational standpoint, though. Bret On Sep 10, 1:44 pm, Guido van Rossum <gu...@python.org> wrote:
Hey Bret! I still see your name in our code base a lot... :-) The Google flags code has a fundamentally different use case than the typical argument parsing -- thanks for pointing this out. In fact, the two use cases are so different that there is barely any overlap. (What Google does with flags is more typically done with environment variables, although I totally understand that that didn't work for you.) Maybe the discussion about flags parsing (which is apparently happening on a list I'm not on :-) is helped by clearly distinguishing the two styles. --Guido On Thu, Sep 10, 2009 at 2:29 PM, Bret Taylor <btaylor@gmail.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Thu, Sep 10, 2009 at 6:42 PM, Guido van Rossum <guido@python.org> wrote:
Other useful notes about the Google flag systems (most of which sounds like it applies to Tornado's system as well), for those who haven't used it: - Google's flag system is used primarily for configuring binaries, rather than command-line option parsing; it just happens to take the form of command-line options. - Accordingly, having a single global options dict is useful for configuring all the different libraries that get linked into a single binary. Most of the flags a given binary exposes come from these libraries, or libraries used by other libraries, etc. - When defining flags in libraries, you have to manually namespace them (mylibrary_rpc_deadline_secs, yourlibrary_rpc_deadline_secs, etc) to avoid collisions. - The flags system is designed to operate across languages: a Python application can define some flags, and that application may use a C++ extension module which defines more flags, and both are configured in the same place. Based on that, I'm not sure that a gflags-like system would be a good replacement for command-line parsing. In fact, gflags sometimes requires unexpected/unusual command-line ordering if you try to use it like a general option parser. Other people who've used gflags may have a different perspective. Collin Winter
participants (7)
-
Andrey Fedorov -
Bret Taylor -
Brett Cannon -
Collin Winter -
Eric Smith -
Guido van Rossum -
Nick Coghlan