Possible context managers in stdlib
Hi,
I compiled a list of some possible new context managers that could be added to the stdlib. Introducing a new feature should IMO also show usage of it in the distribution itself. That wasn't done with decorators (a decorators module is compiled at the moment, if I'm right), but with context managers, there's certainly room to add some. Of course, my list is excessive, it's only some ideas I got by flying over the stdlib docs.
* builtins: with open/file * sys: with sys.redirected_std[in|out|err] with sys.trace with sys.excepthook * warnings: with warnings.filter * pprint: with pprint.printer (used for print) * codecs: with codecs.open with codecs.EncodedFile * decimal: with decimal.Context * os: with os.current_directory with os.modified_env with os.umask/uid/gid etc. with os.popen with os.tmpfile * subprocess: with subprocess.Popen * mutex: with mutexobj * curses: with curses.wrapper * tempfile: with tempfile.[Named]TemporaryFile etc. * locale: with locale.differentlocale * logging: with logging.ignored with logging.Logger * signal: with signal.handler * socket: with socket.socket * threading: with threading.Lock with threading.Condition with threading.Event * mmap: with mmap.mmap * bz2/zipfile/tarfile: with ...open * dl: with dl.open * tty: with tty.raw with tty.cbreak * cgitb: with cgitb.enabled * urllib/urllib2: with urllib.urlopen etc. * httplib: with httplib.HTTPConnection * ftplib: with ftplib.FTP * poplib: with poplib.POP3 * imaplib: with imaplib.IMAP4 * nttplib: with nntplib.NNTP * smtplib: with smtplib.SMTP * telnetlib: with telnetlib.Telnet * xmlrpclib: with xmlrpclib.ServerProxy
Reinhold
On 7/8/05, Reinhold Birkenfeld reinhold-birkenfeld-nospam@wolke7.net wrote:
Hi,
I compiled a list of some possible new context managers that could be added to the stdlib. Introducing a new feature should IMO also show usage of it in the distribution itself. That wasn't done with decorators (a decorators module is compiled at the moment, if I'm right), but with context managers, there's certainly room to add some. Of course, my list is excessive, it's only some ideas I got by flying over the stdlib docs.
I think that is a good point about decorators. While none of us obviously had extensive experience with decorators when they went into the language we at least knew of a couple use cases that would have been handy and it probably would have been good to have examples for people to work off of. I bet iterators have gotten more play thanks to itertools and all of the built-ins that work with them.
I think having basic context managers in a stdlib module that we know for a fact that will be handy is a good idea. We should keep the list short and poignant, but we should have something for people to work off of. The ones I like below for a 'context' module are:
- builtins: with open/file
- sys: with sys.redirected_std[in|out|err]
- decimal: with decimal.Context
- os: with os.current_directory
- mutex: with mutexobj
- threading: with threading.Lock with threading.Condition with threading.Event
- bz2/zipfile/tarfile: with ...open
-Brett
On Jul 8, 2005, at 4:46 PM, Brett Cannon wrote:
I think having basic context managers in a stdlib module that we know for a fact that will be handy is a good idea. We should keep the list short and poignant, but we should have something for people to work off of. The ones I like below for a 'context' module are:
- builtins: with open/file
- sys: with sys.redirected_std[in|out|err]
- decimal: with decimal.Context
- os: with os.current_directory
- mutex: with mutexobj
- threading: with threading.Lock with threading.Condition with threading.Event
- bz2/zipfile/tarfile: with ...open
It is a really bad idea to codify the practice of modifying non- threadlocal global state like sys.std[in|out|err] and current directory with a context manager. A user can do it to themselves now, true, but by putting a context manager for it in the stdlib, you make it look like it'd be a local modification when it is not. I can only see confusion resulting from this. Users will inevitably try to use it like with sys.redirected_stderr(f): print "hello" print "there" instead of explicitly writing to f with print>> or write(). And that is just a terribly bad idea. It looks pretty, yes, but unless stdinouterr are made thread-local, it's just asking for trouble.
James
On 7/8/05, James Y Knight foom@fuhm.net wrote:
It is a really bad idea to codify the practice of modifying non- threadlocal global state like sys.std[in|out|err] and current directory with a context manager. A user can do it to themselves now, true, but by putting a context manager for it in the stdlib, you make it look like it'd be a local modification when it is not. I can only see confusion resulting from this. Users will inevitably try to use it like with sys.redirected_stderr(f):
Which is of course a bug -- print doesn't go to stderr. Not sure if you meant this as an illustration of a typical bug or whether you meant sys.redirect_stdout(f).
print "hello" print "there"instead of explicitly writing to f with print>> or write(). And that is just a terribly bad idea. It looks pretty, yes, but unless stdinouterr are made thread-local, it's just asking for trouble.
Boy, do you have Java (or multi-threading) on your mind. A lot of Python programs are single-threaded, and this idiom has been promoted by examples for many years; I see nothing wrong with it in single-threaded code. I sure hope the world doesn't evolve to one where most programs have to be multi-threaded to be useful! Multi-threading is such a nightmare to program that we should really look for paradigms that hide it completely rather than trying to raise programmers who will write correct multi-threaded programs naturally; the latter just ain't gonna happen.
A major use case for this, BTW, is to take *existing* Python code that was written using print statements and wrap it in something that captures its output. The f.write() or print>>f solutions won't work there...
I compiled a list of some possible new context managers that could
be
added to the stdlib. Introducing a new feature should IMO also show usage of it in the distribution itself. That wasn't done with decorators (a decorators module is compiled at the moment, if I'm
right),
but with context managers, there's certainly room to add some. Of
course,
my list is excessive, it's only some ideas I got by flying over the
stdlib
docs.
The PEP contains plenty of examples. If you're really feeling the need, add something to the demo directory.
For the most part, the applications need to work themselves out over time through wikis, individual patch submissions, ASPN recipes, third-party apps, etc.
The natural evolution of best practices tends to get thwarted by prematurely using the standard library to cast a particular solution in stone.
This doesn't preclude individual patches such as a context manager for decimal.Context objects. Each proposal should be considered on its own merits rather than as a part of an overall effort to ram a bunch of these into the standard library. Over time, plenty of these will sprout-up.
Raymond
Raymond Hettinger wrote:
I compiled a list of some possible new context managers that could
be
added to the stdlib. Introducing a new feature should IMO also show usage of it in the distribution itself. That wasn't done with decorators (a decorators module is compiled at the moment, if I'm
right),
but with context managers, there's certainly room to add some. Of
course,
my list is excessive, it's only some ideas I got by flying over the
stdlib
docs.
The PEP contains plenty of examples. If you're really feeling the need, add something to the demo directory.
For the most part, the applications need to work themselves out over time through wikis, individual patch submissions, ASPN recipes, third-party apps, etc.
The natural evolution of best practices tends to get thwarted by prematurely using the standard library to cast a particular solution in stone.
This doesn't preclude individual patches such as a context manager for decimal.Context objects. Each proposal should be considered on its own merits rather than as a part of an overall effort to ram a bunch of these into the standard library. Over time, plenty of these will sprout-up.
As with decorators? Sure, plenty have sprouted up, but delivering them one release after the feature itself (if the decorators module makes it into 2.5) seems a little bit like lagging behind -- especially when good and useful examples exist.
Of course, not all of these in my list are good ideas to implement.
Reinhold
At 10:24 PM 7/8/2005 +0200, Reinhold Birkenfeld wrote:
with sys.trace
Note that it's currently not possible to inspect the trace/profile hooks from Python code, only from C, so that might be, um, interesting to implement.
- pprint: with pprint.printer (used for print)
Interesting; I'm not sure if I like it.
- os: with os.current_directory
What does this do? Oh, I get it. The name's not very obvious. I would've expected a more imperative name, something like 'with os.setcwd()' or 'with os.pushdir()'.
with os.modified_env with os.umask/uid/gid etc.
Yeah, again I would expect more verbish names, but these are at least easier to grasp than current_directory was.
- curses: with curses.wrapper with logging.Logger
- tty: with tty.raw with tty.cbreak
- cgitb: with cgitb.enabled
What do these do?
Phillip J. Eby wrote:
At 10:24 PM 7/8/2005 +0200, Reinhold Birkenfeld wrote:
with sys.traceNote that it's currently not possible to inspect the trace/profile hooks from Python code, only from C, so that might be, um, interesting to implement.
That was beyond my short view... if it can't be implemented, it won't.
- pprint: with pprint.printer (used for print)
Interesting; I'm not sure if I like it.
- os: with os.current_directory
What does this do? Oh, I get it. The name's not very obvious. I would've expected a more imperative name, something like 'with os.setcwd()' or 'with os.pushdir()'.
I didn't think about the names too long. ;)
with os.modified_env with os.umask/uid/gid etc.Yeah, again I would expect more verbish names, but these are at least easier to grasp than current_directory was.
Names can be found, of course.
- curses: with curses.wrapper with logging.Logger
- tty: with tty.raw with tty.cbreak
- cgitb: with cgitb.enabled
What do these do?
curses: curses.wrapper currently is a function which restores sane terminal settings after the curses program bails out. It could be a context manager too. Similar is tty.raw and tty.cbreak. These switch terminal modes, and in case of an uncaught exception the terminal will stay in this state. Context managers would restore the "sane" mode on exit.
logging.Logger: hm, I didn't think about that properly. cgitb: enables or disables the full traceback view (Ok, it wouldn't be so useful).
Reinhold
On Fri, 2005-07-08 at 16:24, Reinhold Birkenfeld wrote:
I compiled a list of some possible new context managers that could be added to the stdlib.
I agree with Brett and Phillip that a few well-chosen context managers would make sense in the stdlib both for convenience and for example purposes. Keep the list short and sweet. Also, I'd like to see some higher-level cm's in the mix, perhaps implementing concepts like Java's synchronized (perhaps even being named 'with synchronization').
Thinking about the types of code I write over and over again, I think I disagree (slightly) with James about the global state thing. While I agree that 'with redirected_stdio' isn't too useful in the face of print>>, I very often have to write try/finally protections around temporary settings of the cwd, the umask, and other global process values. I'd love to see cm's for those constructs in the stdlib.
-Barry
participants (7)
-
Barry Warsaw -
Brett Cannon -
Guido van Rossum -
James Y Knight -
Phillip J. Eby -
Raymond Hettinger -
Reinhold Birkenfeld