Mailman 3 April 2009 - Python-ideas

yield-from practical experience
by Erik Groeneveld April 14, 2009

April 14, 2009

Andrew McNabb suggested I take a look at the python-ideas thread about yield-from because I have written a little framework called Weightless that supports decomposing a program into generators. I am aware that I am a bit late, but here is my two cents. I read the PEP and the discussions (most of it, to be fair) and I compared it to what I have been creating the last couple of years. For one thing, it is certainly not easy to create a comprehensible way of decomposing programs into generators, especially in the presence of exceptions. It took me two years to understand it, and I have coded it into a single generator called 'compose' (http://www.weightless.io/compose). As some people wanted to see an implementation of 'yield from': here is one that actually uses the normal yield and just assumes any generator is to be descended into. (http://weightless.svn.sourceforge.net/viewvc/weightless/trunk/weightless/_c…) 'compose' allows one to decompose a program into generators, just like it has been proposed. As was is also questioned why all the complexity is needed. I believe it is necessary to create an clear, abstract, comprehensible and consistent programming interface. Consider these examples: Many programmers would expect this to work: def myHandler(): try: request = yield from readRequest() except: handle error But it is really not working without catching exceptions from readRequest() and forward them to myHandler(). Again, many programmers would expect this to work: def a(): yield from b() def b(): yield from c() def c(): raise Exception('b') yield 1 You would like to see a stack trace like: Traceback (most recent call last): File "example.py", line 34, in list(compose(a())) File "example.py", line 27, in a yield b() File "example.py", line 29, in b yield c() File "example.py", line 32, in c raise Exception('b') Exception: b but you will see this, for more of less the same reason as why you weren't able to catch exceptions in the first place: Traceback (most recent call last): File "probeer.py", line 34, in list(compose(a())) File "probeer.py", line 32, in c raise Exception('b') Exception: b We succeeded to create a rather complicated 'compose' that give an intuitive decomposition tool for programmers and we are using it at a daily basis. (see link above) One of the main advantages (besides lightweight threading) is the use of JSP style pipelines. Things like for exampe an HTTP protocol handler are much easier (have a more friendly flow, as Klaas van Schelven puts it) to create than with callbacks. An simple example can bee seen here: http://weightless.svn.sourceforge.net/viewvc/weightless/trunk/examples/call… I've set this up for the Dutch Python Usergroup and it demonstrates the idea. I really would be glad if something like 'yield from' becomes part of the language. I would love to cooperate during the realization and testing so I can validate my own bakery of the last years and perhaps contribute something? Erik Groeneveld

1 0

Proposed addtion to urllib.parse in 3.1 (and urlparse in 2.7)
by Mart Sõmermaa April 12, 2009

April 12, 2009

Appending query parameters to a URL is a very common need. However, there's nothing in urllib.parse (and older urlparse) that caters for that need. Therefore, I propose adding the following to 2.7 and 3.1 in the respective libs: def add_query_params(url, **params): """ Adds additional query parameters to the given url, preserving original parameters. Usage: >>> add_query_params('http://foo.com', a='b') 'http://foo.com?a=b' >>> add_query_params('http://foo.com?a=b', b='c', d='q') 'http://foo.com?a=b&b=c&d=q' The real implementation should be more strict, e.g. raise on the following: >>> add_query_params('http://foo.com?a=b', a='b') 'http://foo.com?a=b&a=b' """ if not params: return url encoded = urllib.urlencode(params) url = urlparse.urlparse(url) return urlparse.urlunparse((url.scheme, url.netloc, url.path, url.params, (encoded if not url.query else url.query + '&' + encoded), url.fragment))

12 26

Re: [Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]
by Greg Ewing April 12, 2009

April 12, 2009

Jacob Holm wrote: > I > am saying that there are examples where it is desirable to move one of > the arguments that this form of refactoring forces you to put in the > constructor so it instead becomes the argument of the first send. I'm having trouble seeing circumstances in which you would need to do that. Can you provide an example in the form of (a) a piece of unfactored code (b) a desired refactoring (c) an explanation of why the desired refactoring can't conveniently be done using an unprimed generator and plain yield-from. -- Greg

4 14

Re: [Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]
by Greg Ewing April 11, 2009

April 11, 2009

Nick Coghlan wrote: >>>>class Tricky(object): > > ... def __iter__(self): > ... return self > ... def next1(self): > ... print "Next 1" > ... Tricky.next = Tricky.next2 I just tried using a class like that in a for loop, and it doesn't work. The next() method has to be defined in the class, because it's actually a special method (it has a slot in the type object). > Greg - looks to me like you're going to have to disallow caching the > method lookup in the PEP. Even if it would work, I don't want to preclude a valuable optimization for the sake of such an obscure use case. -- Greg

2 3

x=(yield from) confusion [was:Yet another alternative name for yield-from]
by Jim Jewett April 10, 2009

April 10, 2009

Summary: I can understand a generator that returns something useful with each yield. I can understand a generator that is used only for cooperative multi-tasking, and returns nothing useful until the end. yield from *as an expression* only really makes sense if the generator is sending useful information *both* ways. I can understand that sort of generator only while reading the PEP; the code smell is strong enough that I forget it by the next day. Details: On 4/2/09, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > Jim Jewett wrote: >> If the "gencall" exhausts the generator f ... >> If the "return value" of the generator really is important ... > The intermediate values aren't necessarily discarded by "yield from" > though: they're passed out to whoever is consuming the values yielded by > the outermost generator. I think this may be where I start to have trouble. When I see the statement form: def outer_g(): yield from inner_g() I expect inner_g to suspend execution, and it isn't that hard to remember that it might do so several times. I also expect the results of inner_g.next() to be passed on out to outer_g's own caller, thereby suspending outer_g. So far, so good. But when yield from is used as an expression, and I see: x = yield from g() I somehow expect only a single call to g.next(), whose value gets assigned to x, and not passed out. I did read the PEP, in several versions, and understood it at the time ... and still managed (several times) to forget and misinterpret by a day or two later. And yes, I realize it doesn't make too much sense to call g.next() only once -- so I kept looking for a loop around that yield from statement. When there wasn't one, I shrugged it off like a with clause -- and still didn't remember the actual PEP-intended meaning. The times I did remember that (even) the expression form looped, I was still boggled that it would return something other than None after it was exhausted. Greg's answer was that it was for threading, and the final return was the real value. This seems like a different category of generator, but I could get my head around it -- so long as I forgot that the yield itself was returning anything useful. -jJ

7 74

yiled-from restrictions [was: x=(yield from) confusion]
by Jim Jewett April 10, 2009

April 10, 2009

On 4/9/09, Jacob Holm <jh(a)improva.dk> wrote: > Guido van Rossum wrote: >> On Wed, Apr 8, 2009 at 7:52 PM, Jacob Holm <jh(a)improva.dk> wrote: >>> Does storing it as part of the frame object count as "naturally in the >>> stack frame"? >> No, that's out too. My point is that I don't want to add *any* state >> beyond what the user thinks of as the "normal" state in the frame >> (i.e. local variables, where it is suspended, and the expression stack >> and try-except stack). Nothing else. Do you mean nothing user-visible, or do you really mean nothing, not even as an implementation detail? > That rules out Gregs and my patches as well. They both need extra state > on the frame object to be able to implement yield-from in the first place. Well, to do it efficiently anyhow... but it doesn't need to be user-visible, which is why I asked for clarification. > One final suggestion I have is to make yield-from raise a RuntimeError > if used on a generator that already has a frame. That would prevent priming the generator. It would also prevent "handled the header lines already; pass the data records off to a different routine." -jJ

3 4

Custom format() proposal redux
by Raymond Hettinger April 8, 2009

April 8, 2009

The original proposal was well-received but it didn't make provisions to handle str.format(). Here is the revised proposal. Only the last paragraph is new. Raymond ------------------------------------------------------------------- Mark Dickinson's decimal test code suggested a good, extensible approach to the problem. Here's the idea in a nutshell: format(value, format_spec='', conventions=None) 'calls value.__format__(format_spec, conventions)' Where conventions is an optional dictionary with formatting control values. Any value object can accept custom controls, but the names for standard ones would be taken from the standards provided by localeconv(): { 'decimal_point': '.', 'grouping': [3, 0], 'negative_sign': '-', 'positive_sign': '', 'thousands_sep': ','} The would let you store several locales using localeconv() and use them at will, thus solving the global variable and threading problems with locale: import locale loc = locale.getlocale() # get current locale locale.setlocale(locale.LC_ALL, 'de_DE') DE = locale.localeconv() locale.setlocale(locale.LC_ALL, 'en_US') US = locale.localeconv() locale.setlocale(locale.LC_ALL, loc) # restore saved locale . . . format(x, '8,.f', DE) format(y, '8,d', US) It also lets you write your own conventions on the fly: DEBUG = dict(thousands_sep='_') # style for debugging EXTERN = dict(thousands_sep=',') # style for external display . . . format(x, '8.1f', DEBUG) format(y, '8d', EXTERN) The dictionaries can be registered for use with the mini-formatting language: locale.setlocale(locale.LC_ALL, 'en_US') str.format.register(US=locale.localeconv()) str.format.register(HY=dict(thousands_sep='-')) . . . 'Nigerian President will forward {0:,d!US} to your account'.format(10000000) format(y, ',d!HY') format(z, ',d!US')

2 1

Draft PEP: Standard daemon process library
by Ben Finney April 8, 2009

April 8, 2009

Howdy all, I am preparing a PEP, and corresponding reference implementation, for a standard implementation of the steps needed to turn a program into a well-behaved Unix daemon. This message is a call for comments, support, and suggestions, prior to submitting this PEP to the editor. :PEP: XXX :Title: Standard daemon process library :Version: 0.1 :Last-Modified: 2009-01-26 08:44 :Author: Ben Finney <ben+python(a)benfinney.id.au> :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 2009-01-26 :Python-Version: 3.1 :Post-History: ======== Abstract ======== Writing a program to become a well-behaved Unix daemon is somewhat complex and tricky to get right, yet the steps are largely similar for any daemon regardless of what else the program may need to do. This PEP introduces a module to the Python standard library that provides a simple interface to the task of becoming a daemon process. .. contents:: .. Table of Contents: Abstract Specification Example usage Interface ``Daemon`` objects ``DaemonError`` objects Motivation Rationale Correct daemon behaviour Reference Implementation References Copyright ============= Specification ============= Example usage ============= Simple example of usage:: import daemon from spam import do_main_program this_daemon = daemon.Daemon() this_daemon.start() do_main_program() More complex example usage:: import os import grp import signal import daemon from spam import ( initial_program_setup, do_main_program, program_cleanup, reload_program_config, ) initial_program_setup() important_file = open('spam.data', 'w') interesting_file = open('eggs.data', 'w') this_daemon = daemon.Daemon() this_daemon.files_preserve = [important_file, interesting_file] this_daemon.working_directory = '/var/lib/foo' this_daemon.umask = 0o002 mail_gid = grp.getgrnam('mail').gr_gid this_daemon.gid = mail_gid this_daemon.terminate_callback = program_cleanup this_daemon.reload_callback = reload_program_config this_daemon.reload_signals = [signal.SIGHUP, signal.SIGUSR1] this_daemon.start() do_main_program() Interface ========= A new module, `daemon`, is added to the standard library. The module defines a class, `Daemon`, used to represent the settings for a daemon process. An exception class, `DaemonError`, is defined for exceptions raised from the module. ``Daemon`` objects ================== A `Daemon` instance represents the behaviour settings for the process when it becomes a daemon. The behaviour is customised by setting attributes on the instance, before calling the `start` method. The following attributes are defined. `files_preserve` :Default: ``None`` List of files that should *not* be closed when starting the daemon. If ``None``, all open file descriptors will be closed. Elements of the list are file descriptors (as returned by a file object's `fileno()` method) or Python `file` objects. Each specifies a file that is not to be closed during daemon start. `chroot_directory` :Default: ``None`` Full path to a directory to set as the effective root directory of the process. If ``None``, specifies that the root directory is not to be changed. `working_directory` :Default: ``'/'`` Full path of the working directory to which the process should change on daemon start. Since a filesystem cannot be unmounted if a process has its current working directory on that filesystem, this should either be left at default or set to a directory that is a sensible “home directory” for the daemon while it is running. `lockfile_directory` :Default: ``'/var/run'`` Absolute directory path to contain the daemon's lockfile. If ``None``, the lockfile behaviour for this daemon is skipped. `lockfile_name` :Default: ``None`` Base name of the lockfile for this daemon, without directory or extension. If ``None``, the name is derived from the process command line. `umask` :Default: ``0`` File access creation mask (“umask”) to set for the process on daemon start. Since a process inherits its umask from its parent process, starting the daemon will reset the umask to this value so that files are created by the daemon with access modes as it expects. `ignore_signals` :Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]`` List of signals that the process should ignore (by setting the signal action to ``signal.SIG_IGN``) on daemon start. `terminate_signals` :Default: ``[signal.SIGTERM]`` List of signals that the process should interpret as a request to terminate cleanly. `terminate_callback` :Default: ``None`` Callable to invoke when the process receives any of the `terminate_signals` signals, before then terminating the process. `reload_signals` :Default: ``[signal.SIGHUP]`` List of signals that the process should interpret as a request to reload runtime configuration. `reload_callback` :Default: ``None`` Callable to invoke when the process receives any of the `reload_signals` signals. `uid` :Default: ``None`` The user ID (“uid”) value to switch the process to on daemon start. `gid` :Default: ``None`` The group ID (“gid”) value to switch the process to on daemon start. `prevent_core` :Default: ``True`` If true, prevents the generation of core files, in order to avoid leaking sensitive information from daemons run as `root`. `stdout` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stdout`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stdout` is not re-bound. `stderr` :Default: ``None`` File-like object, open for writing (in append mode, 'w+'), that will be used as the new value of `sys.stderr`. If it represents an actual file, it should be listed in `files_preserve` to prevent it being closed during daemon start. If ``None``, then `sys.stderr` is not re-bound. The following methods are defined. `start()` :Return: ``None`` Start the daemon. This performs the following steps: * If the `chroot_directory` attribute is not ``None``: * Set the effective root directory of the process to that directory (via `os.chroot`). This allows running the daemon process inside a “chroot gaol” as a means of limiting the system's exposure to rogue behaviour by the process. * If the `lockfile_directory` attribute is not ``None``: * Look in that directory for a file named '`lockfile_name`.pid'; if it exists, raise a `DaemonError` to prevent multiple instances of the daemon process. * Close all open file descriptors, excluding those listed in the `files_preserve` attribute. * Change current working directory to the path specified by the `working_directory` attribute. * Reset the file access creation mask to the value specified by the `umask` attribute. * Detach the current process into its own process group, and disassociate from any controlling terminal. This step is skipped if it is determined to be redundant: if the process was started by `init`, by `initd`, or by `inetd`. * Set signal handlers as specified by the `ignore_signals`, `terminate_signals`, `terminate_callback`, `reload_signals`, `reload_callback` attributes. * If the `prevent_core` attribute is true: * Set the resource limits for the process to prevent any core dump from the process. * Set the process uid and gid to the true uid and gid of the process, to relinquish any elevated privilege. * If the `lockfile_directory` attribute is not ``None``: * Create the lockfile for this daemon in that directory, by writing a text line containing the current process ID (“pid”) to a file named '`lockfile_name`.pid'. * If either of the attributes `uid` or `gid` are not ``None``: * Set the process uid and/or gid to the specified values. * If either of the attributes `stdout` or `stderr` are not ``None``: * Bind the names `sys.stdout` and/or `sys.stderr` to the corresponding objects. `reload()` :Return: ``None`` Reload the daemon configuration. The meaning of this is entirely defined by the customisation of this daemon: if the `reload_callback` attribute is not ``None``, call that object. The return value is discarded. `stop()` :Return: ``None`` Stop the daemon. This performs the following steps: * If the `terminate_callback` attribute is not ``None``: * Call that object. The return value is discarded. * If the `lockfile_directory` attribute is not ``None``: * Delete the lockfile for this daemon. * Raise a `SystemExit` exception. ``DaemonError`` objects ======================= The `DaemonError` class inherits from `Exception`. The module implementation will raise an instance of `DaemonError` when an error occurs in processing daemon behaviour. ========== Motivation ========== The majority of programs written to be Unix daemons either implement behaviour very similar to that in the `Specification`_, or are poorly-behaved daemons by the `Rationale`_. Since these steps should be much the same in most implementations but are very particular and easy to omit or implement incorrectly, they are a prime target for a standard well-tested implementation in the standard library. ========= Rationale ========= Correct daemon behaviour ======================== According to Stevens in [stevens]_ §2.6, a program should perform the following steps to become a Unix daemon process. * Close all open file descriptors. * Change current working directory. * Reset the file access creation mask. * Run in the background. * Disassociate from process group. * Ignore terminal I/O signals. * Disassociate from control terminal. * Don't reacquire a control terminal. * Correctly handle the following circumstances: * Started by System V `init` process. * Daemon termination by ``SIGTERM`` signal. * Children generate ``SIGCLD`` signal. The `daemon` package [daemon]_ lists (in its summary of features) behaviour that should be performed when turning a program into a well-behaved Unix daemon process. The following are appropriate for a daemon started once the program is already running: * Sets up the correct process context for a daemon. * Behaves sensibly when started by `initd(8)` or `inetd(8)`. * Revokes any suid or sgid privileges to reduce security risks in case daemon is incorrectly installed with special privileges. * Prevents the generation of core files to prevent leaking sensitive information from daemons run as root (optional). * Names the daemon by creating and locking a PID file to guarantee that only one daemon with the given name can execute at any given time (optional). * Sets the user and group under which to run the daemon (optional, root only). * Creates a chroot gaol (optional, root only). * Captures the daemon's stdout and stderr and directs them to syslog (optional). ======================== Reference Implementation ======================== The `python-daemon` package [python-daemon]_. As of 2009-01-26, the package is under active development and is not yet a full implementation of this PEP. ========== References ========== .. [stevens] `Unix Network Programming`, W. Richard Stevens, 1994 Prentice Hall. .. [daemon] The (non-Python) `daemon` package, `<http://www.libslack.org/daemon/>`_ by “raf” <raf(a)raf.org>. .. [python-daemon] The `python-daemon` package at the Python Package Index, `<http://pypi.python.org/pypi/python-daemon>`_. This is the successor to [bda.daemon]_. .. [bda.daemon] The `bda.daemon` package at the Python Package Index, `<http://pypi.python.org/pypi/bda.daemon>`_. This is an implementation of [cookbook-66012]_. .. [cookbook-66012] Many good ideas were contributed by the community to Python cookbook recipe 66012, “Fork a daemon process on Unix” `<http://code.activestate.com/recipes/66012/>`_. ========= Copyright ========= This work is hereby placed in the public domain. To the extent that placing a work in the public domain is not legally possible, the copyright holder hereby grants to all recipients of this work all rights and freedoms that would otherwise be restricted by copyright. -- \ “Pinky, are you pondering what I'm pondering?” “Well, I think | `\ so (hiccup), but Kevin Costner with an English accent?” —_Pinky | _o__) and The Brain_ | Ben Finney

21 81

why not "name = value if condition"?
by spir April 7, 2009

April 7, 2009

Hello, What's the reason why name = value if condition is invalid? Meaning: there _must_ be an else clause. [I imagine this has been discussed and refused consciously, but I couldn't find it in PEP308, nore in archives.] It would be practicle in many situations, e.g. to give default values to parameters: def writeFile(text, title, format=STANDARD, fileName=None): fileName = title+".py" if fileName is None ... Denis PS: Sure, one can write if condition: name = value but the rationale in favour of, or against, a one-liner shortcut is the same as for the ternary case (with else). One can also write: name = value if condition else name but... <no comment>. I find the present situation a bit frustrating, like sitting between 2 chairs. ------ la vita e estrany

5 5

Re: [Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]
by Guido van Rossum April 6, 2009

April 6, 2009

On Sun, Apr 5, 2009 at 7:46 AM, Jacob Holm <jh(a)improva.dk> wrote: > The argument that we have no value to send before we have yielded is wrong. > The generator containing the "yield-from" could easily have a value to send > (or throw), and if iter(EXPR) returns a coroutine or a non-generator it > could easily be ready to accept it. That is the idea behind my attempted > fixes to the @coroutine issue. I think it's simpler to refrain from yield-from in that case and spell it out. If the value to send doesn't come from outside the outer generator, yield-from is not the solution. > This is where the fun begins. In an earlier thread we concluded that if the > thrown exception is a StopIteration and the *same* StopIteration instance > escapes the throw() call, it should be reraised rather than caught and > turned into a RETVAL. The reasoning was the following example: > > def inner(): > for i in xrange(10): > yield i > > def outer(): > yield from inner() > print "if StopIteration is thrown in we shouldn't get here" > > Which we wanted to be equivalent to: > > def outer(): > for i in xrange(10): > yield i > print "if StopIteration is thrown in we shouldn't get here" > > The same argument goes for ReturnFromGenerator, so the expansion at this > stage should be more like: [snip] This example and reasoning are invalid. You shouldn't be throwing StopIteration (or ReturnFromGenerator) *into* a generator. That's something that should only come *out*. > Next issue is that the value returned by it.close() is thrown away by > yield-from. Here is a silly example: > > def inner(): > i = 0 > while True > try: > yield > except GeneratorExit: > return i > i += 1 > > def outer(): > try: > yield from inner() > except GeneratorExit: > # nothing I can write here will get me the value returned from inner() > > Also the trivial: > > def outer(): > return yield from inner() > > Would swallow the return value as well. > > I have previously suggested attaching the return value to the (re)raised > GeneratorExit, and/or saving the return value on the generator and making > close return the value each time it is called. We could also choose to > define this as broken behavior and raise a RuntimeError, although it seems a > bit strange to have yield-from treat it as an error when close doesn't. > Silently having the yield-from construct swallow the returned value is my > least favored option. Attaching it to the GeneratorExit is just plain wrong -- this is an exception you throw *in*, not something that is thrown out (except when it bounces back). One solution is not to use yield-from but write it out using yield and send (just like the full expansion, but you can probably drop most of the complexity for any particular example). Another solution is not to use close() and GeneratorExit but some application-specific exception to signal the end. But perhaps it would be okay to change the GeneratorExit handler in the expansion so that it passes through the return value with a StopIteration exception: rv = it.close() if rv is None: raise StopIteration(rv) # Or ReturnFromGenerator(rv) else: raise Alternatively, simpler: it.throw(GeneratorExit) # We only get here if it yielded a value raise RuntimeError(...) (Though this isn't exactly if we were to use duck typing.) We could then write the first version of outer() like this: def outer(): try: yield from inner() except StopIteration as e: ...access return value as e.value... and I think the second (trivial) outer() will return inner()'s return value just fine, since it just passes through as a StopIteration value. > Like Greg, I am in favor of duck-typing this as closely as possible. OK, noted. I think it's probably fine. FWIW, I'm beginning to think that ReturnFromGenerator is a bit of a nuisance, and that it's actually fine to allow "return value" inside a generator to mean "raise StopIteration(value)" (well not quite at that point in the code but once we are about to clean up the frame). Maybe I've overstated the case for preventing beginners' mistakes. After all they'll notice that their generator returns prematurely when they include any kind of return value. Also if the StopIteration ends being printed as a traceback the value will be printed, which is the kind of hint newbies love. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

3 6