[Python-Dev] Simple generators, round 2

[Christian Tismer]

Neil Schemenauer wrote:
> 
> I've got a different implementation.  There are no new keywords
> and its simpler to wrap a high level interface around the low
> interface.
> 
>     http://arctrix.com/nas/python/generator2.diff
> 
> What the patch does:
> 
>     Split the big for loop and switch statement out of eval_code2
>     into PyEval_EvalFrame.
> 
>     Add a new "why" flag for ceval, WHY_SUSPEND.  It is similar to
>     WHY_RETURN except that the frame value stack and the block stack
>     are not touched.  The frame is also marked resumable before
>     returning (f_stackbottom != NULL).
> 
>     Add two new methods to frame objects, suspend and resume.
>     suspend takes one argument which gets attached to the frame
>     (f_suspendvalue).  This tells ceval to suspend as soon as control
>     gets back to this frame.  resume, strangely enough, resumes a
>     suspended frame.  Execution continues at the point it was
>     suspended.  This is done by calling PyEval_EvalFrame on the frame
>     object.
> 
>     Make frame_dealloc clean up the stack and decref f_suspendvalue
>     if it exists.
> 
> There are probably still bugs and it slows down ceval too much
> but otherwise things are looking good.  Here are some examples
> (the're a little long and but illustrative).  Low level
> interface, similar to my last example:

I've had a closer look at your patch (without actually applying
and running it) and it looks good to me.
A possible bug may be in frame_resume, where you are doing
+       f->f_back = tstate->frame;
without taking care of the prior value of f_back.

There is a little problem with your approach, which I have
to mention: I believe, without further patching it will be
easy to crash Python.
By giving frames the suspend and resume methods, you are
opening frames to everybody in a way that allows to treat
them as kind of callable objects. This is the same problem
that Stackless had imposed.
By doing so, it might be possible to call any frame, also
if it is currently run by a nested interpreter.

I see two solutions to get out of this:

1) introduce a lock flag for frames which are currently
   executed by some interpreter on the C stack. This is
   what Stackless does currently.
   Maybe you can just use your new f_suspendvalue field.
   frame_resume must check that this value is not NULL
   on entry, and set it zero before resuming.
   See below for more.

2) Do not expose the resume and suspend methods to the
   Python user, and recode Generator.py as an extension
   module in C. This should prevent abuse of frames.

Proposal for a different interface:
I would change the interface of PyEval_EvalFrame
to accept a return value passed in, like Stackless
has its "passed_retval", and maybe another variable
that explicitly tells the kind of the frame call,
i.e. passing the desired why_code. This also would
make it easier to cope with the other needs of Stackless
later in a cleaner way.
Well, I see you are clearing the f_suspendvalue later.
Maybe just adding the why_code to the parameters
would do. f_suspendvalue can be used for different
things, it can also become the place to store a return
value, or a coroutine transfer parameter.

In the future, there will not obly be the suspend/resume
interface. Frames will be called for different reasons:
suspend  with a value  (generators)
return   with a value  (normal function calls)
transfer with a value  (coroutines)
transfer with no value (microthreads)

ciao - chris

-- 
Christian Tismer             :^)   <mailto:tismer@tismer.com>
Mission Impossible 5oftware  :     Have a break! Take a ride on Python's
Kaunstr. 26                  :    *Starship* http://starship.python.net/
14163 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
PGP Fingerprint       E182 71C7 1A9D 66E9 9D15  D3CC D4D7 93E2 1FAE F6DF
     where do you want to jump today?   http://www.stackless.com/