From tjreedy at udel.edu  Fri Aug  1 05:00:46 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 31 Jul 2008 23:00:46 -0400
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <70e6cf130807301157v630f8be1pb8dd29e749ae3797@mail.gmail.com>
References: <486CDCF5.6080805@korokithakis.net>
	<488EEDCC.7050303@onego.ru>	<c7450fba0807290955y57437896wca3ad1b037ad824d@mail.gmail.com>	<488F64A7.9010902@onego.ru>
	<g6o1g5$h5h$1@ger.gmane.org>	<d11dcfba0807291504w43eb06cfvb68b1709d606302f@mail.gmail.com>	<g6ol7o$215$1@ger.gmane.org>	<d11dcfba0807292229q3b2dfc0r3d3a4e5bf1824068@mail.gmail.com>	<g6pvmu$10v$1@ger.gmane.org>	<d11dcfba0807300926r26dd02d4oef57edefd3afc4cb@mail.gmail.com>
	<70e6cf130807301157v630f8be1pb8dd29e749ae3797@mail.gmail.com>
Message-ID: <g6tu8u$3n5$1@ger.gmane.org>



David Borowitz wrote:
> Precisely: you're talking about two different accepted definitions of 
> semantics:
> http://en.wikipedia.org/wiki/Formal_semantics_of_programming_languages
> 
> I'm not going to say one type of formal semantics is better than 
> another, but it's nice when everyone is at least on the same page :)

Thank you for the reference.  I see that denotational and operational 
are just the beginning of the semantics of semantics.

In writing about and comparing algorithms, I need a term that is 
slightly in between but closer to operational than merely denotational. 
  I am thinking of using 'operationally equivalent' to describe two 
algorithms that are denotationally (functionally) identical and which 
compute the same essential intermediate objects (values) in the same 
order, while not being 'operationally identical' (which would make them 
the same thing).

Wikipedia refers 'operational equivalence' to 'observational 
equivalence'.  Other hits from Google suggest that it have been used 
variably to mean things from operationally identical to merely 
denotationally (functionally identical), but therefore a complete 
substitute in the users operations.

In the present case, we agree that [genexp] == list(genexp] in the 
Python meaning, denotational and operational, of '==', which is to 
compare the denotational meaning of two expressions (the resulting 
objects and their values).  I also claim a bit more: [genexp] could be 
but does not have to be, and in current CPython is not, operationally 
identical to list(genexp).  But I do expect that it is operationally 
equivalent in the sense above.  The essential objects and operations are 
the list contents and the list that starts empty and grows one reference 
at a time via append().

Terry Jan Reedy



From borowitz at stanford.edu  Fri Aug  1 19:08:42 2008
From: borowitz at stanford.edu (David Borowitz)
Date: Fri, 1 Aug 2008 10:08:42 -0700
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <g6tu8u$3n5$1@ger.gmane.org>
References: <486CDCF5.6080805@korokithakis.net> <488F64A7.9010902@onego.ru>
	<g6o1g5$h5h$1@ger.gmane.org>
	<d11dcfba0807291504w43eb06cfvb68b1709d606302f@mail.gmail.com>
	<g6ol7o$215$1@ger.gmane.org>
	<d11dcfba0807292229q3b2dfc0r3d3a4e5bf1824068@mail.gmail.com>
	<g6pvmu$10v$1@ger.gmane.org>
	<d11dcfba0807300926r26dd02d4oef57edefd3afc4cb@mail.gmail.com>
	<70e6cf130807301157v630f8be1pb8dd29e749ae3797@mail.gmail.com>
	<g6tu8u$3n5$1@ger.gmane.org>
Message-ID: <70e6cf130808011008x68842876td86e1f43f81780bc@mail.gmail.com>

Another good reference is the "as-if" rule in C. It's similar to operational
semantics, but may have enough denotational semantics semantics to be what
you're looking for. Don't know of a canonical reference, but Google is
useful:
http://docs.sun.com/app/docs/doc/819-5265/6n7c29d9f?l=ko&a=view

On Thu, Jul 31, 2008 at 20:00, Terry Reedy <tjreedy at udel.edu> wrote:

>
>
> David Borowitz wrote:
>
>> Precisely: you're talking about two different accepted definitions of
>> semantics:
>> http://en.wikipedia.org/wiki/Formal_semantics_of_programming_languages
>>
>> I'm not going to say one type of formal semantics is better than another,
>> but it's nice when everyone is at least on the same page :)
>>
>
> Thank you for the reference.  I see that denotational and operational are
> just the beginning of the semantics of semantics.
>
> In writing about and comparing algorithms, I need a term that is slightly
> in between but closer to operational than merely denotational.  I am
> thinking of using 'operationally equivalent' to describe two algorithms that
> are denotationally (functionally) identical and which compute the same
> essential intermediate objects (values) in the same order, while not being
> 'operationally identical' (which would make them the same thing).
>
> Wikipedia refers 'operational equivalence' to 'observational equivalence'.
>  Other hits from Google suggest that it have been used variably to mean
> things from operationally identical to merely denotationally (functionally
> identical), but therefore a complete substitute in the users operations.
>
> In the present case, we agree that [genexp] == list(genexp] in the Python
> meaning, denotational and operational, of '==', which is to compare the
> denotational meaning of two expressions (the resulting objects and their
> values).  I also claim a bit more: [genexp] could be but does not have to
> be, and in current CPython is not, operationally identical to list(genexp).
>  But I do expect that it is operationally equivalent in the sense above.
>  The essential objects and operations are the list contents and the list
> that starts empty and grows one reference at a time via append().
>
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
It is better to be quotable than to be honest.
-Tom Stoppard

Borowitz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080801/035d6e2b/attachment.html>

From guido at python.org  Sat Aug  2 03:56:05 2008
From: guido at python.org (Guido van Rossum)
Date: Fri, 1 Aug 2008 18:56:05 -0700
Subject: [Python-ideas] [Python-Dev] Base-96
In-Reply-To: <a9890139-9120-432c-91d6-003279860103@f36g2000hsa.googlegroups.com>
References: <a9890139-9120-432c-91d6-003279860103@f36g2000hsa.googlegroups.com>
Message-ID: <ca471dc20808011856p1b251fecp57ddbb7f7201186f@mail.gmail.com>

This sounds more like something to bring up in
python-ideas at python.org. Also, rather than being vague about the
motivation ("would be very interesting", you ought to think of a
realistic use case. For example, are there existing encodings of
binary data using base-96? I'm not aware of any.

On Fri, Aug 1, 2008 at 4:06 PM, Kless <jonas.esp at googlemail.com> wrote:
> I think that would be very interesting thay Python would have a module
> for working on base 96 too. [1]
>
> It could be converted to base 96 the digests from hashlib module, and
> random bytes used on crypto (to create the salt, the IV, or a key).
>
> As you can see here [2], the printable ASCII characters are 94
> (decimal code range of 33-126). So only left to add another 2
> characters more; the space (code 32), and one not-printable char
> (which doesn't create any problem) by last.
>
>
> [1] http://svn.python.org/view/python/trunk/Modules/binascii.c
> [2] http://en.wikipedia.org/wiki/ISO/IEC_8859-1

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From dangyogi at gmail.com  Sat Aug  2 21:20:15 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Sat, 02 Aug 2008 15:20:15 -0400
Subject: [Python-ideas] New PEP proposal: C Micro-Threading (ala Twisted)
	(long!)
Message-ID: <4894B36F.4030805@gmail.com>

Here is a proposal for making the benefits of Twisted available without
having to write python code in an event driven style.

(I'm not sure whether this list accepts HTML, so am just copying plain
 text -- you may want to run rst2html on this)...

I look forward to your ideas and feedback!

-bruce


Abstract
========

This PEP adds micro-threading (or `green threads`_) at the C level so that
micro-threading is built in and can be used with very little coding effort
at the python level.

The implementation is quite similar to the Twisted_ [#twisted-fn]_
Deferred_/Reactor_ model, but applied at the C level by extending the
`C API`_ [#c_api]_ slightly.  Doing this provides the Twisted
capabilities to python, without requiring the python programmer to code
in the Twisted event driven style.  Thus, legacy python code would gain the
benefits that Twisted provides with very little modification.

Burying the event driven mechanism in the C level should also give the same
benefits to python GUI interface tools so that the python programmers don't
have to deal with event driven programming there either.

This capability may also be used to provide some of the features that
`Stackless Python`_ [#stackless]_ provides, such as microthreads and
channels (here, called micro_pipes).

.. _Twisted: http://twistedmatrix.com/trac/
.. _Deferred:
   http://twistedmatrix.com/projects/core/documentation/howto/defer.html
.. _Reactor:
   
http://twistedmatrix.com/projects/core/documentation/howto/reactor-basics.html
.. _C API: http://docs.python.org/api/api.html
.. _green threads: http://en.wikipedia.org/wiki/Green_threads

Motivation
==========

The popularity of the Twisted project has demonstrated the need for
micro-threads as an alternative the standard thread_ [#thread-module]_ and
threading_ [#threading-module]_ packages.  These allow large numbers 
(1000's)
of simultaneous connections to python servers, as well as fan-outs to large
numbers of downstream connections.

The advantages to the Twisted approach over standard threads are:

#. much less memory is required per thread
#. faster thread creation
#. faster context switching (I'm guessing on this one, is this really true?)
#. synchronization between threads is easier because there is no preemption,
   making it much easier to write critical sections of code.

The disadvantages are:

#. the python developer must write his/her program in an event driven style
#. the approach can not be used with standard python code that wasn't
   written in this event driven style
#. the approach does not take advantage of multiple processor architectures
#. since there is no preemption, a long running micro-thread will starve
   other micro-threads

This PEP attempts to retain all of the advantages that Twisted has
demonstrated, but resolve the first two disadvantages to make these
advantages accessible to all python programs, including legacy programs
not written in the Twisted style.  This should make it very easy for legacy
programs like WSGI apps, Django and TurboGears to reap the Twisted benefits.

Another example of event driven mechanisms are the GUI/windows events.  This
PEP would also make it easy for python GUI interface toolkits (like wxpython
and qtpython) to hide the GUI/windows event driven style of programming from
the python programmer.  For example, you would no longer need to use modal
dialog boxes just to make the programming easier.

This PEP does not address the last two disadvantages, and thus also has
these disadvantages itself.


Specification of C Layer Enhancements
=====================================

Deferred
--------

``PyDef_Deferred`` is written as a new exception type for use by the C code
to defer execution.  This is a subclass of ``NotImplementedError``.  
Instances
are not raised as a normal exception (e.g., with ``PyErr_SetObject``), 
but by
calling ``PyWatch_Defer``.  This registers the ``PyDef_Deferred`` associated
with the currently running micro_thread as the current error object, but 
also
readies it for its primary job -- deferring execution.  As an exception,
it creates its own error message, if needed, which is "Deferred execution
not yet implemented by %s" % c_function_name.

``PyErr_ExceptionMatches`` may be used with these.  This allows them to be
treated as exceptions by non micro-threading aware (unmodified) C functions.

But these ``PyDef_Deferred`` objects are special indicators that are treated
differently than normal exceptions by micro-threading aware (modified) C
code.  Modified C functions do this by calling ``PyDef_AddCallback``,
``PyDef_Final`` or explicitly checking
``PyErr_ExceptionMatches(PyDef_Deferred)`` after receiving an error return
status from a called function.

``PyDef_Deferred`` instances offer the following methods (in addition to the
normal exception methods):

- ``int PyDef_AddCallbackEx(PyObject *deferred, const char *caller_name,
  const char *called_name, PyObject *(*callback_fn)(PyObject 
*returned_object,
  void *state), void *state)``

  - The *caller_name* and *called_name* are case sensitive.  The 
*called_name*
    must match exactly the *caller_name* used by the called function when it
    dealt with this ``PyExc_Deferred``.  If the names are different, the
    ``PyExc_Deferred`` knows that an intervening unmodified C function was
    called.  This is what triggers it to act like an exception.

    The *called_name* must be ``NULL`` when called by the function that
    executed the ``PyWatch_Defer`` to defer execution.

  - The *callback_fn* will be called with the ``PyObject`` of the results of
    the prior registered callback_fn.  An exception is passed to
    *callback_fn* by setting the exception and passing ``NULL`` (just like
    returning an exception from a C function).  In the case that the
    ``PyExc_Deferred`` initially accepts *callback_fn* and then later has
    to reject it (because of the exception case, above), it will pass a
    ``SystemError`` to all registered callback_fns to allow them to 
clean up.
    But this ``SystemError`` is only a "behind the scenes" measure that will
    only be seen by these callback_fns.  It will be cleared and the prior
    error indicator reestablished before ``PyDef_AddCallback`` returns.
  - The *callback_fn* is always guaranteed to be called exactly once at some
    point in the future.  It will be passed the same *state* value as was
    passed with it to ``PyDef_AddCallback``.  It is up to the *callback_fn*
    to deal with the memory management of this *state* object.
  - The *callback_fn* may be ``NULL`` if no callback is required.  But in
    this case ``PyDef_AddCallback`` must still be called to notify the
    ``PyExc_Deferred`` that the C function is micro-threading aware.
  - This returns 0 if it fails (is acting like an exception), 1 otherwise.
    If it fails, the caller should do any needed clean up because the caller
    won't be resumed by the ``PyExc_Deferred`` (i.e., *callback_fn* will not
    be called).

- ``int PyDef_AddCallback(const char *caller_name, const char *called_name,
  PyObject *(*callback_fn)(PyObject *returned_object, void *state),
  void *state)``

  - Same as ``PyDef_AddCallbackEx``, except that the deferred object is 
taken
    from the *value* object returned by ``PyErr_Fetch``.  If the *type*
    returned by ``PyErr_Fetch`` is not ``PyDef_Deferred``, 0 is returned.
    Thus, this function can be called after any exception and then other
    standard exception processing done if 0 is returned (including checking
    for other kinds of exceptions).

- ``int PyDef_FinalEx(PyObject *deferred, const char *called_fn)``

  - Only used by the top-level C function (a reactor) to verify that its
    *called_fn* is micro-threading aware.  Returns 1 if everything looks 
good,
    0 otherwise.  If 0 is returned, then this instance to be treated as an
    exception.

- ``int PyDef_Final(const char *called_fn)``

  - Same as ``PyDef_FinalEx``, except that the deferred object is taken
    from the *value* object returned by ``PyErr_Fetch``.  If the *type*
    returned by ``PyErr_Fetch`` is not ``PyDef_Deferred``, 0 is returned.
    Thus, this function can be called after any exception and then other
    standard exception processing done if 0 is returned (including checking
    for other kinds of exceptions).

- ``int PyDef_IsExceptionEx(PyObject *deferred)``

  - Only used by the top-level C function (a reactor) to determine whether
    to treat the *deferred* as an exception or to do deferred processing.

- ``int PyDef_IsException(void)``

  - Same as ``PyDef_IsExceptionEx``, except that the deferred object is 
taken
    from the *value* object returned by ``PyErr_Fetch``.  If the *type*
    returned by ``PyErr_Fetch`` is not ``PyDef_Deferred``, 1 is returned.
    Thus, this function can be called after any exception and then other
    standard exception processing done if 1 is returned (including checking
    for other kinds of exceptions).

- ``PyObject *PyDef_Callback(PyObject *deferred, PyObject 
*returned_object)``

  - This calls the callback_fn sequence passing *returned_object* to the
    first registered callback_fn, and each callback_fn's returned 
``PyObject``
    to the next registered callback_fn.  The result of the final callback_fn
    is returned (which may be ``NULL`` if an exception was encountered).
  - To signal an exception to the callbacks, first set the error indicator
    (e.g. with ``PyErr_SetString``) and then call ``PyDef_Callback`` passing
    ``NULL`` as the returned_object (just like returning ``NULL`` from a C
    function to signal an exception).
  - If a callback_fn wants to defer execution, this same ``PyDef_Deferred``
    object will be used (since the callback_fn is running in the same
    micro_thread).  The ``PyDef_Deferred`` keeps the newly added 
callback_fns
    in the proper sequence relative the existing callback_fns that have not
    yet been executed.  When ``PyDef_Deferred`` is returned from the
    callback_fn, no further callback_fns are called.

    Note that this check is also done on the starting *returned_object*, so
    that if this ``PyDef_Deferred`` exception is passed in, then none of its
    callback_fns are executed and it simply returns.

  - If this function returns a ``PyDef_Deferred`` exception (itself), no
    ``PyDef_Final`` needs to be done on it, rather a ``PyDef_IsException``
    is done to see whether to treat it as an exception or not.

- ``void PyDef_Abort(PyObject *deferred)``

  - Acts as if ``SystemError`` has been raised to its callback_fns.  
Clears the
    ``SystemError`` and reestablishes any prior error indicator prior to
    returning.
  - If *deferred* is currently registered with a Watcher_, deregister it.

- ``void PyDef_Terminate(PyObject *deferred)``

  - Acts as if ``SystemExit`` has been raised to its callback_fns.  
Clears the
    ``SystemExit`` and reestablishes any prior error indicator prior to
    returning.
  - If *deferred* is currently registered with a Watcher_, deregister it.

Each micro_thread has its own ``PyDef_Deferred`` object associated with it.
This is possible because each micro_thread may only be suspended for one
thing at a time.  This also allows us to re-use ``PyDef_Deferreds`` and,
through the following trick, means that we don't need a lot of
``PyDef_Deferred`` instances when a micro_thread is deferred many times at
different points in the call stack.

One peculiar thing about the stored callbacks, is that they're not really a
queue.  When the ``PyDef_Deferred`` is first used and has no saved 
callbacks,
the callbacks are saved in straight FIFO manor.  Let's say that four
callbacks are saved in this order: ``D'``, ``C'``, ``B'``, ``A'`` (meaning
that ``A`` called ``B``, called ``C``, called ``D`` which deferred):

- after ``D'`` is added, the queue looks like: ``D'``
- after ``C'`` is added, the queue looks like: ``D'``, ``C'``
- after ``B'`` is added, the queue looks like: ``D'``, ``C'``, ``B'``
- after ``A'`` is added, the queue looks like: ``D'``, ``C'``, ``B'``, 
``A'``

Upon resumption, ``D'`` is called, then ``C'`` is called.  ``C'`` then calls
``E`` which calls ``F`` which now wants to defer execution again.  
``B'`` and
``A'`` are still in the deferred's callback queue.  When ``F'``, then ``E'``
then ``C''`` are pushed, they go in front of the callbacks still present
from the last defer:

- after ``F'`` is added, the queue looks like: ``F'``, ``B'``, ``A'``
- after ``E'`` is added, the queue looks like: ``F'``, ``E'``, ``B'``, 
``A'``
- after ``C''`` is added, the queue looks like: ``F'``, ``E'``, ``C''``,
  ``B'``, ``A'``

These callback functions are basically a reflection of the C stack at the
point the micro_thread is deferred.


Reactor
-------

The Reactor logic is divided into two levels:

- The top level function.  There is only one long running invocation of
  this function (per standard thread_).
- A list of Watchers_.  Each of these knows how to watch for a different
  type of external event, such as a file being ready for IO or a signal
  having been received.

.. _Watchers: Watcher_


Top Level
'''''''''

The top level function pops (deferred, returned_object) pairs, doing the
``PyDef_Callback`` on each, until either the ``EventCheckingThreshold``
number of deferreds have been popped, or there are no more deferreds 
scheduled.

It then runs through the ``WatcherList`` (which is maintained in descending
``PyWatch_Priority`` order) to give each watcher_ a chance to poll for its
events.  If there are then still no deferreds scheduled, it goes to each
watcher in turn asking it to do a ``PyWatch_TimedWait`` for
``TimedWaitSeconds`` until one doesn't return -1.  Then it polls the
remaining watchers again and goes back to running scheduled deferreds.

If there is only one watcher, a ``PyWatch_WaitForever`` is used, rather than
first polling with ``PyWatch_Poll`` and then ``PyWatch_TimedWait``.

The top level also manages a list of timers for the watchers.  It calls
``PyWatch_Timeout`` each time a timer pops.

- ``int PyTop_Schedule(PyObject *deferred, PyObject *returned_object)``

  - Returns 0 on error, 1 otherwise.

- ``int PyTop_ScheduleException(PyObject *deferred,
  PyObject *exc_type, PyObject *exc_value, PyObject *exc_traceback)``

  - Returns 0 on error, 1 otherwise.

- ``int PyTop_SetTimer(PyObject *watcher, PyObject *deferred, double 
seconds)``

  - Returns 0 on error, 1 otherwise.

- ``int PyTop_ClearTimer(PyObject *watcher, PyObject *deferred)``

  - Returns 0 on error, 1 otherwise.

- ``int PyTop_SetEventCheckingThreshold(long num_continues)``

  - Returns 0 on error, 1 otherwise.

- ``int PyTop_SetTimedWaitSeconds(double seconds)``

  - Returns 0 on error, 1 otherwise.


Watcher
'''''''

- ``int PyWatch_Priority(PyObject *watcher)``
 
  - Returns the priority of this watcher.  (-1 for error).  Higher numbers
    have higher priorities.

- ``int PyWatch_RegisterDeferred(PyObject *watcher, PyObject *deferred,
  PyObject *wait_reason, double max_wait_seconds)``

  - *Max_wait_seconds* of 0.0 means no time limit.  Otherwise, register
    *deferred* with ``PyTop_SetTimer`` (above).
  - Adds *deferred* to the list of waiting objects, for *wait_reason*.
  - The meaning of *wait_reason* is determined by the watcher.  It can be
    used, for example, to indicate whether to wait for input or output on a
    file.
  - Returns 0 on error, 1 otherwise.

- ``void PyWatch_Defer(PyObject *watcher, PyObject *wait_reason,
  double max_wait_seconds)``

  - Passes the ``PyDef_Deferred`` of the current micro_thread to
    ``PyWatch_RegisterDeferred``, and then raises the ``PyDef_Deferred`` as
    an exception.  *Wait_reason* and *max_wait_seconds* are passed on to
    ``PyWatch_RegisterDeferred``.
  - This function has no return value.  It always generates an exception.

- ``int PyWatch_Poll(PyObject *watcher)``

  - Poll for events and schedule the appropriate ``PyDef_Deferreds``.  
Do not
    cause the process to be put to sleep.  Return 0 on error, 1 on success
    (whether or not any events were discovered).

- ``int PyWatch_TimedWait(PyObject *watcher, double seconds)``

  - Wait for events and schedule the appropriate ``PyDef_Deferreds``.  
Do not
    cause the process to be put to sleep for more than the indicated number
    of *seconds*.  Return -1 if *watcher* is not capable of doing timed
    sleeps, 0 on error, 1 on success (whether or not any events were
    discovered).  Return a 1 if the wait was terminated due to the process
    having received a signal.
  - If *watcher* is not capable of doing timed waits, it does a poll and
    returns -1.

- ``int PyWatch_WaitForever(PyObject *watcher)``

  - Suspend the process until an event occurs and schedule the appropriate
    ``PyDef_Deferreds``.  The process may be put to sleep indefinitely.
    Return 0 on error, 1 on success (whether or not any ``PyDef_Deferreds``
    were scheduled).  Return a 1 if the wait was terminated due to the 
process
    having received a signal.

- ``int PyWatch_Timeout(PyObject *watcher, PyObject *deferred)``
 
  - Called by top level when the timer set by ``PyTop_SetTimer`` expires.
  - Passes a ``TimeoutException`` to the deferred using ``PyDef_Callback``.
  - Return 0 on error, 1 otherwise.

- ``int PyWatch_DeregisterDeferred(PyObject *watcher, PyObject *deferred,
  PyObject *returned_object)``

  - Deregisters *deferred*.
  - Passes *returned_object* to *deferred* using ``PyDef_Callback``.
  - *Returned_object* may be ``NULL`` to indicate an exception to the 
callbacks.
  - Returns 0 on error, 1 otherwise.


Specification of Python Layer Enhancements
==========================================

Fortunately, at the python level, the programmer does not see deferred,
reactor, or watcher objects.  The python programmer will see three things:

#. An addition of non_blocking modes of accessing files, sockets, time.sleep
   and other functions that may block.  It is not clear yet exactly what 
these
   will look like.  The possibilities are:

   - Add an argument to the object creation functions to specify blocking or
     non-blocking.
   - Add an operation to change the blocking mode after the object has been
     created.
   - Add new non-blocking versions of the methods on the objects that may
     block (e.g., read_d/write_d/send_d/recv_d/sleep_d).
   - Some combination of these.

   If an object is used in blocking mode, then all microthreads (within its
   Posix thread_) will block.  So the python programmer must set 
non-blocking
   mode on these objects as a first step to take advantage of 
micro-threading.

#. Micro_thread objects.  Each of these will have a re-usable
   ``PyDef_Deferred`` object attached to it, since each micro_thread can 
only
   be suspended waiting for one thing at a time.  The current micro_thread
   would be stored within a C global variable, much like
   _PyThreadState_Current.  If the python programmer isn't interested in
   micro_threading, micro_threads can be safely ignored (like posix 
threads_,
   you get one for free, but don't have to be aware of it).  If the
   programmer *is* interested in micro-threading, then s/he must create
   micro_threads.  This would be done with::

       micro_thread(function, *args, **kwargs)

   I am thinking that there are three usage scenarios:

   #. Create a micro-thread to do something, without regard to any final
      return value from *function*.  An example here would be a web server
      that has a top-level ``socket.accept`` loop that runs a
      ``handle_client`` function on each new connection.  Once launched,
      the ``socket.accept`` thread is no longer interested in the
      ``handle_client`` threads.

      In this case, the normal return value of the ``handle_client`` 
function
      can be discarded.  But what should be done with exceptions that 
are not
      caught in the child threads?

      Therefore, this style of use would be indicated by providing an
      top-level exception handler for the new thread as a keyword argument,
      e.g. ``exception_handler=traceback.print_exception`` in a developer
      environment and ``exception_handler=my_exception_logger`` in a
      production environment.

      If this keyword argument is provided, then the parent thread does not
      need to do any kind of *wait* after the child thread is complete.  It
      will either complete normally and go away silently, or raise an 
uncaught
      exception, which is passed to the indicated exception_handler, and 
then
      go away with no more ado.

   #. Create micro_threads to run multiple long-running *functions* in
      parallel where the final return value from each *function* is 
needed in
      the parent thread.

      In this case, the ``exception_handler`` argument is not specified 
and the
      parent thread needs to *wait* on the child thread (when the parent is
      ready to do so).  Thus, completed micro_threads will form zombie
      threads until their parents retrieve their final return values (much
      like unix processes).

      This ends up being a kind of parallel execution strategy and it 
might be
      nice to have a ``threaded_map`` function that will create a 
micro_thread
      for each element of its *iterable* argument in order to run the
      *function* on them in parallel and then return an iterable of the
      waited for results.

      On doing the *wait*, an uncaught exception in the child 
micro_thread is
      re-raised in the parent micro_thread.

   #. In the above examples, the child micro_threads are completely
      independent of each other.  This final scenario uses *micro_pipes* to
      allow threads to cooperatively solve problems (much like unix pipes).

#. Micro_pipes.  Micro_pipes are one-way pipes that allow synchronized
   communication between micro_threads.
   
   The protocol for the receiving side of the pipe is simply the standard
   python iterator protocol.

   The sending side has these methods:
   
   - ``put(object)`` to send *object* to the receiving side (retrieved with
     the ``__next__`` method).
   - ``take_from(iterable)`` to send a series of objects to the 
receiving side
     (retrieved with multiple ``__next__`` calls).
   - ``close()`` causes ``StopIteration`` on the next ``__next__`` call.
     A ``put`` done after a ``close`` silently terminates the micro_thread
     doing the ``put`` (in case the receiving side closes the micro_pipe).

   Micro_pipes are automatically associated with micro_threads, making 
it less
   likely to hang the program:

   >>> pipe = micro_pipe()
   >>> next(pipe)  # hangs the program!  No micro_thread created to feed 
pipe...

   So each micro_thread will automatically have a stdout micro_pipe assigned
   to it and can (optionally) be assigned a stdin micro_pipe (some other
   micro_thread's stdout micro_pipe).  When the micro_thread terminates, it
   automatically calls ``close`` on its stdout micro_pipe.

   To access the stdout micro_pipe of the current micro_thread, new ``put``
   and ``take_from`` built-in functions are provided.

   Micro_pipes lets us write generator functions in a new way by having the
   generator do ``put(object)`` rather than ``yield object``.  In this case,
   the generator function has no ``yield`` statement, so is not treated
   specially by the compiler.  Basically this means that calling a new-style
   generator does not automatically create a new micro_thread (sort of what
   calling an old-style generator does).

   The ``put(object)`` does the same thing as ``yield object``, but
   allows the generator to share the micro_pipe with other new-style
   generators functions (by simply calling them) and old-style 
generators (or
   any iterable) by calling ``take_from`` on them.  This lets the generator
   delegate to other generators without having to get involved with passing
   the results back to its caller.

   For example, a generator to output all the even numbers from 1-n,
   followed by all of the odd numbers::

       def even_odd(n):
           take_from(range(2, n, 2))
           take_from(range(1, n, 2))

   These "new-style" generators would have to be run in their own
   micro_thread:

   >>> pipe = micro_thread(even_odd, 100).stdout
   >>> # now pipe is an iterable representing the generator:
   >>> print tuple(pipe)

   But the generator is then not restricted to running within its own
   micro_thread.  It could also sometimes be used as a helper by other
   generators within their micro_thread.  This would allow generators to
   still use each other as helpers.  For example::

       def even(n):
           take_from(range(2, n, 2))

       def odd(n):
           take_from(range(1, n, 2))

       def even_odd(n):
           even(n)
           odd(n)

   At this point a micro_thread may be created on any of the above 
generators.


Open Questions
==============

#. How are exceptions propagated from one ``PyDef_Deferred`` to the next?

   - This would happen when the final result of a micro_thread is needed by
     another micro_thread.  This happens in two cases:

     - When a *wait* is done.  In this case the exception is propagated 
to the
       caller.
     - When dealing with pipes, it seems that an exception on the 
``put`` side
       should be propagated to the ``__next__`` side.  This is another 
reason
       to have the stdout pipe associated with the micro_thread.  
Because when
       the exception occurs, it will not be in the ``put`` call; so without
       attaching the pipe to the micro_thread, there would be no way of
       knowing which pipe that micro_thread was outputting to.

       Thus exception would propagate from the ``put`` side of a 
micro_pipe to
       the ``__next__`` side, but not the other direction.

#. How are tracebacks handled?
#. Do we:

   #. Treat each python-to-python call as a separate C call, with it's own
      callback_fn?
   #. Only register one callback_fn for each continuous string of
      python-to-python calls and then process them iteratively rather than
      recursively in the callback_fn (but not in the original calls)?
   #. Treat python-to-python calls iteratively both in the original calls
      and in the callback_fn?

#. How is process termination handled?
   
   - I guess we can keep a list of micro_threads and terminate each of them.
     There's a question of whether to allow the micro_threads to complete or
     to abort them mid-stream.  Kind of like a unix shutdown.  Maybe two 
kinds
     of process termination?

#. How does this interact with the existing posix thread_ package?

   - Each micro_thread would be associated with a posix thread.  Or,
     conversely, each posix thread would have its own list of micro_threads.

#. How does this impact the debugger/profiler/sys.settrace?
#. Should functions (C and python) that may defer be indicated with some
   naming convention (e.g., ends in '_d') to make it easier for programmers
   to avoid them within their critical sections of code (in terms of
   synchronization)?


Rationale
=========

The implementation is done by treating the C-level deferreds as a special
case of C-level exceptions so that the new ``PyDef_Deferred`` objects 
will be
be treated like any other exception if *any* C function within the call 
chain
hasn't been modified to deal with them.  In this case, the normal execution
of the program is interrupted, but in a well understood way (by an 
exception)
with the name of the offending C function contained in the exception message
so that the python developer knows where to go to fix it.

Only when *all* C functions in the call chain properly recognize and 
deal with
the ``PyDef_Deferred`` is the new deferred object applied to implement the
new micro-threading behavior.

No change is required for C functions that could never get a
``PyDef_Deferred``.

This takes advantage of the fact that the current exception mechanism 
already
unwinds the C stack.  It also adds deferred processing without adding
additional checks after each C function call to see whether to defer
execution.  The check that is already being done for exceptions doubles as a
check for deferred processing.


Other Approaches
================

Here's a brief comparison to other approaches to micro-threading in python:

- `Stackless Python`_ [#stackless]_
 
  - As near as I can tell, stackless went through two incarnations:
    
    #. The first incarnation involved an implementation of Frame 
continuations
       which were then used to provide the rest of the stackless 
functionality.
       
       - A new ``Py_UnwindToken`` was created to unwind the C stack.  
This is
         similar to the new ``PyDef_Deferred`` proposed in this PEP, except
         that ``Py_UnwindToken`` is treated as a special case of a normal
         ``PyObject`` return value, while the ``PyDef_Deferred`` is treated
         as a special case of a normal exception.

         Consider the case where unmodified C function ``A`` calls 
modified C
         function ``B``, and function ``B`` wants to defer execution.  In
         both cases function ``B`` returns a special value indicating it
         wants to defer.

         The difference comes in how function ``A`` (which isn't prepared to
         cooperate with this new type of request) handles this situation.

         With the stackless ``Py_UnwindToken`` as a return value, function
         ``A`` tries to act on ``Py_UnwindToken`` as an ordinary return 
value,
         which it is not.  This may or may not lead to some other exception
         being thrown due to a type inconsistency between ``Py_UnwindToken``
         and what was expected by function ``A``.  It also means that 
further
         up the call chain in the code that called function ``A``, the fact
         that a ``Py_UnwindToken`` was returned isn't known if function 
``A``
         does not simply return it.  In short, who knows what will happen...

         But this PEP treats the request to defer as a special exception.
         So function ``A``, receiving a ``PyDef_Deferred`` exception will
         treat it as an ordinary exception.  Function ``A``, not recognizing
         this exception, will perform any required clean up and simply
         forward the ``PyDef_Deferred`` on to its caller (re-raise it).
         The top-level code that receives this ``PyDef_Deferred`` knows that
         it's a broken ``PyDef_Deferred`` and raises it as a normal 
exception,
         which will state "Deferred execution not yet implemented by A".
         This makes much more sense.

       - Another difference between the two styles of continuations is that
         the stackless continuation is designed to be able to be continued
         multiple times.  In other words, you can continue the execution of
         the program from the point the continuation was made as many times
         as you wish, passing different seed values each time.

         The ``PyDef_Deferred`` described in this PEP (like the Twisted
         Deferred) is designed to be continued oonce.
      
       - The stackless approach provides a python-level continuation
         mechanism (at the Frame level) that only makes python functions
         continuable.  It provides no way for C functions to register
         continuations so that C functions can be unwound from the stack
         and later continued (other than those related to the byte code
         interpreter).

         In contrast, this PEP proposes a C-level continuation mechanism
         very similar to the Twisted Deferred.  Each C function registers a
         callback to be run when the Deferred is continued.  From this
         perspective, the byte code interpreter is just another C function.

    #. The second incarnation involved a way of hacking the underlying C
       stack to copy it and later restore it as a means of continuing the
       execution.

       - This doesn't appear to be portable to different CPU/C Compiler
         configurations.
       - This doesn't deal with other global state (global/static variables,
         file pointers, etc) that may also be used by this saved stack.
       - In contrast, this PEP uses a single C stack and makes no 
assumptions
         about the underlying C stack implementation.  It is completely
         portable to any CPU/C compiler configuration.

- `Implementing "weightless threads" with Python generators`_ [#weightless]_

  - This requires you code each thread as generators.  The generator
    executes a 'yield' to relinquish control.
  - It's not clear how this scales.  It seems that to pause in a lower
    python function, it and all intermediate functions must be generators.

- python-safethread_ [#safethread]_

  - This is an alternate implementation to thread_ that adds monitors for
    mutable types, deadlock detection, improves exception propagation
    across threads and program finalization, and removes the GIL lock.  As
    such, it is not a "micro" threading approach, though by removing the GIL
    lock it may be able to better use multiple processor configurations
    than the approach proposed in this PEP.

- `Sandboxed Threads in Python`_ [#sandboxed-threads]_

  - Another alternate implementation to thread_, this one only shares
    immutable objects between threads, modifying the referencing counting
    system to avoid synchronization issues with the reference count for
    shared objects.  Again, not a "micro" threading approach, but perhaps
    also better with multiple processors.

.. _Implementing "weightless threads" with Python generators:
   http://www.ibm.com/developerworks/library/l-pythrd.html
.. _python-safethread: https://launchpad.net/python-safethread
.. _Sandboxed Threads in Python:
   http://mail.python.org/pipermail/python-dev/2005-October/057082.html
.. _Stackless Python: http://www.stackless.com/
.. _thread: http://docs.python.org/lib/module-thread.html
.. _threading: http://docs.python.org/lib/module-threading.html


Backwards Compatibility
=======================

This PEP doesn't break any existing code.  Existing code just won't take
advantage of any of the new features.

But there are two possible problem areas:

#. Code uses micro-threading, but then causes an unmodified C function
   to call a modified C function which tries to defer execution.

   In this case an exception will be generated stating that the unmodified C
   function needs to be converted before this program will work.

#. Code originally written in a single threaded environment is now used in a
   micro-threaded environment.  The old code was not written taking
   synchronization issues into account, which may cause problems if the old
   code calls a function which causes it to defer in the middle of its
   critical section.  This could cause very strange behavior, but can't
   result in any C-level errors (e.g., segmentation violation).

   This old code would have to be fixed to run with the new features.  I
   expect that this will not be a frequent problem.


References
==========

.. [#twisted-fn] Twisted, Twisted Matrix Labs
   (http://twistedmatrix.com/trac/)
.. [#c_api] Python/C API Reference Manual, Rossum
   (http://docs.python.org/api/api.html)
.. [#stackless] Stackless Python, Tismer
   (http://www.stackless.com/)
.. [#thread-module] thread -- Multiple threads of control
   (http://docs.python.org/lib/module-thread.html)
.. [#threading-module] threading -- Higher-level threading interface
   (http://docs.python.org/lib/module-threading.html)
.. [#weightless] Charming Python: Implementing "weightless threads" with
   Python generators, Mertz
   (http://www.ibm.com/developerworks/library/l-pythrd.html)
.. [#safethread] Threading extensions to the Python Language,
   (https://launchpad.net/python-safethread)
.. [#sandboxed-threads] Sandboxed Threads in Python, Olsen
   (http://mail.python.org/pipermail/python-dev/2005-October/057082.html)


Copyright
=========

This document has been placed in the public domain.



From josiah.carlson at gmail.com  Sun Aug  3 07:20:51 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Sat, 2 Aug 2008 22:20:51 -0700
Subject: [Python-ideas] New PEP proposal: C Micro-Threading (ala
	Twisted) (long!)
In-Reply-To: <4894B36F.4030805@gmail.com>
References: <4894B36F.4030805@gmail.com>
Message-ID: <e6511dbf0808022220p761d7b8ex132730fbe9be91@mail.gmail.com>

A few comments...

Firstly, this is a great PEP, you've obviously put a lot of thought
and effort into getting it together.

Secondly, isn't the greenlet package already available online for
current Python revisions?  If so, has it gained significant numbers of
users?

Regardless of the answer to the above, how does the current greenlet
package plug into Python without having API hooks?  Does it execute an
alternate mainloop that does the stack switching?

 - Josiah

P.S. After having used event-driven frameworks for quite a few years
(asyncore derived socket servers and clients, wxPython,
publish/subscribe architectures, ...), they really aren't all that
bad.  Then again, I don't find threading all that bad either (I try to
stick with simple queues, explicitly synchronized interfaces, etc.,
which reduces complexity by a few orders of magnitude).

On Sat, Aug 2, 2008 at 12:20 PM, Bruce Frederiksen <dangyogi at gmail.com> wrote:
> Here is a proposal for making the benefits of Twisted available without
> having to write python code in an event driven style.
>
> (I'm not sure whether this list accepts HTML, so am just copying plain
> text -- you may want to run rst2html on this)...
>
> I look forward to your ideas and feedback!
>
> -bruce
>
>
> Abstract
> ========
>
> This PEP adds micro-threading (or `green threads`_) at the C level so that
> micro-threading is built in and can be used with very little coding effort
> at the python level.
>
> The implementation is quite similar to the Twisted_ [#twisted-fn]_
> Deferred_/Reactor_ model, but applied at the C level by extending the
> `C API`_ [#c_api]_ slightly.  Doing this provides the Twisted
> capabilities to python, without requiring the python programmer to code
> in the Twisted event driven style.  Thus, legacy python code would gain the
> benefits that Twisted provides with very little modification.
>
> Burying the event driven mechanism in the C level should also give the same
> benefits to python GUI interface tools so that the python programmers don't
> have to deal with event driven programming there either.
>
> This capability may also be used to provide some of the features that
> `Stackless Python`_ [#stackless]_ provides, such as microthreads and
> channels (here, called micro_pipes).
>
> .. _Twisted: http://twistedmatrix.com/trac/
> .. _Deferred:
>  http://twistedmatrix.com/projects/core/documentation/howto/defer.html
> .. _Reactor:
>  http://twistedmatrix.com/projects/core/documentation/howto/reactor-basics.html
> .. _C API: http://docs.python.org/api/api.html
> .. _green threads: http://en.wikipedia.org/wiki/Green_threads
>
> Motivation
> ==========
>
> The popularity of the Twisted project has demonstrated the need for
> micro-threads as an alternative the standard thread_ [#thread-module]_ and
> threading_ [#threading-module]_ packages.  These allow large numbers
> (1000's)
> of simultaneous connections to python servers, as well as fan-outs to large
> numbers of downstream connections.
>
> The advantages to the Twisted approach over standard threads are:
>
> #. much less memory is required per thread
> #. faster thread creation
> #. faster context switching (I'm guessing on this one, is this really true?)
> #. synchronization between threads is easier because there is no preemption,
>  making it much easier to write critical sections of code.
>
> The disadvantages are:
>
> #. the python developer must write his/her program in an event driven style
> #. the approach can not be used with standard python code that wasn't
>  written in this event driven style
> #. the approach does not take advantage of multiple processor architectures
> #. since there is no preemption, a long running micro-thread will starve
>  other micro-threads
>
> This PEP attempts to retain all of the advantages that Twisted has
> demonstrated, but resolve the first two disadvantages to make these
> advantages accessible to all python programs, including legacy programs
> not written in the Twisted style.  This should make it very easy for legacy
> programs like WSGI apps, Django and TurboGears to reap the Twisted benefits.
>
> Another example of event driven mechanisms are the GUI/windows events.  This
> PEP would also make it easy for python GUI interface toolkits (like wxpython
> and qtpython) to hide the GUI/windows event driven style of programming from
> the python programmer.  For example, you would no longer need to use modal
> dialog boxes just to make the programming easier.
>
> This PEP does not address the last two disadvantages, and thus also has
> these disadvantages itself.
>
>
> Specification of C Layer Enhancements
> =====================================
>
> Deferred
> --------
>
> ``PyDef_Deferred`` is written as a new exception type for use by the C code
> to defer execution.  This is a subclass of ``NotImplementedError``.
>  Instances
> are not raised as a normal exception (e.g., with ``PyErr_SetObject``), but
> by
> calling ``PyWatch_Defer``.  This registers the ``PyDef_Deferred`` associated
> with the currently running micro_thread as the current error object, but
> also
> readies it for its primary job -- deferring execution.  As an exception,
> it creates its own error message, if needed, which is "Deferred execution
> not yet implemented by %s" % c_function_name.
>
> ``PyErr_ExceptionMatches`` may be used with these.  This allows them to be
> treated as exceptions by non micro-threading aware (unmodified) C functions.
>
> But these ``PyDef_Deferred`` objects are special indicators that are treated
> differently than normal exceptions by micro-threading aware (modified) C
> code.  Modified C functions do this by calling ``PyDef_AddCallback``,
> ``PyDef_Final`` or explicitly checking
> ``PyErr_ExceptionMatches(PyDef_Deferred)`` after receiving an error return
> status from a called function.
>
> ``PyDef_Deferred`` instances offer the following methods (in addition to the
> normal exception methods):
>
> - ``int PyDef_AddCallbackEx(PyObject *deferred, const char *caller_name,
>  const char *called_name, PyObject *(*callback_fn)(PyObject
> *returned_object,
>  void *state), void *state)``
>
>  - The *caller_name* and *called_name* are case sensitive.  The
> *called_name*
>   must match exactly the *caller_name* used by the called function when it
>   dealt with this ``PyExc_Deferred``.  If the names are different, the
>   ``PyExc_Deferred`` knows that an intervening unmodified C function was
>   called.  This is what triggers it to act like an exception.
>
>   The *called_name* must be ``NULL`` when called by the function that
>   executed the ``PyWatch_Defer`` to defer execution.
>
>  - The *callback_fn* will be called with the ``PyObject`` of the results of
>   the prior registered callback_fn.  An exception is passed to
>   *callback_fn* by setting the exception and passing ``NULL`` (just like
>   returning an exception from a C function).  In the case that the
>   ``PyExc_Deferred`` initially accepts *callback_fn* and then later has
>   to reject it (because of the exception case, above), it will pass a
>   ``SystemError`` to all registered callback_fns to allow them to clean up.
>   But this ``SystemError`` is only a "behind the scenes" measure that will
>   only be seen by these callback_fns.  It will be cleared and the prior
>   error indicator reestablished before ``PyDef_AddCallback`` returns.
>  - The *callback_fn* is always guaranteed to be called exactly once at some
>   point in the future.  It will be passed the same *state* value as was
>   passed with it to ``PyDef_AddCallback``.  It is up to the *callback_fn*
>   to deal with the memory management of this *state* object.
>  - The *callback_fn* may be ``NULL`` if no callback is required.  But in
>   this case ``PyDef_AddCallback`` must still be called to notify the
>   ``PyExc_Deferred`` that the C function is micro-threading aware.
>  - This returns 0 if it fails (is acting like an exception), 1 otherwise.
>   If it fails, the caller should do any needed clean up because the caller
>   won't be resumed by the ``PyExc_Deferred`` (i.e., *callback_fn* will not
>   be called).
>
> - ``int PyDef_AddCallback(const char *caller_name, const char *called_name,
>  PyObject *(*callback_fn)(PyObject *returned_object, void *state),
>  void *state)``
>
>  - Same as ``PyDef_AddCallbackEx``, except that the deferred object is taken
>   from the *value* object returned by ``PyErr_Fetch``.  If the *type*
>   returned by ``PyErr_Fetch`` is not ``PyDef_Deferred``, 0 is returned.
>   Thus, this function can be called after any exception and then other
>   standard exception processing done if 0 is returned (including checking
>   for other kinds of exceptions).
>
> - ``int PyDef_FinalEx(PyObject *deferred, const char *called_fn)``
>
>  - Only used by the top-level C function (a reactor) to verify that its
>   *called_fn* is micro-threading aware.  Returns 1 if everything looks good,
>   0 otherwise.  If 0 is returned, then this instance to be treated as an
>   exception.
>
> - ``int PyDef_Final(const char *called_fn)``
>
>  - Same as ``PyDef_FinalEx``, except that the deferred object is taken
>   from the *value* object returned by ``PyErr_Fetch``.  If the *type*
>   returned by ``PyErr_Fetch`` is not ``PyDef_Deferred``, 0 is returned.
>   Thus, this function can be called after any exception and then other
>   standard exception processing done if 0 is returned (including checking
>   for other kinds of exceptions).
>
> - ``int PyDef_IsExceptionEx(PyObject *deferred)``
>
>  - Only used by the top-level C function (a reactor) to determine whether
>   to treat the *deferred* as an exception or to do deferred processing.
>
> - ``int PyDef_IsException(void)``
>
>  - Same as ``PyDef_IsExceptionEx``, except that the deferred object is taken
>   from the *value* object returned by ``PyErr_Fetch``.  If the *type*
>   returned by ``PyErr_Fetch`` is not ``PyDef_Deferred``, 1 is returned.
>   Thus, this function can be called after any exception and then other
>   standard exception processing done if 1 is returned (including checking
>   for other kinds of exceptions).
>
> - ``PyObject *PyDef_Callback(PyObject *deferred, PyObject
> *returned_object)``
>
>  - This calls the callback_fn sequence passing *returned_object* to the
>   first registered callback_fn, and each callback_fn's returned ``PyObject``
>   to the next registered callback_fn.  The result of the final callback_fn
>   is returned (which may be ``NULL`` if an exception was encountered).
>  - To signal an exception to the callbacks, first set the error indicator
>   (e.g. with ``PyErr_SetString``) and then call ``PyDef_Callback`` passing
>   ``NULL`` as the returned_object (just like returning ``NULL`` from a C
>   function to signal an exception).
>  - If a callback_fn wants to defer execution, this same ``PyDef_Deferred``
>   object will be used (since the callback_fn is running in the same
>   micro_thread).  The ``PyDef_Deferred`` keeps the newly added callback_fns
>   in the proper sequence relative the existing callback_fns that have not
>   yet been executed.  When ``PyDef_Deferred`` is returned from the
>   callback_fn, no further callback_fns are called.
>
>   Note that this check is also done on the starting *returned_object*, so
>   that if this ``PyDef_Deferred`` exception is passed in, then none of its
>   callback_fns are executed and it simply returns.
>
>  - If this function returns a ``PyDef_Deferred`` exception (itself), no
>   ``PyDef_Final`` needs to be done on it, rather a ``PyDef_IsException``
>   is done to see whether to treat it as an exception or not.
>
> - ``void PyDef_Abort(PyObject *deferred)``
>
>  - Acts as if ``SystemError`` has been raised to its callback_fns.  Clears
> the
>   ``SystemError`` and reestablishes any prior error indicator prior to
>   returning.
>  - If *deferred* is currently registered with a Watcher_, deregister it.
>
> - ``void PyDef_Terminate(PyObject *deferred)``
>
>  - Acts as if ``SystemExit`` has been raised to its callback_fns.  Clears
> the
>   ``SystemExit`` and reestablishes any prior error indicator prior to
>   returning.
>  - If *deferred* is currently registered with a Watcher_, deregister it.
>
> Each micro_thread has its own ``PyDef_Deferred`` object associated with it.
> This is possible because each micro_thread may only be suspended for one
> thing at a time.  This also allows us to re-use ``PyDef_Deferreds`` and,
> through the following trick, means that we don't need a lot of
> ``PyDef_Deferred`` instances when a micro_thread is deferred many times at
> different points in the call stack.
>
> One peculiar thing about the stored callbacks, is that they're not really a
> queue.  When the ``PyDef_Deferred`` is first used and has no saved
> callbacks,
> the callbacks are saved in straight FIFO manor.  Let's say that four
> callbacks are saved in this order: ``D'``, ``C'``, ``B'``, ``A'`` (meaning
> that ``A`` called ``B``, called ``C``, called ``D`` which deferred):
>
> - after ``D'`` is added, the queue looks like: ``D'``
> - after ``C'`` is added, the queue looks like: ``D'``, ``C'``
> - after ``B'`` is added, the queue looks like: ``D'``, ``C'``, ``B'``
> - after ``A'`` is added, the queue looks like: ``D'``, ``C'``, ``B'``,
> ``A'``
>
> Upon resumption, ``D'`` is called, then ``C'`` is called.  ``C'`` then calls
> ``E`` which calls ``F`` which now wants to defer execution again.  ``B'``
> and
> ``A'`` are still in the deferred's callback queue.  When ``F'``, then ``E'``
> then ``C''`` are pushed, they go in front of the callbacks still present
> from the last defer:
>
> - after ``F'`` is added, the queue looks like: ``F'``, ``B'``, ``A'``
> - after ``E'`` is added, the queue looks like: ``F'``, ``E'``, ``B'``,
> ``A'``
> - after ``C''`` is added, the queue looks like: ``F'``, ``E'``, ``C''``,
>  ``B'``, ``A'``
>
> These callback functions are basically a reflection of the C stack at the
> point the micro_thread is deferred.
>
>
> Reactor
> -------
>
> The Reactor logic is divided into two levels:
>
> - The top level function.  There is only one long running invocation of
>  this function (per standard thread_).
> - A list of Watchers_.  Each of these knows how to watch for a different
>  type of external event, such as a file being ready for IO or a signal
>  having been received.
>
> .. _Watchers: Watcher_
>
>
> Top Level
> '''''''''
>
> The top level function pops (deferred, returned_object) pairs, doing the
> ``PyDef_Callback`` on each, until either the ``EventCheckingThreshold``
> number of deferreds have been popped, or there are no more deferreds
> scheduled.
>
> It then runs through the ``WatcherList`` (which is maintained in descending
> ``PyWatch_Priority`` order) to give each watcher_ a chance to poll for its
> events.  If there are then still no deferreds scheduled, it goes to each
> watcher in turn asking it to do a ``PyWatch_TimedWait`` for
> ``TimedWaitSeconds`` until one doesn't return -1.  Then it polls the
> remaining watchers again and goes back to running scheduled deferreds.
>
> If there is only one watcher, a ``PyWatch_WaitForever`` is used, rather than
> first polling with ``PyWatch_Poll`` and then ``PyWatch_TimedWait``.
>
> The top level also manages a list of timers for the watchers.  It calls
> ``PyWatch_Timeout`` each time a timer pops.
>
> - ``int PyTop_Schedule(PyObject *deferred, PyObject *returned_object)``
>
>  - Returns 0 on error, 1 otherwise.
>
> - ``int PyTop_ScheduleException(PyObject *deferred,
>  PyObject *exc_type, PyObject *exc_value, PyObject *exc_traceback)``
>
>  - Returns 0 on error, 1 otherwise.
>
> - ``int PyTop_SetTimer(PyObject *watcher, PyObject *deferred, double
> seconds)``
>
>  - Returns 0 on error, 1 otherwise.
>
> - ``int PyTop_ClearTimer(PyObject *watcher, PyObject *deferred)``
>
>  - Returns 0 on error, 1 otherwise.
>
> - ``int PyTop_SetEventCheckingThreshold(long num_continues)``
>
>  - Returns 0 on error, 1 otherwise.
>
> - ``int PyTop_SetTimedWaitSeconds(double seconds)``
>
>  - Returns 0 on error, 1 otherwise.
>
>
> Watcher
> '''''''
>
> - ``int PyWatch_Priority(PyObject *watcher)``
>
>  - Returns the priority of this watcher.  (-1 for error).  Higher numbers
>   have higher priorities.
>
> - ``int PyWatch_RegisterDeferred(PyObject *watcher, PyObject *deferred,
>  PyObject *wait_reason, double max_wait_seconds)``
>
>  - *Max_wait_seconds* of 0.0 means no time limit.  Otherwise, register
>   *deferred* with ``PyTop_SetTimer`` (above).
>  - Adds *deferred* to the list of waiting objects, for *wait_reason*.
>  - The meaning of *wait_reason* is determined by the watcher.  It can be
>   used, for example, to indicate whether to wait for input or output on a
>   file.
>  - Returns 0 on error, 1 otherwise.
>
> - ``void PyWatch_Defer(PyObject *watcher, PyObject *wait_reason,
>  double max_wait_seconds)``
>
>  - Passes the ``PyDef_Deferred`` of the current micro_thread to
>   ``PyWatch_RegisterDeferred``, and then raises the ``PyDef_Deferred`` as
>   an exception.  *Wait_reason* and *max_wait_seconds* are passed on to
>   ``PyWatch_RegisterDeferred``.
>  - This function has no return value.  It always generates an exception.
>
> - ``int PyWatch_Poll(PyObject *watcher)``
>
>  - Poll for events and schedule the appropriate ``PyDef_Deferreds``.  Do not
>   cause the process to be put to sleep.  Return 0 on error, 1 on success
>   (whether or not any events were discovered).
>
> - ``int PyWatch_TimedWait(PyObject *watcher, double seconds)``
>
>  - Wait for events and schedule the appropriate ``PyDef_Deferreds``.  Do not
>   cause the process to be put to sleep for more than the indicated number
>   of *seconds*.  Return -1 if *watcher* is not capable of doing timed
>   sleeps, 0 on error, 1 on success (whether or not any events were
>   discovered).  Return a 1 if the wait was terminated due to the process
>   having received a signal.
>  - If *watcher* is not capable of doing timed waits, it does a poll and
>   returns -1.
>
> - ``int PyWatch_WaitForever(PyObject *watcher)``
>
>  - Suspend the process until an event occurs and schedule the appropriate
>   ``PyDef_Deferreds``.  The process may be put to sleep indefinitely.
>   Return 0 on error, 1 on success (whether or not any ``PyDef_Deferreds``
>   were scheduled).  Return a 1 if the wait was terminated due to the process
>   having received a signal.
>
> - ``int PyWatch_Timeout(PyObject *watcher, PyObject *deferred)``
>
>  - Called by top level when the timer set by ``PyTop_SetTimer`` expires.
>  - Passes a ``TimeoutException`` to the deferred using ``PyDef_Callback``.
>  - Return 0 on error, 1 otherwise.
>
> - ``int PyWatch_DeregisterDeferred(PyObject *watcher, PyObject *deferred,
>  PyObject *returned_object)``
>
>  - Deregisters *deferred*.
>  - Passes *returned_object* to *deferred* using ``PyDef_Callback``.
>  - *Returned_object* may be ``NULL`` to indicate an exception to the
> callbacks.
>  - Returns 0 on error, 1 otherwise.
>
>
> Specification of Python Layer Enhancements
> ==========================================
>
> Fortunately, at the python level, the programmer does not see deferred,
> reactor, or watcher objects.  The python programmer will see three things:
>
> #. An addition of non_blocking modes of accessing files, sockets, time.sleep
>  and other functions that may block.  It is not clear yet exactly what these
>  will look like.  The possibilities are:
>
>  - Add an argument to the object creation functions to specify blocking or
>    non-blocking.
>  - Add an operation to change the blocking mode after the object has been
>    created.
>  - Add new non-blocking versions of the methods on the objects that may
>    block (e.g., read_d/write_d/send_d/recv_d/sleep_d).
>  - Some combination of these.
>
>  If an object is used in blocking mode, then all microthreads (within its
>  Posix thread_) will block.  So the python programmer must set non-blocking
>  mode on these objects as a first step to take advantage of micro-threading.
>
> #. Micro_thread objects.  Each of these will have a re-usable
>  ``PyDef_Deferred`` object attached to it, since each micro_thread can only
>  be suspended waiting for one thing at a time.  The current micro_thread
>  would be stored within a C global variable, much like
>  _PyThreadState_Current.  If the python programmer isn't interested in
>  micro_threading, micro_threads can be safely ignored (like posix threads_,
>  you get one for free, but don't have to be aware of it).  If the
>  programmer *is* interested in micro-threading, then s/he must create
>  micro_threads.  This would be done with::
>
>      micro_thread(function, *args, **kwargs)
>
>  I am thinking that there are three usage scenarios:
>
>  #. Create a micro-thread to do something, without regard to any final
>     return value from *function*.  An example here would be a web server
>     that has a top-level ``socket.accept`` loop that runs a
>     ``handle_client`` function on each new connection.  Once launched,
>     the ``socket.accept`` thread is no longer interested in the
>     ``handle_client`` threads.
>
>     In this case, the normal return value of the ``handle_client`` function
>     can be discarded.  But what should be done with exceptions that are not
>     caught in the child threads?
>
>     Therefore, this style of use would be indicated by providing an
>     top-level exception handler for the new thread as a keyword argument,
>     e.g. ``exception_handler=traceback.print_exception`` in a developer
>     environment and ``exception_handler=my_exception_logger`` in a
>     production environment.
>
>     If this keyword argument is provided, then the parent thread does not
>     need to do any kind of *wait* after the child thread is complete.  It
>     will either complete normally and go away silently, or raise an uncaught
>     exception, which is passed to the indicated exception_handler, and then
>     go away with no more ado.
>
>  #. Create micro_threads to run multiple long-running *functions* in
>     parallel where the final return value from each *function* is needed in
>     the parent thread.
>
>     In this case, the ``exception_handler`` argument is not specified and
> the
>     parent thread needs to *wait* on the child thread (when the parent is
>     ready to do so).  Thus, completed micro_threads will form zombie
>     threads until their parents retrieve their final return values (much
>     like unix processes).
>
>     This ends up being a kind of parallel execution strategy and it might be
>     nice to have a ``threaded_map`` function that will create a micro_thread
>     for each element of its *iterable* argument in order to run the
>     *function* on them in parallel and then return an iterable of the
>     waited for results.
>
>     On doing the *wait*, an uncaught exception in the child micro_thread is
>     re-raised in the parent micro_thread.
>
>  #. In the above examples, the child micro_threads are completely
>     independent of each other.  This final scenario uses *micro_pipes* to
>     allow threads to cooperatively solve problems (much like unix pipes).
>
> #. Micro_pipes.  Micro_pipes are one-way pipes that allow synchronized
>  communication between micro_threads.
>    The protocol for the receiving side of the pipe is simply the standard
>  python iterator protocol.
>
>  The sending side has these methods:
>    - ``put(object)`` to send *object* to the receiving side (retrieved with
>    the ``__next__`` method).
>  - ``take_from(iterable)`` to send a series of objects to the receiving side
>    (retrieved with multiple ``__next__`` calls).
>  - ``close()`` causes ``StopIteration`` on the next ``__next__`` call.
>    A ``put`` done after a ``close`` silently terminates the micro_thread
>    doing the ``put`` (in case the receiving side closes the micro_pipe).
>
>  Micro_pipes are automatically associated with micro_threads, making it less
>  likely to hang the program:
>
>  >>> pipe = micro_pipe()
>  >>> next(pipe)  # hangs the program!  No micro_thread created to feed
> pipe...
>
>  So each micro_thread will automatically have a stdout micro_pipe assigned
>  to it and can (optionally) be assigned a stdin micro_pipe (some other
>  micro_thread's stdout micro_pipe).  When the micro_thread terminates, it
>  automatically calls ``close`` on its stdout micro_pipe.
>
>  To access the stdout micro_pipe of the current micro_thread, new ``put``
>  and ``take_from`` built-in functions are provided.
>
>  Micro_pipes lets us write generator functions in a new way by having the
>  generator do ``put(object)`` rather than ``yield object``.  In this case,
>  the generator function has no ``yield`` statement, so is not treated
>  specially by the compiler.  Basically this means that calling a new-style
>  generator does not automatically create a new micro_thread (sort of what
>  calling an old-style generator does).
>
>  The ``put(object)`` does the same thing as ``yield object``, but
>  allows the generator to share the micro_pipe with other new-style
>  generators functions (by simply calling them) and old-style generators (or
>  any iterable) by calling ``take_from`` on them.  This lets the generator
>  delegate to other generators without having to get involved with passing
>  the results back to its caller.
>
>  For example, a generator to output all the even numbers from 1-n,
>  followed by all of the odd numbers::
>
>      def even_odd(n):
>          take_from(range(2, n, 2))
>          take_from(range(1, n, 2))
>
>  These "new-style" generators would have to be run in their own
>  micro_thread:
>
>  >>> pipe = micro_thread(even_odd, 100).stdout
>  >>> # now pipe is an iterable representing the generator:
>  >>> print tuple(pipe)
>
>  But the generator is then not restricted to running within its own
>  micro_thread.  It could also sometimes be used as a helper by other
>  generators within their micro_thread.  This would allow generators to
>  still use each other as helpers.  For example::
>
>      def even(n):
>          take_from(range(2, n, 2))
>
>      def odd(n):
>          take_from(range(1, n, 2))
>
>      def even_odd(n):
>          even(n)
>          odd(n)
>
>  At this point a micro_thread may be created on any of the above generators.
>
>
> Open Questions
> ==============
>
> #. How are exceptions propagated from one ``PyDef_Deferred`` to the next?
>
>  - This would happen when the final result of a micro_thread is needed by
>    another micro_thread.  This happens in two cases:
>
>    - When a *wait* is done.  In this case the exception is propagated to the
>      caller.
>    - When dealing with pipes, it seems that an exception on the ``put`` side
>      should be propagated to the ``__next__`` side.  This is another reason
>      to have the stdout pipe associated with the micro_thread.  Because when
>      the exception occurs, it will not be in the ``put`` call; so without
>      attaching the pipe to the micro_thread, there would be no way of
>      knowing which pipe that micro_thread was outputting to.
>
>      Thus exception would propagate from the ``put`` side of a micro_pipe to
>      the ``__next__`` side, but not the other direction.
>
> #. How are tracebacks handled?
> #. Do we:
>
>  #. Treat each python-to-python call as a separate C call, with it's own
>     callback_fn?
>  #. Only register one callback_fn for each continuous string of
>     python-to-python calls and then process them iteratively rather than
>     recursively in the callback_fn (but not in the original calls)?
>  #. Treat python-to-python calls iteratively both in the original calls
>     and in the callback_fn?
>
> #. How is process termination handled?
>    - I guess we can keep a list of micro_threads and terminate each of them.
>    There's a question of whether to allow the micro_threads to complete or
>    to abort them mid-stream.  Kind of like a unix shutdown.  Maybe two kinds
>    of process termination?
>
> #. How does this interact with the existing posix thread_ package?
>
>  - Each micro_thread would be associated with a posix thread.  Or,
>    conversely, each posix thread would have its own list of micro_threads.
>
> #. How does this impact the debugger/profiler/sys.settrace?
> #. Should functions (C and python) that may defer be indicated with some
>  naming convention (e.g., ends in '_d') to make it easier for programmers
>  to avoid them within their critical sections of code (in terms of
>  synchronization)?
>
>
> Rationale
> =========
>
> The implementation is done by treating the C-level deferreds as a special
> case of C-level exceptions so that the new ``PyDef_Deferred`` objects will
> be
> be treated like any other exception if *any* C function within the call
> chain
> hasn't been modified to deal with them.  In this case, the normal execution
> of the program is interrupted, but in a well understood way (by an
> exception)
> with the name of the offending C function contained in the exception message
> so that the python developer knows where to go to fix it.
>
> Only when *all* C functions in the call chain properly recognize and deal
> with
> the ``PyDef_Deferred`` is the new deferred object applied to implement the
> new micro-threading behavior.
>
> No change is required for C functions that could never get a
> ``PyDef_Deferred``.
>
> This takes advantage of the fact that the current exception mechanism
> already
> unwinds the C stack.  It also adds deferred processing without adding
> additional checks after each C function call to see whether to defer
> execution.  The check that is already being done for exceptions doubles as a
> check for deferred processing.
>
>
> Other Approaches
> ================
>
> Here's a brief comparison to other approaches to micro-threading in python:
>
> - `Stackless Python`_ [#stackless]_
>
>  - As near as I can tell, stackless went through two incarnations:
>     #. The first incarnation involved an implementation of Frame
> continuations
>      which were then used to provide the rest of the stackless
> functionality.
>            - A new ``Py_UnwindToken`` was created to unwind the C stack.
>  This is
>        similar to the new ``PyDef_Deferred`` proposed in this PEP, except
>        that ``Py_UnwindToken`` is treated as a special case of a normal
>        ``PyObject`` return value, while the ``PyDef_Deferred`` is treated
>        as a special case of a normal exception.
>
>        Consider the case where unmodified C function ``A`` calls modified C
>        function ``B``, and function ``B`` wants to defer execution.  In
>        both cases function ``B`` returns a special value indicating it
>        wants to defer.
>
>        The difference comes in how function ``A`` (which isn't prepared to
>        cooperate with this new type of request) handles this situation.
>
>        With the stackless ``Py_UnwindToken`` as a return value, function
>        ``A`` tries to act on ``Py_UnwindToken`` as an ordinary return value,
>        which it is not.  This may or may not lead to some other exception
>        being thrown due to a type inconsistency between ``Py_UnwindToken``
>        and what was expected by function ``A``.  It also means that further
>        up the call chain in the code that called function ``A``, the fact
>        that a ``Py_UnwindToken`` was returned isn't known if function ``A``
>        does not simply return it.  In short, who knows what will happen...
>
>        But this PEP treats the request to defer as a special exception.
>        So function ``A``, receiving a ``PyDef_Deferred`` exception will
>        treat it as an ordinary exception.  Function ``A``, not recognizing
>        this exception, will perform any required clean up and simply
>        forward the ``PyDef_Deferred`` on to its caller (re-raise it).
>        The top-level code that receives this ``PyDef_Deferred`` knows that
>        it's a broken ``PyDef_Deferred`` and raises it as a normal exception,
>        which will state "Deferred execution not yet implemented by A".
>        This makes much more sense.
>
>      - Another difference between the two styles of continuations is that
>        the stackless continuation is designed to be able to be continued
>        multiple times.  In other words, you can continue the execution of
>        the program from the point the continuation was made as many times
>        as you wish, passing different seed values each time.
>
>        The ``PyDef_Deferred`` described in this PEP (like the Twisted
>        Deferred) is designed to be continued oonce.
>          - The stackless approach provides a python-level continuation
>        mechanism (at the Frame level) that only makes python functions
>        continuable.  It provides no way for C functions to register
>        continuations so that C functions can be unwound from the stack
>        and later continued (other than those related to the byte code
>        interpreter).
>
>        In contrast, this PEP proposes a C-level continuation mechanism
>        very similar to the Twisted Deferred.  Each C function registers a
>        callback to be run when the Deferred is continued.  From this
>        perspective, the byte code interpreter is just another C function.
>
>   #. The second incarnation involved a way of hacking the underlying C
>      stack to copy it and later restore it as a means of continuing the
>      execution.
>
>      - This doesn't appear to be portable to different CPU/C Compiler
>        configurations.
>      - This doesn't deal with other global state (global/static variables,
>        file pointers, etc) that may also be used by this saved stack.
>      - In contrast, this PEP uses a single C stack and makes no assumptions
>        about the underlying C stack implementation.  It is completely
>        portable to any CPU/C compiler configuration.
>
> - `Implementing "weightless threads" with Python generators`_ [#weightless]_
>
>  - This requires you code each thread as generators.  The generator
>   executes a 'yield' to relinquish control.
>  - It's not clear how this scales.  It seems that to pause in a lower
>   python function, it and all intermediate functions must be generators.
>
> - python-safethread_ [#safethread]_
>
>  - This is an alternate implementation to thread_ that adds monitors for
>   mutable types, deadlock detection, improves exception propagation
>   across threads and program finalization, and removes the GIL lock.  As
>   such, it is not a "micro" threading approach, though by removing the GIL
>   lock it may be able to better use multiple processor configurations
>   than the approach proposed in this PEP.
>
> - `Sandboxed Threads in Python`_ [#sandboxed-threads]_
>
>  - Another alternate implementation to thread_, this one only shares
>   immutable objects between threads, modifying the referencing counting
>   system to avoid synchronization issues with the reference count for
>   shared objects.  Again, not a "micro" threading approach, but perhaps
>   also better with multiple processors.
>
> .. _Implementing "weightless threads" with Python generators:
>  http://www.ibm.com/developerworks/library/l-pythrd.html
> .. _python-safethread: https://launchpad.net/python-safethread
> .. _Sandboxed Threads in Python:
>  http://mail.python.org/pipermail/python-dev/2005-October/057082.html
> .. _Stackless Python: http://www.stackless.com/
> .. _thread: http://docs.python.org/lib/module-thread.html
> .. _threading: http://docs.python.org/lib/module-threading.html
>
>
> Backwards Compatibility
> =======================
>
> This PEP doesn't break any existing code.  Existing code just won't take
> advantage of any of the new features.
>
> But there are two possible problem areas:
>
> #. Code uses micro-threading, but then causes an unmodified C function
>  to call a modified C function which tries to defer execution.
>
>  In this case an exception will be generated stating that the unmodified C
>  function needs to be converted before this program will work.
>
> #. Code originally written in a single threaded environment is now used in a
>  micro-threaded environment.  The old code was not written taking
>  synchronization issues into account, which may cause problems if the old
>  code calls a function which causes it to defer in the middle of its
>  critical section.  This could cause very strange behavior, but can't
>  result in any C-level errors (e.g., segmentation violation).
>
>  This old code would have to be fixed to run with the new features.  I
>  expect that this will not be a frequent problem.
>
>
> References
> ==========
>
> .. [#twisted-fn] Twisted, Twisted Matrix Labs
>  (http://twistedmatrix.com/trac/)
> .. [#c_api] Python/C API Reference Manual, Rossum
>  (http://docs.python.org/api/api.html)
> .. [#stackless] Stackless Python, Tismer
>  (http://www.stackless.com/)
> .. [#thread-module] thread -- Multiple threads of control
>  (http://docs.python.org/lib/module-thread.html)
> .. [#threading-module] threading -- Higher-level threading interface
>  (http://docs.python.org/lib/module-threading.html)
> .. [#weightless] Charming Python: Implementing "weightless threads" with
>  Python generators, Mertz
>  (http://www.ibm.com/developerworks/library/l-pythrd.html)
> .. [#safethread] Threading extensions to the Python Language,
>  (https://launchpad.net/python-safethread)
> .. [#sandboxed-threads] Sandboxed Threads in Python, Olsen
>  (http://mail.python.org/pipermail/python-dev/2005-October/057082.html)
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From solipsis at pitrou.net  Sun Aug  3 14:33:47 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 3 Aug 2008 12:33:47 +0000 (UTC)
Subject: [Python-ideas]
	=?utf-8?q?New_PEP_proposal=3A_C_Micro-Threading_?=
	=?utf-8?b?KGFsYQlUd2lzdGVkKSAobG9uZyEp?=
References: <4894B36F.4030805@gmail.com>
	<e6511dbf0808022220p761d7b8ex132730fbe9be91@mail.gmail.com>
Message-ID: <loom.20080803T122453-113@post.gmane.org>


Hi,

> P.S. After having used event-driven frameworks for quite a few years
> (asyncore derived socket servers and clients, wxPython,
> publish/subscribe architectures, ...), they really aren't all that
> bad.

Probably. Well, wxPython is quite lousy, and Twisted is very good.
However, the problem is there are lots of them, and by construction they can't
easily be mixed together (although there are various hacks to do so).

Having a standard way to define low-level asynchronicity (especially with a C
API) would allow to merge those worlds quite naturally.

Another interesting aspect of this proposal is that the core functions would be
implemented in C, which would be very interesting performance-wise, and would
also allow to write accelerators in C so that some events don't even involve
interpreting Python bytecode. The Twisted reactor overhead is prohibitive when
dealing with lots of tiny messages.

Regards

Antoine.




From aahz at pythoncraft.com  Sun Aug  3 15:44:47 2008
From: aahz at pythoncraft.com (Aahz)
Date: Sun, 3 Aug 2008 06:44:47 -0700
Subject: [Python-ideas] New PEP proposal: C Micro-Threading (ala
	Twisted) (long!)
In-Reply-To: <4894B36F.4030805@gmail.com>
References: <4894B36F.4030805@gmail.com>
Message-ID: <20080803134447.GB12115@panix.com>

On Sat, Aug 02, 2008, Bruce Frederiksen wrote:
>
> Here is a proposal for making the benefits of Twisted available without
> having to write python code in an event driven style.

This overall looks good, but I would suggest re-arranging a bit to put
the gory C-level details later and substitute an overview where you
currently have the gory details.  Even if your PEP is not accepted,
you've done a tremendous job writing up this proposal, one that will
serve as useful documentation for other people.  I think you might as
well go ahead and post your next draft to comp.lang.python in addition
to python-ideas -- you'll get more feedback that way.

Side note: I hope you already know this won't get into 2.6/3.0 -- they
are in beta.

> (I'm not sure whether this list accepts HTML, so am just copying plain
> text -- you may want to run rst2html on this)...

Definitely reST is preferred over HTML.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!


From tjreedy at udel.edu  Mon Aug  4 00:44:22 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 03 Aug 2008 18:44:22 -0400
Subject: [Python-ideas] New PEP proposal: C Micro-Threading (ala
	Twisted) (long!)
In-Reply-To: <20080803134447.GB12115@panix.com>
References: <4894B36F.4030805@gmail.com> <20080803134447.GB12115@panix.com>
Message-ID: <g75cc4$v5n$1@ger.gmane.org>



Aahz wrote:
> On Sat, Aug 02, 2008, Bruce Frederiksen wrote:
>> Here is a proposal for making the benefits of Twisted available without
>> having to write python code in an event driven style.
> 
> This overall looks good, but I would suggest re-arranging a bit to put
> the gory C-level details later and substitute an overview where you
> currently have the gory details.

The writeup on the C details seemed to assume knowledge of Twisted in 
addition to the current C API.  There also seemed to be references near 
the top to new things not yet defined.  I gave up about in the third 
paragraph.  So I agree that rearrangement would help some readers.

Fortunately for my curiosity, I skipped down about halfway where things 
became clearer again.

tjr



From dangyogi at gmail.com  Mon Aug  4 01:55:25 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Sun, 03 Aug 2008 19:55:25 -0400
Subject: [Python-ideas] New PEP proposal: C Micro-Threading (ala
 Twisted) (long!)
In-Reply-To: <e6511dbf0808022220p761d7b8ex132730fbe9be91@mail.gmail.com>
References: <4894B36F.4030805@gmail.com>
	<e6511dbf0808022220p761d7b8ex132730fbe9be91@mail.gmail.com>
Message-ID: <4896456D.5040303@gmail.com>

Yes, the greenlet package is available for current Python
releases, but there is a note that it doesn't work "out of the box"
on the Win32 platform.

The greenlet package uses much the same approach as the
second incarnation of stackless -- it copies the C stack.  This
limits portability to different CPU/C compiler architectures (I'm
not sure if this is why it doesn't work on Win32).

It also lacks a reactor, though one could be written for it.

I don't believe that it has an alternate main loop, but that it copies
the C stack between calls to greenlet.switch.  Each copy is retained
in the heap making it available to be copied back to the C stack
when it becomes the target of another greenlet.switch.

Josiah Carlson wrote:
> A few comments...
>
> Firstly, this is a great PEP, you've obviously put a lot of thought
> and effort into getting it together.
>
> Secondly, isn't the greenlet package already available online for
> current Python revisions?  If so, has it gained significant numbers of
> users?
>
> Regardless of the answer to the above, how does the current greenlet
> package plug into Python without having API hooks?  Does it execute an
> alternate mainloop that does the stack switching?
>
>  - Josiah
>
> P.S. After having used event-driven frameworks for quite a few years
> (asyncore derived socket servers and clients, wxPython,
> publish/subscribe architectures, ...), they really aren't all that
> bad.  Then again, I don't find threading all that bad either (I try to
> stick with simple queues, explicitly synchronized interfaces, etc.,
> which reduces complexity by a few orders of magnitude).
>
>   



From bborcic at gmail.com  Wed Aug 20 14:37:22 2008
From: bborcic at gmail.com (Boris Borcic)
Date: Wed, 20 Aug 2008 14:37:22 +0200
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <c7450fba0807090442t6a6213ald670b71bcca2b3fb@mail.gmail.com>
References: <486CDCF5.6080805@korokithakis.net>	<486CE075.3000104@vector-seven.com>
	<486CE11F.50601@korokithakis.net>	<486CE904.2040403@vector-seven.com>	<740c3aec0807031353g23d00279x5f797750b1f5f9d8@mail.gmail.com>	<D3C8B31F6E824022874BAB01C78D2407@RaymondLaptop1>	<088C35D1-D1B9-4A39-B004-F9C06B29E7AC@acm.org>	<ca471dc20807032033n319ae95fi2967dd2e0aaf1b59@mail.gmail.com>	<4872CCA9.1040908@acm.org>
	<g50at9$b5m$1@ger.gmane.org>
	<c7450fba0807090442t6a6213ald670b71bcca2b3fb@mail.gmail.com>
Message-ID: <g8h388$n1p$1@ger.gmane.org>

Andrew Toulouse wrote:
> If I'm understanding you correctly, this would address something I've 
> wanted list comprehensions to do for a little while:
> 
> [x for x in range(0,10) until greaterthanN(4,x)]

Note that you can do something like

 >>> def yet(b) :
	if b : raise StopIteration

	
 >>> list(x for x in range(0,10) if not yet(x>4))
[0, 1, 2, 3, 4]


Cheers, BB



From tjreedy at udel.edu  Wed Aug 20 19:03:19 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Aug 2008 13:03:19 -0400
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <g8h388$n1p$1@ger.gmane.org>
References: <486CDCF5.6080805@korokithakis.net>	<486CE075.3000104@vector-seven.com>	<486CE11F.50601@korokithakis.net>	<486CE904.2040403@vector-seven.com>	<740c3aec0807031353g23d00279x5f797750b1f5f9d8@mail.gmail.com>	<D3C8B31F6E824022874BAB01C78D2407@RaymondLaptop1>	<088C35D1-D1B9-4A39-B004-F9C06B29E7AC@acm.org>	<ca471dc20807032033n319ae95fi2967dd2e0aaf1b59@mail.gmail.com>	<4872CCA9.1040908@acm.org>	<g50at9$b5m$1@ger.gmane.org>	<c7450fba0807090442t6a6213ald670b71bcca2b3fb@mail.gmail.com>
	<g8h388$n1p$1@ger.gmane.org>
Message-ID: <g8hioo$n27$1@ger.gmane.org>



Boris Borcic wrote:
> Andrew Toulouse wrote:
>> If I'm understanding you correctly, this would address something I've 
>> wanted list comprehensions to do for a little while:
>>
>> [x for x in range(0,10) until greaterthanN(4,x)]
> 
> Note that you can do something like
> 
>  >>> def yet(b) :
>     if b : raise StopIteration
> 
>     
>  >>> list(x for x in range(0,10) if not yet(x>4))
> [0, 1, 2, 3, 4]

I don't think it a good idea to abuse StopIteration in this way.  It is 
intended only for use in iterators. The obvious list comp abbreviation 
does not work in CPython 3.0.

 >>> [x for x in range(0,10) if not yet(x>4)]
Traceback (most recent call last):
   File "<pyshell#4>", line 1, in <module>
     [x for x in range(0,10) if not yet(x>4)]
   File "<pyshell#4>", line 1, in <listcomp>
     [x for x in range(0,10) if not yet(x>4)]
   File "<pyshell#1>", line 2, in yet
     if b : raise StopIteration
StopIteration

So it breaks the intended identity in Py3: list(genexp) == [genexp].

This was discussed on the Py3 dev list and it was agreed the difference 
in behavior was a bug but would be difficult to impossible to fix 
without performance degradation and unnecessary to fix because the 
problem only arises if someone uses StopIteration outside its documented 
usage in iterators.



From greg.ewing at canterbury.ac.nz  Thu Aug 21 02:41:14 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 Aug 2008 12:41:14 +1200
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <g8hioo$n27$1@ger.gmane.org>
References: <486CDCF5.6080805@korokithakis.net>
	<486CE075.3000104@vector-seven.com> <486CE11F.50601@korokithakis.net>
	<486CE904.2040403@vector-seven.com>
	<740c3aec0807031353g23d00279x5f797750b1f5f9d8@mail.gmail.com>
	<D3C8B31F6E824022874BAB01C78D2407@RaymondLaptop1>
	<088C35D1-D1B9-4A39-B004-F9C06B29E7AC@acm.org>
	<ca471dc20807032033n319ae95fi2967dd2e0aaf1b59@mail.gmail.com>
	<4872CCA9.1040908@acm.org> <g50at9$b5m$1@ger.gmane.org>
	<c7450fba0807090442t6a6213ald670b71bcca2b3fb@mail.gmail.com>
	<g8h388$n1p$1@ger.gmane.org> <g8hioo$n27$1@ger.gmane.org>
Message-ID: <48ACB9AA.5070609@canterbury.ac.nz>

Terry Reedy wrote:

> This was discussed on the Py3 dev list and it was agreed the difference 
> in behavior was a bug

I don't think it was agreed that it was a bug. Rather
that it's just something that shows that a list comp and
list(genexp) are not exactly equivalent in all ways,
and that it doesn't matter if they differ in some
corner cases like this.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Thu Aug 21 02:17:09 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 Aug 2008 12:17:09 +1200
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <g8h388$n1p$1@ger.gmane.org>
References: <486CDCF5.6080805@korokithakis.net>
	<486CE075.3000104@vector-seven.com> <486CE11F.50601@korokithakis.net>
	<486CE904.2040403@vector-seven.com>
	<740c3aec0807031353g23d00279x5f797750b1f5f9d8@mail.gmail.com>
	<D3C8B31F6E824022874BAB01C78D2407@RaymondLaptop1>
	<088C35D1-D1B9-4A39-B004-F9C06B29E7AC@acm.org>
	<ca471dc20807032033n319ae95fi2967dd2e0aaf1b59@mail.gmail.com>
	<4872CCA9.1040908@acm.org> <g50at9$b5m$1@ger.gmane.org>
	<c7450fba0807090442t6a6213ald670b71bcca2b3fb@mail.gmail.com>
	<g8h388$n1p$1@ger.gmane.org>
Message-ID: <48ACB405.3030900@canterbury.ac.nz>

Boris Borcic wrote:

> Note that you can do something like
> 
>  >>> def yet(b) :
>     if b : raise StopIteration
> 
>     
>  >>> list(x for x in range(0,10) if not yet(x>4))
> [0, 1, 2, 3, 4]

However, note that this doesn't work if you use a
list comprehension instead of list(genexp).

The reason is that the StopIteration is not stopping
the for-loop, but rather the implicit iteration over
the results of the genexp being done by list().

So it only appears to work by accident.

-- 
Greg


From bborcic at gmail.com  Thu Aug 21 17:20:34 2008
From: bborcic at gmail.com (Boris Borcic)
Date: Thu, 21 Aug 2008 17:20:34 +0200
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <g8hioo$n27$1@ger.gmane.org>
References: <486CDCF5.6080805@korokithakis.net>	<486CE075.3000104@vector-seven.com>	<486CE11F.50601@korokithakis.net>	<486CE904.2040403@vector-seven.com>	<740c3aec0807031353g23d00279x5f797750b1f5f9d8@mail.gmail.com>	<D3C8B31F6E824022874BAB01C78D2407@RaymondLaptop1>	<088C35D1-D1B9-4A39-B004-F9C06B29E7AC@acm.org>	<ca471dc20807032033n319ae95fi2967dd2e0aaf1b59@mail.gmail.com>	<4872CCA9.1040908@acm.org>	<g50at9$b5m$1@ger.gmane.org>	<c7450fba0807090442t6a6213ald670b71bcca2b3fb@mail.gmail.com>	<g8h388$n1p$1@ger.gmane.org>
	<g8hioo$n27$1@ger.gmane.org>
Message-ID: <g8k161$idd$1@ger.gmane.org>

Terry Reedy wrote:
> 
>>
>>  >>> def yet(b) :
>>     if b : raise StopIteration
>>
>>      >>> list(x for x in range(0,10) if not yet(x>4))
>> [0, 1, 2, 3, 4]
> 
> I don't think it a good idea to abuse StopIteration in this way.  It is 
> intended only for use in iterators. The obvious list comp abbreviation 
> does not work in CPython 3.0.
> 
>  >>> [x for x in range(0,10) if not yet(x>4)]
> Traceback (most recent call last):
>   File "<pyshell#4>", line 1, in <module>
>     [x for x in range(0,10) if not yet(x>4)]
>   File "<pyshell#4>", line 1, in <listcomp>
>     [x for x in range(0,10) if not yet(x>4)]
>   File "<pyshell#1>", line 2, in yet
>     if b : raise StopIteration
> StopIteration
> 
> So it breaks the intended identity in Py3: list(genexp) == [genexp].

FYI, there is nothing specific to Py3, Python 2.5 and 2.4 behave the same.

> 
> This was discussed on the Py3 dev list and it was agreed the difference 
> in behavior was a bug but would be difficult to impossible to fix 
> without performance degradation and unnecessary to fix because the 
> problem only arises if someone uses StopIteration outside its documented 
> usage in iterators.

Whatever, consistency with Python 2+ is maintained.

Cheers, BB



From bborcic at gmail.com  Thu Aug 21 17:32:28 2008
From: bborcic at gmail.com (Boris Borcic)
Date: Thu, 21 Aug 2008 17:32:28 +0200
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <48ACB405.3030900@canterbury.ac.nz>
References: <486CDCF5.6080805@korokithakis.net>	<486CE075.3000104@vector-seven.com>
	<486CE11F.50601@korokithakis.net>	<486CE904.2040403@vector-seven.com>	<740c3aec0807031353g23d00279x5f797750b1f5f9d8@mail.gmail.com>	<D3C8B31F6E824022874BAB01C78D2407@RaymondLaptop1>	<088C35D1-D1B9-4A39-B004-F9C06B29E7AC@acm.org>	<ca471dc20807032033n319ae95fi2967dd2e0aaf1b59@mail.gmail.com>	<4872CCA9.1040908@acm.org>
	<g50at9$b5m$1@ger.gmane.org>	<c7450fba0807090442t6a6213ald670b71bcca2b3fb@mail.gmail.com>	<g8h388$n1p$1@ger.gmane.org>
	<48ACB405.3030900@canterbury.ac.nz>
Message-ID: <g8k1sd$l8v$1@ger.gmane.org>

Greg Ewing wrote:
> Boris Borcic wrote:
> 
>> Note that you can do something like
>>
>>  >>> def yet(b) :
>>     if b : raise StopIteration
>>
>>      >>> list(x for x in range(0,10) if not yet(x>4))
>> [0, 1, 2, 3, 4]
> 
> However, note that this doesn't work if you use a
> list comprehension instead of list(genexp).

Note that this was written in the cited context
of the OP's original list comprehension.

> 
> The reason is that the StopIteration is not stopping
> the for-loop, but rather the implicit iteration over
> the results of the genexp being done by list().
> 
> So it only appears to work by accident
> 

...under the assumption of a specific equivalence.

Cheers, BB



From tjreedy at udel.edu  Fri Aug 22 20:41:21 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 22 Aug 2008 14:41:21 -0400
Subject: [Python-ideas] "While" suggestion
In-Reply-To: <g8k161$idd$1@ger.gmane.org>
References: <486CDCF5.6080805@korokithakis.net>	<486CE075.3000104@vector-seven.com>	<486CE11F.50601@korokithakis.net>	<486CE904.2040403@vector-seven.com>	<740c3aec0807031353g23d00279x5f797750b1f5f9d8@mail.gmail.com>	<D3C8B31F6E824022874BAB01C78D2407@RaymondLaptop1>	<088C35D1-D1B9-4A39-B004-F9C06B29E7AC@acm.org>	<ca471dc20807032033n319ae95fi2967dd2e0aaf1b59@mail.gmail.com>	<4872CCA9.1040908@acm.org>	<g50at9$b5m$1@ger.gmane.org>	<c7450fba0807090442t6a6213ald670b71bcca2b3fb@mail.gmail.com>	<g8h388$n1p$1@ger.gmane.org>	<g8hioo$n27$1@ger.gmane.org>
	<g8k161$idd$1@ger.gmane.org>
Message-ID: <g8n18h$fns$1@ger.gmane.org>



Boris Borcic wrote:
> Terry Reedy wrote:

>> So it breaks the intended identity in Py3: list(genexp) == [genexp].
> 
> FYI, there is nothing specific to Py3, Python 2.5 and 2.4 behave the same.

FYI, there *is* something specific to Py3: the willingness to break old 
code.  Originally, list comps were defined as having the same result as 
an equivalent series of for loops and conditional statements:

x = [f(i) for i in seq if g(i)] # same as

_ = []
for i in seq:
   if g(i):
     _.append(f(i))
x = _

About the time 2.5 came out (or maybe before), and generator expressions 
were in, it was decided that this definition, which results in 'leaking' 
iteration variables out of comprehensions, was a design mistake, and 
that the new identify given above would be better.  When implementing 
[genexp] as list(genexp) was found to take much longer (2x?), an 
alternative was found that was nearly as fast as the 2.x implementation 
but stopped the leakage and seemed otherwise identical in effect to 
list(genexp).  The only exception I have seen discussed arises from the 
use of StopIteration outside an iterator .__next__ metrhod.

tjr



From fredrik.johansson at gmail.com  Sun Aug 24 12:43:59 2008
From: fredrik.johansson at gmail.com (Fredrik Johansson)
Date: Sun, 24 Aug 2008 12:43:59 +0200
Subject: [Python-ideas] Inline 'raises' expression
Message-ID: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>

Hi,

It happens that I'm just interested in whether an expression raises an
exception, not the return value. This might look something like

    try:
        check_status()
    except IOError:
        cleanup()

which could be written more simply as

    if check_status() raises IOError:
        cleanup()

Also, instead of the inconvenient

    self.assertRaises(ZeroDivisionError, lambda: 1/0)

one could just write

    assert 1/0 raises ZeroDivisionError

Something like this would especially be useful for those of us who
aren't fans of the unittest framework. Alternatively, just the
assert--raises form could be permitted.

Thoughts?

Fredrik


From russ.paielli at gmail.com  Mon Aug 25 00:43:28 2008
From: russ.paielli at gmail.com (Russ Paielli)
Date: Sun, 24 Aug 2008 15:43:28 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
Message-ID: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>

Hello,

I have written a PEP, and I was warned by none other than Mr. Van Rossum
himself that it would be controversial at best. Well, here goes anyway.

First, let me introduce myself. I am an aerospace engineer, and I am using
Python to develop a research prototype of a conflict alerting aid for air
traffic controllers. It is intended to replace the current legacy system
which has been operational since the 1970s. I have also developed a free
Python package to represent physical scalars in a unique manner that can be
as efficient as built-in numeric types. I am using it extensively in my
conflict alerting aid. You can read about it in the current edition of The
Python Papers at http://pythonpapers.org or on my website at
http://RussP.us/scalar.htm .

Now to the PEP. Let me start by saying that I fully understand the history
and controversy regarding the explicit use of "self" in Python. I am not
going to say that it was a mistake, nor am I going to say that the first
argument of a class instance method should not refer to the instance for
which it is called. What I will say is that I think Python syntax could be
significantly simplified with the simple little convention that I am
proposing. All I ask is that you carefully read my proposal before you
reply. Thank you.

Russ
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080824/e13d126c/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pep-XXX.txt
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080824/e13d126c/attachment.txt>

From brett at python.org  Mon Aug 25 01:53:15 2008
From: brett at python.org (Brett Cannon)
Date: Sun, 24 Aug 2008 16:53:15 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
Message-ID: <bbaeab100808241653v3a811dc1jd11ca1e654157206@mail.gmail.com>

On Sun, Aug 24, 2008 at 3:43 PM, Russ Paielli <russ.paielli at gmail.com> wrote:
> Hello,
>
> I have written a PEP, and I was warned by none other than Mr. Van Rossum
> himself that it would be controversial at best. Well, here goes anyway.
>
> First, let me introduce myself. I am an aerospace engineer, and I am using
> Python to develop a research prototype of a conflict alerting aid for air
> traffic controllers. It is intended to replace the current legacy system
> which has been operational since the 1970s. I have also developed a free
> Python package to represent physical scalars in a unique manner that can be
> as efficient as built-in numeric types. I am using it extensively in my
> conflict alerting aid. You can read about it in the current edition of The
> Python Papers at http://pythonpapers.org or on my website at
> http://RussP.us/scalar.htm .
>
> Now to the PEP. Let me start by saying that I fully understand the history
> and controversy regarding the explicit use of "self" in Python. I am not
> going to say that it was a mistake, nor am I going to say that the first
> argument of a class instance method should not refer to the instance for
> which it is called. What I will say is that I think Python syntax could be
> significantly simplified with the simple little convention that I am
> proposing. All I ask is that you carefully read my proposal before you
> reply. Thank you.
>

Couple things. One, tilde and carat are not even possible options as
they are used for binary negation and XOR, respectively.

Two, I disagree that it is in any way difficult to remember what the
first argument is since everyone names it either 'self' or 'cls'
depending on whether it is a classmethod or not.

Three, I don't see how a bunch of dollar signs in code is not just as
much clutter as 'self.'. Sure, it's shorter by four characters, but
that is not much in the grand scheme of things.

Four, I just think it's ugly to have a dollar sign prepend a name like that.

And five, the dollar sign is distracting. My eye automatically
gravitates to non-letter and non-digit characters, and having the code
littered with '$' is going to be distracting, IMO.

Thanks for trying, Russ, but this gets a -1 from me.

-Brett


From greg.ewing at canterbury.ac.nz  Mon Aug 25 01:47:10 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 25 Aug 2008 11:47:10 +1200
Subject: [Python-ideas] Inline 'raises' expression
In-Reply-To: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
Message-ID: <48B1F2FE.6090007@canterbury.ac.nz>

Fredrik Johansson wrote:

> one could just write
> 
>     assert 1/0 raises ZeroDivisionError

It's been pointed out that using assert for testing
is not the best idea, since then you can't test that
the code works properly with assertions turned off.

-- 
Greg


From aahz at pythoncraft.com  Mon Aug 25 02:02:29 2008
From: aahz at pythoncraft.com (Aahz)
Date: Sun, 24 Aug 2008 17:02:29 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
Message-ID: <20080825000229.GA12765@panix.com>

On Sun, Aug 24, 2008, Russ Paielli wrote:
>
> Abstract
> 
>     In Python, the class instance for which a method is called is
>     referred to within the method by the first formal argument.  By
>     convention, that first argument is typically named "self".
>     Attributes of "self" are referred to using the standard "dot"
>     notation, as in "self.attr".  As a result, complex methods can
>     become cluttered with many occurrences of "self.".  This PEP
>     proposes to allow the dollar symbol, "$", to be used as a
>     shorthand symbol for "self." or, in general, for "<arg1>.", where
>     "<arg1>" is the name of the first argument.  Thus, "self.attr"
>     could be written more succinctly as "$attr".  

<sniff><sniff>  Smells like Perl.  -1

(Sorry, but that *will* be the automatic reaction from many people, and
because your proposal doesn't deal with the issue up-front, it has no
chance to fly.)
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!


From jimjjewett at gmail.com  Mon Aug 25 02:17:09 2008
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sun, 24 Aug 2008 20:17:09 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
Message-ID: <fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>

On Sun, Aug 24, 2008 at 6:43 PM, Russ Paielli <russ.paielli at gmail.com> wrote:
> ... I fully understand the history
> and controversy regarding the explicit use of "self" in Python.

One of the problems with the current syntax is that calling sites
don't match the definition site the way they do for normal functions.
This proposal doesn't seem to help with that.

Normal function --

    def fn(a, b, z=None): ...

    fn(a, b, z)
    fn(a, b)

Method --

    def m(self, a, b, z=None): ...

    # self is moved outside the parens, changing the tuple size
    self.m(a, b, z)
    self.m(a, b)

-jJ


From python at rcn.com  Mon Aug 25 02:16:45 2008
From: python at rcn.com (Raymond Hettinger)
Date: Sun, 24 Aug 2008 21:16:45 -0300
Subject: [Python-ideas] Inline 'raises' expression
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
Message-ID: <B640FFB56C3E42FF848479B430CB5E36@RaymondLaptop1>

From: "Fredrik Johansson" <fredrik.johansson at gmail.com>
> one could just write
> 
>    assert 1/0 raises ZeroDivisionError

+1

This would be a nice extension to the assertion syntax.


Raymond


From greg.ewing at canterbury.ac.nz  Mon Aug 25 02:34:39 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 25 Aug 2008 12:34:39 +1200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
Message-ID: <48B1FE1F.3010800@canterbury.ac.nz>

Russ Paielli wrote:

>     Another option is to
>     explicitly "strip off" the "self." by writing "attr = self.attr"
>     to create a local reference to the attribute within the method.

This is very often done for another reason -- it's
*much* faster to refer to a local variable than to
look up an attribute, so if the attribute is being
used more than once or twice, it's more efficient
to pull it into a local first.

Given that, methods which use attributes of self a
lot tend not to look as cluttered as you might
expect.

-- 
Greg


From tjreedy at udel.edu  Mon Aug 25 03:26:49 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 24 Aug 2008 21:26:49 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
Message-ID: <g8t1on$7ne$1@ger.gmane.org>



Jim Jewett wrote:
> On Sun, Aug 24, 2008 at 6:43 PM, Russ Paielli <russ.paielli at gmail.com> wrote:
>> ... I fully understand the history
>> and controversy regarding the explicit use of "self" in Python.
> 
> One of the problems with the current syntax is that calling sites
> don't match the definition site the way they do for normal functions.
> This proposal doesn't seem to help with that.
> 
> Normal function --
> 
>     def fn(a, b, z=None): ...
> 
>     fn(a, b, z)
>     fn(a, b)
> 
> Method --

Methods are normal functions accessed through a class attribute binding.
In 3.0, the unbound method wrapper that might have obscured this is gone.

>     def m(self, a, b, z=None): ...
> 
>     # self is moved outside the parens, changing the tuple size
>     self.m(a, b, z)
>     self.m(a, b)

If m is an attribute of type(s) (which is s.__class__), this shrinkage 
is a convenient abbreviation, not a requirement.  The above calls are 
the same as
type(s).m(s.a,b,z) or type(s).m(s,a,b).  Or, if you prefer,
fn = type(s).m # or s.__class__.m
fn(s,a,b,z)
If m is an attribute s, and s in not a class, the shrinkage is not 
available, and one must write s.m(s,a,b,z) or s.m(s.a,b).

Terry Jan Reedy



From tjreedy at udel.edu  Mon Aug 25 05:22:49 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 24 Aug 2008 23:22:49 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
Message-ID: <g8t8i7$q3c$1@ger.gmane.org>



Russ Paielli wrote:
...
> instance for which it is called. What I will say is that I think Python 
> syntax could be significantly simplified with the simple little 
> convention that I am proposing. All I ask is that you carefully read my 
> proposal before you reply. Thank you.

I have.  It strikes me as an improvement over .attr and other ideas 
posted on c.l.p.

     In Python, the class instance for which a method is called is
     referred to within the method by the first formal argument.

Somehow awkward.

     By convention, that first argument is typically named "self".

For public code among English speakers.  But how far this extends and 
for how long, given Unicode identifiers in 3.0, is a moot point for your 
proposal.

     In Python, class instance methods are functions in which the first
     formal argument refers to the instance for which the method was
     called.

In Python, class instance methods are functions accessed as an attribute 
of the class, in particular of the object passed as the first argument. 
  To me your discussion is weakened by the implication that Python has a 
method class separate from functions.  (Bound methods are specialized 
partial-function wrappers; functools.partial is the general version.) 
For instance

     could even work for regular functions if the first argument is an
     object with attributes and/or methods that can be accessed with
     standard "dot" notation.

Def statements produce functions.  Period.

     This proposal takes no position of
     whether the latter should be allowed or not.

To repeat: former *is* latter with a particular access path.  So the 
question is not whether to 'allow' $ to work in functions; that *is* 
your proposal, whether you know it or not.  To do otherwise would 
require some other significant change to Python, one that would 
complicate it.

      Four possible candidates for the choice of that symbol are "~",
     "@", "^", and "$".

You left out "?" (not too good ;-) and now, in 3.0, "`" (backtick, no 
longer an repr synonym).  The latter is lighter than "$" but perhaps 
more visible than ".", depending on the font.  Hmmm, a comparison:

self.d = sqrt(self.x*self.x + self.y*self.y)

s.d = sqrt(s.x*s.x + s.x*s.x)

.d = sqrt(.x*.x + .y*.y)

$d = sqrt($x*$x + $y*$y)

`d = sqrt(`s*`x + `y*`y)

I think I like this last best.

Terry Jan Reedy



From cvrebert at gmail.com  Mon Aug 25 05:45:51 2008
From: cvrebert at gmail.com (Chris Rebert)
Date: Sun, 24 Aug 2008 20:45:51 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <g8t8i7$q3c$1@ger.gmane.org>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<g8t8i7$q3c$1@ger.gmane.org>
Message-ID: <47c890dc0808242045r584cf451pa49faa29ca8818a2@mail.gmail.com>

On Sun, Aug 24, 2008 at 8:22 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>
>
> Russ Paielli wrote:
> ...
>>
>> instance for which it is called. What I will say is that I think Python
>> syntax could be significantly simplified with the simple little convention
>> that I am proposing. All I ask is that you carefully read my proposal before
>> you reply. Thank you.
>
> I have.  It strikes me as an improvement over .attr and other ideas posted
> on c.l.p.
>
>    In Python, the class instance for which a method is called is
>    referred to within the method by the first formal argument.
>
> Somehow awkward.
>
>    By convention, that first argument is typically named "self".
>
> For public code among English speakers.  But how far this extends and for
> how long, given Unicode identifiers in 3.0, is a moot point for your
> proposal.
>
>    In Python, class instance methods are functions in which the first
>    formal argument refers to the instance for which the method was
>    called.
>
> In Python, class instance methods are functions accessed as an attribute of
> the class, in particular of the object passed as the first argument.  To me
> your discussion is weakened by the implication that Python has a method
> class separate from functions.  (Bound methods are specialized
> partial-function wrappers; functools.partial is the general version.) For
> instance
>
>    could even work for regular functions if the first argument is an
>    object with attributes and/or methods that can be accessed with
>    standard "dot" notation.
>
> Def statements produce functions.  Period.
>
>    This proposal takes no position of
>    whether the latter should be allowed or not.
>
> To repeat: former *is* latter with a particular access path.  So the
> question is not whether to 'allow' $ to work in functions; that *is* your
> proposal, whether you know it or not.  To do otherwise would require some
> other significant change to Python, one that would complicate it.
>
>     Four possible candidates for the choice of that symbol are "~",
>    "@", "^", and "$".
>
> You left out "?" (not too good ;-) and now, in 3.0, "`" (backtick, no longer
> an repr synonym).  The latter is lighter than "$" but perhaps more visible
> than ".", depending on the font.  Hmmm, a comparison:

Minor point, but the backtick is NOT up for grabs, at least as of a
discussion on this very list from last year.

Quoting a response from Guido:
"""
On 1/2/07, Chris Rebert <cvrebert at gmail.com> wrote:
> Thus, I propose one of the following as the new use for the backtick (`):

You're missing one of the main reasons for removing the backtick
syntax in the first place: the character itself causes trouble by
looking too much like a regular quote (depending on your font), is
routinely mangled by typesetting software (as every Python book author
can testify), and requires a four-finger chord on Swiss keyboards. No
new uses for it will be accepted in Python 3000 no matter how good the
idea.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
"""

Not that I understand the relevance of Swiss keyboards, but anyway,
there you have it.
Perhaps we need to alias __future__.backtick to __future__.braces so
people remember this ;)

- Chris
========
Follow the path of the Iguana...
Rebertia: http://rebertia.com
Blog: http://blog.rebertia.com

>
> self.d = sqrt(self.x*self.x + self.y*self.y)
>
> s.d = sqrt(s.x*s.x + s.x*s.x)
>
> .d = sqrt(.x*.x + .y*.y)
>
> $d = sqrt($x*$x + $y*$y)
>
> `d = sqrt(`s*`x + `y*`y)
>
> I think I like this last best.
>
> Terry Jan Reedy
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From russ.paielli at gmail.com  Mon Aug 25 07:04:22 2008
From: russ.paielli at gmail.com (Russ Paielli)
Date: Sun, 24 Aug 2008 22:04:22 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <bbaeab100808242139y24e56e1awb47771b0da13a75e@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<bbaeab100808241653v3a811dc1jd11ca1e654157206@mail.gmail.com>
	<d03afaa20808241925q35ba797ahc9bad0ae7fd18b90@mail.gmail.com>
	<bbaeab100808242139y24e56e1awb47771b0da13a75e@mail.gmail.com>
Message-ID: <d03afaa20808242204h74222910nfe9768f2b65c68e0@mail.gmail.com>

On Sun, Aug 24, 2008 at 9:39 PM, Brett Cannon <brett at python.org> wrote:

> On Sun, Aug 24, 2008 at 7:25 PM, Russ Paielli <russ.paielli at gmail.com>
> wrote:
>
> > Think about this way: it's 80% less clutter.  I am a compulsive
> minimalist,
> > and one of the reasons I like Python is because it minimizes clutter. I
> > probably let clutter bother me more than I should. I really appreciate
> the
> > lack of semicolons all over the place. Some would call that trivial, but
> I
> > call it significant.
>
> You call it clutter, I call it information. We have kept 'self'
> explicit for a reason; it's self-documenting.
>

First,  "self."  conveys no more "information" than the "$" I am proposing,
but it requires five times as many characters. If that's not clutter, I
don't know what is.

Second, "self." actually conveys *less* information than "$", because it's
meaning depends on whether or not the first formal argument was actually
"self".

For the record, I gladly concede that I probably don't know as much about
Python as most of the other people on this mailing list (as I wrote earlier,
I am an aerospace engineer). But I also sense that many Python experts have
a blind spot on this matter for some reason. I guess these folks are just so
used to seeing "self." everywhere that it has burned itself into their brain
to the point that they don't see it as the clutter that it is.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080824/e4f161b6/attachment.html>

From andrew at atoulou.se  Mon Aug 25 08:10:21 2008
From: andrew at atoulou.se (Andrew Akira Toulouse)
Date: Sun, 24 Aug 2008 23:10:21 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808242204h74222910nfe9768f2b65c68e0@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<bbaeab100808241653v3a811dc1jd11ca1e654157206@mail.gmail.com>
	<d03afaa20808241925q35ba797ahc9bad0ae7fd18b90@mail.gmail.com>
	<bbaeab100808242139y24e56e1awb47771b0da13a75e@mail.gmail.com>
	<d03afaa20808242204h74222910nfe9768f2b65c68e0@mail.gmail.com>
Message-ID: <c7450fba0808242310o47b5afd0q5efb501a79e7c4b8@mail.gmail.com>

I agree with Brett.

While it's true that $ is four character shorter than 'self.', it also loses
expressiveness by doing so. The point here seems to be that it distinguishes
local variables with instance variables. Whereas 'self.' conveys the idea
that this is an instance variable clearly, a symbol such as '$' or '@' does
not.

It's also true that using 'self' is a convention -- you can easily use 's'.
So if you're enough of a nutcase to REALLY want to save those precious four
extra characters, well, you can save three. Meanwhile, adding @ or $ to
syntax would confuse a well-established, well-understood, and
newbie-friendly practice and complicate the language's grammar. Not to
mention, prefixing a variable with $ might be distracting to programmers who
do PHP for a living. Yet another mental load to bear.

It is not a blind spot. I have said before, if not on this list, then to
fellow programmers, that I like Python because 'self.' is not implicit or
substituted with a symbol that I don't automatically grok. This is, in my
opinion, what Brett meant by self-documenting, and I'm inclined to agree. I
don't like programming in languages that don't make as clear a distinction
between local variables and instance variables. And it's not because I'm
used to programming in Python, either, as I am a student have seen more Java
code than Python, and took up Python as a hobby. Perhaps it is you who,
having (possibly) seen implicit or abbreviated instance variables day in and
day out, have burnt them into *your* brain. I certainly haven't.

--Andy

On Sun, Aug 24, 2008 at 10:04 PM, Russ Paielli <russ.paielli at gmail.com>wrote:

> On Sun, Aug 24, 2008 at 9:39 PM, Brett Cannon <brett at python.org> wrote:
>
>> On Sun, Aug 24, 2008 at 7:25 PM, Russ Paielli <russ.paielli at gmail.com>
>> wrote:
>>
>> > Think about this way: it's 80% less clutter.  I am a compulsive
>> minimalist,
>> > and one of the reasons I like Python is because it minimizes clutter. I
>> > probably let clutter bother me more than I should. I really appreciate
>> the
>> > lack of semicolons all over the place. Some would call that trivial, but
>> I
>> > call it significant.
>>
>> You call it clutter, I call it information. We have kept 'self'
>> explicit for a reason; it's self-documenting.
>>
>
> First,  "self."  conveys no more "information" than the "$" I am proposing,
> but it requires five times as many characters. If that's not clutter, I
> don't know what is.
>
> Second, "self." actually conveys *less* information than "$", because it's
> meaning depends on whether or not the first formal argument was actually
> "self".
>
> For the record, I gladly concede that I probably don't know as much about
> Python as most of the other people on this mailing list (as I wrote earlier,
> I am an aerospace engineer). But I also sense that many Python experts have
> a blind spot on this matter for some reason. I guess these folks are just so
> used to seeing "self." everywhere that it has burned itself into their brain
> to the point that they don't see it as the clutter that it is.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080824/60fb249f/attachment.html>

From russ.paielli at gmail.com  Mon Aug 25 08:29:57 2008
From: russ.paielli at gmail.com (Russ Paielli)
Date: Sun, 24 Aug 2008 23:29:57 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <c7450fba0808242310o47b5afd0q5efb501a79e7c4b8@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<bbaeab100808241653v3a811dc1jd11ca1e654157206@mail.gmail.com>
	<d03afaa20808241925q35ba797ahc9bad0ae7fd18b90@mail.gmail.com>
	<bbaeab100808242139y24e56e1awb47771b0da13a75e@mail.gmail.com>
	<d03afaa20808242204h74222910nfe9768f2b65c68e0@mail.gmail.com>
	<c7450fba0808242310o47b5afd0q5efb501a79e7c4b8@mail.gmail.com>
Message-ID: <d03afaa20808242329n4e29cd91h89df36b142269e7d@mail.gmail.com>

On Sun, Aug 24, 2008 at 11:10 PM, Andrew Akira Toulouse
<andrew at atoulou.se>wrote:

> I agree with Brett.
>
> While it's true that $ is four character shorter than 'self.', it also
> loses expressiveness by doing so. The point here seems to be that it
> distinguishes local variables with instance variables. Whereas 'self.'
> conveys the idea that this is an instance variable clearly, a symbol such as
> '$' or '@' does not.
>

Yes it does. Or, rather, it could if allowed to do so.

The notion that Python programmers are too simple minded to adapt to a
simple notational convention seems a bit bizarre to me. Of course it does
not seem "natural" to you yet, because it is a new idea and you have not had
a chance to grow accustomed to it. But I think that anyone with half or more
of a brain could become completely comfortable with the idea within an hour
or two if they would just give a chance.

--Russ
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080824/e8bd49f2/attachment.html>

From andrew at atoulou.se  Mon Aug 25 08:45:57 2008
From: andrew at atoulou.se (Andrew Akira Toulouse)
Date: Sun, 24 Aug 2008 23:45:57 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808242329n4e29cd91h89df36b142269e7d@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<bbaeab100808241653v3a811dc1jd11ca1e654157206@mail.gmail.com>
	<d03afaa20808241925q35ba797ahc9bad0ae7fd18b90@mail.gmail.com>
	<bbaeab100808242139y24e56e1awb47771b0da13a75e@mail.gmail.com>
	<d03afaa20808242204h74222910nfe9768f2b65c68e0@mail.gmail.com>
	<c7450fba0808242310o47b5afd0q5efb501a79e7c4b8@mail.gmail.com>
	<d03afaa20808242329n4e29cd91h89df36b142269e7d@mail.gmail.com>
Message-ID: <c7450fba0808242345i23234d95kbba8d6b48bae8da6@mail.gmail.com>

Then what of the notion that you are not too simple-minded to adapt to a
simple notational convention? Or your conjecture that Python programmers are
too accustomed to 'self.' to see it for the clutter it is. Could I not, by
the same reasoning, say that you are too accustomed to $ to see its
arbitrary and highly overloaded meaning? Could you not become accustomed to
'self.' if you give it a chance? I think you could.

--Andy

(sorry Russ for the double-email, I forgot to reply-all)

On Sun, Aug 24, 2008 at 11:29 PM, Russ Paielli <russ.paielli at gmail.com>wrote:

>
>
> On Sun, Aug 24, 2008 at 11:10 PM, Andrew Akira Toulouse <andrew at atoulou.se
> > wrote:
>
>> I agree with Brett.
>>
>> While it's true that $ is four character shorter than 'self.', it also
>> loses expressiveness by doing so. The point here seems to be that it
>> distinguishes local variables with instance variables. Whereas 'self.'
>> conveys the idea that this is an instance variable clearly, a symbol such as
>> '$' or '@' does not.
>>
>
> Yes it does. Or, rather, it could if allowed to do so.
>
> The notion that Python programmers are too simple minded to adapt to a
> simple notational convention seems a bit bizarre to me. Of course it does
> not seem "natural" to you yet, because it is a new idea and you have not had
> a chance to grow accustomed to it. But I think that anyone with half or more
> of a brain could become completely comfortable with the idea within an hour
> or two if they would just give a chance.
>
> --Russ
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080824/7fbfe501/attachment.html>

From bruce at leapyear.org  Mon Aug 25 09:45:34 2008
From: bruce at leapyear.org (Bruce Leban)
Date: Mon, 25 Aug 2008 00:45:34 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <g8t1on$7ne$1@ger.gmane.org>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
Message-ID: <cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>

Jim Jewett wrote:

> One of the problems with the current syntax is that calling sites
>> don't match the definition site the way they do for normal functions.
>> This proposal doesn't seem to help with that.
>> ...
>>
>    def m(self, a, b, z=None): ...
>>
>>    # self is moved outside the parens, changing the tuple size
>>    self.m(a, b, z)
>>    self.m(a, b)
>>
>
> On Sun, Aug 24, 2008 at 6:26 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> If m is an attribute of type(s) (which is s.__class__), this shrinkage is a
> convenient abbreviation, not a requirement.  The above calls are the same as
> type(s).m(s.a,b,z) or type(s).m(s,a,b).  Or, if you prefer,
> fn = type(s).m # or s.__class__.m
> fn(s,a,b,z)
> If m is an attribute s, and s in not a class, the shrinkage is not
> available, and one must write s.m(s,a,b,z) or s.m(s.a,b).
>
> Terry Jan Reedy


If I could write:

  class foo:
    def self.bar():
      self.rebar(self.babar)

then the call to  object.bar() would match the declaration.

Back to Russ's proposal: it would be better accomodated IMHO by allowing $
as a character in a variable name, just like _ is. Then, conventionally,
people could use $ as self:

  def $.bar():
    $.rebar($.babar)

and for whatever it's worth, I find $.bar easier to read then $bar as the
explicit dot reminds me it's doing an attribute get rather than looking like
a special variable name.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080825/8dbb2899/attachment.html>

From bmintern at gmail.com  Mon Aug 25 12:35:33 2008
From: bmintern at gmail.com (Brandon Mintern)
Date: Mon, 25 Aug 2008 06:35:33 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
Message-ID: <4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>

On Mon, Aug 25, 2008 at 3:45 AM, Bruce Leban <bruce at leapyear.org> wrote:
> Jim Jewett wrote:
> On Sun, Aug 24, 2008 at 6:26 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> If m is an attribute of type(s) (which is s.__class__), this shrinkage is
>> a convenient abbreviation, not a requirement.  The above calls are the same
>> as
>> type(s).m(s.a,b,z) or type(s).m(s,a,b).  Or, if you prefer,
>> fn = type(s).m # or s.__class__.m
>> fn(s,a,b,z)
>> If m is an attribute s, and s in not a class, the shrinkage is not
>> available, and one must write s.m(s,a,b,z) or s.m(s.a,b).
>>
>> Terry Jan Reedy
>
> If I could write:
>
>   class foo:
>     def self.bar():
>       self.rebar(self.babar)
>
> then the call to  object.bar() would match the declaration.

+1. I like this proposal: it's clean and sensible, and I think it
looks a lot more clear, especially to someone who is new to Python.

> Back to Russ's proposal: it would be better accomodated IMHO by allowing $
> as a character in a variable name, just like _ is. Then, conventionally,
> people could use $ as self:
>
>   def $.bar():
>     $.rebar($.babar)
>
> and for whatever it's worth, I find $.bar easier to read then $bar as the
> explicit dot reminds me it's doing an attribute get rather than looking like
> a special variable name.

+1 I'm in agreement here as well. The closest I would get to
supporting the $-for-self. proposal is allowing $ in variable names.
Having used Lisp before Python, I'm used to just about any character
being allowed in identifier names. The more characters available, the
more expressive the names can be, and I think it could only help
things. The overhead of people having to get used to new symbols being
used in variable names would be small in comparison to the flexibility
offered.

Brandon


From russ.paielli at gmail.com  Mon Aug 25 13:19:08 2008
From: russ.paielli at gmail.com (Russ Paielli)
Date: Mon, 25 Aug 2008 04:19:08 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
Message-ID: <d03afaa20808250419tff72d4dm9069107fcf03ced7@mail.gmail.com>

On Mon, Aug 25, 2008 at 12:45 AM, Bruce Leban <bruce at leapyear.org> wrote:

> Jim Jewett wrote:
>
> If I could write:
>
>   class foo:
>     def self.bar():
>       self.rebar(self.babar)
>
> then the call to  object.bar() would match the declaration.
>
> Back to Russ's proposal: it would be better accomodated IMHO by allowing $
> as a character in a variable name, just like _ is. Then, conventionally,
> people could use $ as self:
>
>   def $.bar():
>     $.rebar($.babar)
>
> and for whatever it's worth, I find $.bar easier to read then $bar as the
> explicit dot reminds me it's doing an attribute get rather than looking like
> a special variable name.
>

  def $.bar():
    $.rebar($.babar)

That's two separate proposals, but I think I like both of them.

Of course, Python already allows "S", which is very similar to "$", as the
first argument, so we're almost there on that aspect of it. Come to think of
it, I may start using "S" and see how it works out.

So how about

    def S.bar():
        S.rebar(S.babar)

--Russ
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080825/e66c77cb/attachment.html>

From grosser.meister.morti at gmx.net  Mon Aug 25 14:43:41 2008
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Mon, 25 Aug 2008 14:43:41 +0200
Subject: [Python-ideas] Inline 'raises' expression
In-Reply-To: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
Message-ID: <48B2A8FD.7030906@gmx.net>

Fredrik Johansson schrieb:
> Hi,
>
> It happens that I'm just interested in whether an expression raises an
> exception, not the return value. This might look something like
>
>     try:
>         check_status()
>     except IOError:
>         cleanup()
>
> which could be written more simply as
>
>     if check_status() raises IOError:
>         cleanup()
>
> Also, instead of the inconvenient
>
>     self.assertRaises(ZeroDivisionError, lambda: 1/0)
>
> one could just write
>
>     assert 1/0 raises ZeroDivisionError
>
> Something like this would especially be useful for those of us who
> aren't fans of the unittest framework. Alternatively, just the
> assert--raises form could be permitted.
>
> Thoughts?
>
> Fredrik

I think an extra syntax for such an easy task is unnecessary.
The following function emulates this functionality sufficiently:

def raises(f,e):
	try:
		f()
	except e:
		return True
	except:
		return False
	else:
		return False

def f(x):
	if x != 2:
		raise ValueError()

>>> raises(lambda:f(1),ValueError)
True
>>> raises(lambda:f(2),ValueError)
False
>>> raises(lambda:f(1),Exception)
True
>>> raises(lambda:f(2),Exception)
False
>>> raises(lambda:f(1),TypeError)
False
>>> raises(lambda:f(2),TypeError)
False


From tom at vector-seven.com  Mon Aug 25 15:13:11 2008
From: tom at vector-seven.com (Thomas Lee)
Date: Mon, 25 Aug 2008 23:13:11 +1000
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808250419tff72d4dm9069107fcf03ced7@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>	<g8t1on$7ne$1@ger.gmane.org>	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<d03afaa20808250419tff72d4dm9069107fcf03ced7@mail.gmail.com>
Message-ID: <48B2AFE7.1050000@vector-seven.com>

I really don't think any of the proposed solutions are logically, 
functionally or aesthetically better than what we currently have with 
"def foo(self)".

I find all proposed variants of "$.foo" to be jarring. "def self.bar()" 
is somewhat less ugly, but misleading since the method is actually being 
bound to its outer class, not "self" (presumably, any instances of that 
class rather than the class itself):

 >>> class Foo(object):
...     def bar(self):
...             print "Hello, 0x%x" % id(self)
...
 >>> foo = Foo()
 >>> Foo.bar(foo)
Hello, 0xb7729f8c
 >>> foo.bar()
Hello, 0xb7729f8c
 >>>

Note that the declaration of "bar" binds the function to class "Foo", 
not to the instance "foo". Would these semantics change with the 
"self.bar()" syntax?

Either way it's a -1 from me until a more Pythonic proposal surfaces. 
I'm not sure there's a way to improve on the simplicity of the existing 
semantics though.

Cheers,
T

Russ Paielli wrote:
>
>
> On Mon, Aug 25, 2008 at 12:45 AM, Bruce Leban <bruce at leapyear.org 
> <mailto:bruce at leapyear.org>> wrote:
>
>     Jim Jewett wrote:
>
>     If I could write:
>
>       class foo:
>         def self.bar():
>           self.rebar(self.babar)
>
>     then the call to  object.bar() would match the declaration.
>
>     Back to Russ's proposal: it would be better accomodated IMHO by
>     allowing $ as a character in a variable name, just like _ is.
>     Then, conventionally, people could use $ as self:
>
>       def $.bar():
>         $.rebar($.babar)
>
>     and for whatever it's worth, I find $.bar easier to read then $bar
>     as the explicit dot reminds me it's doing an attribute get rather
>     than looking like a special variable name.
>
>
>   def $.bar():
>     $.rebar($.babar)
>
> That's two separate proposals, but I think I like both of them.
>
> Of course, Python already allows "S", which is very similar to "$", as 
> the first argument, so we're almost there on that aspect of it. Come 
> to think of it, I may start using "S" and see how it works out.
>
> So how about
>
>     def S.bar():
>         S.rebar(S.babar)
>
> --Russ
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>   



From fredrik.johansson at gmail.com  Mon Aug 25 15:55:25 2008
From: fredrik.johansson at gmail.com (Fredrik Johansson)
Date: Mon, 25 Aug 2008 15:55:25 +0200
Subject: [Python-ideas] Inline 'raises' expression
In-Reply-To: <48B2A8FD.7030906@gmx.net>
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
	<48B2A8FD.7030906@gmx.net>
Message-ID: <3d0cebfb0808250655w536df8e1uf1da0f681bb7d611@mail.gmail.com>

On Sun, Aug 24, 2008 at 4:01 PM, Paul Hankin <paul.hankin at pobox.com> wrote:
> There's not much need for a language modification; you can code it up
> directly. The obvious problem with what you want is that it's only
> good if you want to catch a single exception.
>
> def raises(thunk, exn):
>    try:
>        thunk()
>        return False
>    except exn:
>        return True

True, but that's a lot of noise for something very simple. And the
extra required "lambda:" makes the "raises" expression much less
readable.

On Mon, Aug 25, 2008 at 1:47 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Fredrik Johansson wrote:
>
>> one could just write
>>
>>    assert 1/0 raises ZeroDivisionError
>
> It's been pointed out that using assert for testing
> is not the best idea, since then you can't test that
> the code works properly with assertions turned off.

Good point, though I've never personally felt the need to run Python
with assertions turned off. (It would arguably be more useful if
assertions were enabled on a per module basis, but this is a
tangential matter.)

Fredrik


From rrr at ronadam.com  Mon Aug 25 16:19:02 2008
From: rrr at ronadam.com (Ron Adam)
Date: Mon, 25 Aug 2008 09:19:02 -0500
Subject: [Python-ideas] Inline 'raises' expression
In-Reply-To: <B640FFB56C3E42FF848479B430CB5E36@RaymondLaptop1>
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
	<B640FFB56C3E42FF848479B430CB5E36@RaymondLaptop1>
Message-ID: <48B2BF56.4060907@ronadam.com>



Raymond Hettinger wrote:
> From: "Fredrik Johansson" <fredrik.johansson at gmail.com>
>> one could just write
>>
>>    assert 1/0 raises ZeroDivisionError
> 
> +1
>
> This would be a nice extension to the assertion syntax.

+1

And to do the same thing in other ways isn't as easy as it seems.




Regarding ignoring asserts by -O,  Would it be possible to change asserts 
so they default to off instead of on?  I'd rather have -debug set __debug__ 
to True rather than -0 setting it to False.

Personally I'd like asserts to be part of the testing framework.  ie... 
they are *always skipped* unless possibly a -test or -debug  flag is specified.

Another useful feature would be to specify where asserts sends it's error.


What I'd like to do is to be able to sprinkle asserts throughout my code as 
a way to do simple tests and then possibly ...

    python -debug module.py   ... enable asserts to output to stderr



Or to use plain asserts effectively in unittests and doctests ...

    python -test module.py    ... enamble asserts and run unittests
                                          or doctests for module.

    python -test                  ... run python test suite


Is it too late to go this direction for 3.0, 3.1... ?

Ron















From tjreedy at udel.edu  Mon Aug 25 17:46:06 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Aug 2008 11:46:06 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>	<g8t1on$7ne$1@ger.gmane.org>	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
Message-ID: <g8uk3r$9sf$1@ger.gmane.org>



Brandon Mintern wrote:

> Having used Lisp before Python, I'm used to just about any character
> being allowed in identifier names. The more characters available, the
> more expressive the names can be, and I think it could only help
> things. 

3.0 uses the standard Unicode syntax for identifiers:
"
identifier  ::=  id_start id_continue*

id_start    ::=  <all characters in general categories Lu, Ll, Lt, Lm, 
Lo, Nl, the underscore, and characters with the Other_ID_Start property>

id_continue ::=  <all characters in id_start, plus characters in the 
categories Mn, Mc, Nd, Pc and others with the Other_ID_Continue property>
"
This is 10s of thousands of characters ;-)
-- but no symbols.
"
The Unicode category codes mentioned above stand for:

Lu - uppercase letters
Ll - lowercase letters
Lt - titlecase letters
Lm - modifier letters
Lo - other letters
Nl - letter numbers
Mn - nonspacing marks
Mc - spacing combining marks
Nd - decimal numbers
Pc - connector punctuations

All identifiers are converted into the normal form NFC while parsing; 
comparison of identifiers is based on NFC.
"

I think we should stick with this international cross-language standard.

tjr



From aahz at pythoncraft.com  Mon Aug 25 19:08:46 2008
From: aahz at pythoncraft.com (Aahz)
Date: Mon, 25 Aug 2008 10:08:46 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808242329n4e29cd91h89df36b142269e7d@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<bbaeab100808241653v3a811dc1jd11ca1e654157206@mail.gmail.com>
	<d03afaa20808241925q35ba797ahc9bad0ae7fd18b90@mail.gmail.com>
	<bbaeab100808242139y24e56e1awb47771b0da13a75e@mail.gmail.com>
	<d03afaa20808242204h74222910nfe9768f2b65c68e0@mail.gmail.com>
	<c7450fba0808242310o47b5afd0q5efb501a79e7c4b8@mail.gmail.com>
	<d03afaa20808242329n4e29cd91h89df36b142269e7d@mail.gmail.com>
Message-ID: <20080825170846.GB29088@panix.com>

On Sun, Aug 24, 2008, Russ Paielli wrote:
>
> The notion that Python programmers are too simple minded to adapt to
> a simple notational convention seems a bit bizarre to me. Of course
> it does not seem "natural" to you yet, because it is a new idea and
> you have not had a chance to grow accustomed to it. But I think
> that anyone with half or more of a brain could become completely
> comfortable with the idea within an hour or two if they would just
> give a chance.

Consider that most of us came to Python from some other background, most
notably P*** -- and in many cases, things like dollar signs were part of
we're escaping.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!


From dangyogi at gmail.com  Mon Aug 25 18:48:22 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Mon, 25 Aug 2008 12:48:22 -0400
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
Message-ID: <48B2E256.6080008@gmail.com>

It was suggested that I rearrange the micro-threading PEP proposal to 
place the juicy Python stuff up front.

So I've done this here.  And now that I see that people are back from 
the toils of creating 3.0b3 and starting to comment again on 
python-ideas, it seems like a good time to repost this!  (I guess my 
first post was really bad timing..., sorry about that!)

I ask that you provide feedback.  I have no direct need for this, so 
don't really have a horse in the race.  But it was an idea that I 
thought might be very useful to the Python community, seeing the 
emphasis on web servers, so am making an effort here to run it up the 
flagpole...  If this ends up going into a trial version, I am prepared 
to help considerably with the implementation.   If you don't think that 
Python needs such silliness, that's OK and I'd like to hear that too (it 
will mean a lot less work for me! ;-) ).

I don't imagine that this PEP represents an /easy/ way to solve this 
problem, but do imagine that it is the /right/ way to solve it.  Other 
similar proposals have been made in past years that looked at easier 
ways out.  These have all been rejected.  But I don't think that there 
are really any easy ways out that are robust solutions, and so I offer 
this one.  If I am wrong, and the reason that the prior proposals were 
rejected is due to a lack of need, rather than a lack of robustness, 
then this proposal should also be rejected.  This might be the case if, 
for example, all Python programs end up being unavoidably CPU bound so 
that micro-threading would provide little real benefit.

If there /is/ a perceived need for this, then I am sure that this PEP 
would benefit from your TLC and other ideas!

If you read the previous version, the only changes here are a little 
more specificity in the Python section.

Thank you for your attention on this!

-bruce


Abstract
========

This PEP adds micro-threading (or `green threads`_) at the C level so that
micro-threading is built in and can be used with very little coding effort
at the Python level.

The implementation is quite similar to the Twisted_ [#twisted-fn]_
Deferred_/Reactor_ model, but applied at the C level by extending the
`C API`_ [#c_api]_ slightly.  Doing this provides the Twisted
capabilities to Python, but without requiring the Python programmer to code
in the Twisted event driven style.  Thus, legacy Python code would gain the
benefits that Twisted provides with very little modification.

Burying the event driven mechanism in the C level also makes the same
benefits available to Python GUI interface tools so that the Python
programmers don't have to deal with event driven programming there either.

This capability is also used to provide some of the features that
`Stackless Python`_ [#stackless]_ provides, such as microthreads and
channels (here, called micro_pipes).

.. _Twisted: http://twistedmatrix.com/trac/
.. _Deferred:
   http://twistedmatrix.com/projects/core/documentation/howto/defer.html
.. _Reactor:
   
http://twistedmatrix.com/projects/core/documentation/howto/reactor-basics.html
.. _C API: http://docs.python.org/api/api.html
.. _green threads: http://en.wikipedia.org/wiki/Green_threads

Motivation
==========

The popularity of the Twisted project has demonstrated the need for a
micro-threading alternative to the standard Posix thread_ [#thread-module]_
and threading_ [#threading-module]_ packages.  Micro-threading allows large
numbers (1000's) of simultaneous connections to Python servers, as well
as fan-outs to large numbers of downstream connections.

The advantages to the Twisted approach over Posix threads are:

#. much less memory is required per thread
#. faster thread creation
#. faster context switching (I'm guessing on this one, is this really true?)
#. synchronization between threads is easier because there is no preemption,
   making it much easier to write critical sections of code.

The disadvantages are:

#. the Python developer must write his/her program in an event driven style
#. the approach can not be used with standard Python code that wasn't
   written in this event driven style
#. the approach does not take advantage of multiple processor architectures
#. since there is no preemption, a long running micro-thread will starve
   other micro-threads

This PEP attempts to retain all of the advantages that Twisted has
demonstrated, and to resolve the first two disadvantages to make the
advantages accessible to all Python programs, including legacy programs
not written in the Twisted style.  This should make it very easy for legacy
programs like WSGI apps, Django and TurboGears to reap the benefits of
Twisted.

Another example of event driven mechanisms are the GUI/windows events.  This
PEP also makes it easy for Python GUI interface toolkits (like wxpython
and qtpython) to hide the GUI/windows event driven style of programming from
the Python programmer.  For example, you would no longer need to use modal
dialog boxes just to make the programming easier.

This PEP does not address the last two disadvantages, and thus also has
these disadvantages itself.

The primary inspiration for this PEP comes from the Twisted_ [#twisted-fn]_
project.

If the C level deals with the Deferred objects, then the Python level 
wouldn't
have to.  And if that is the case, this would greatly lower the bar to 
Python
programmers desiring the benefits that Twisted provides and make those
benefits available to all Python programmers essentially for free.

The secondary inspiration was to treat the Deferreds as a special case of
exceptions, which are already designed to unwind the C stack.  This lets us
take a more piecemeal approach to implementing the PEP at the C level 
because
an unmodified C function used in a situation where its execution would have
to be deferred is gracefully caught as a standard exception.  In addition,
this exception can report the name of the unmodified C function in its
message.  So we don't need to change *everything* that might be affected on
a first roll out.

It also adds deferred processing without adding additional checks after each
C function call to see whether to defer execution.  The check that is 
already
being done for exceptions doubles as a check for deferred processing.

Finally, once Python has this deferred mechanism in place at the C level,
many things become quite easy at the Python level.  This includes full
micro-threading, micro-pipes between micro-threads, new-style generators 
that
can delegate responsibility for generating values to called functions 
without
having to intervene between their caller and the called function, parallel
execution constructs (``parallel_map``).

It is expected that many more of these kind of devices will be easily
implementable once the underlying deferred mechanism in place.


Specification of Python Layer Enhancements
==========================================

Fortunately, at the Python level, the programmer does not see the underlying
`C deferred`_, `reactor function`_, or notifier_ objects.  The Python
programmer will see three things:

#. An addition of non_blocking modes of accessing files, sockets, time.sleep
   and other functions that may block.  It is not clear yet exactly what 
these
   will look like.  The possibilities are:

   - Add an argument to the object creation functions to specify blocking or
     non-blocking.
   - Add an operation to change the blocking mode after the object has been
     created.
   - Add new non-blocking versions of the methods on the objects that may
     block (e.g., read_d/write_d/send_d/recv_d/sleep_d).
   - Some combination of these.

   If an object is used in blocking mode, then all micro-threads (within its
   Posix thread_) will block.  So the Python programmer must set 
non-blocking
   mode on these objects as a first step towards taking advantage of
   micro-threading.

   It may also be useful to add a locking capability to files and sockets so
   that code (like traceback.print_exception) that outputs several lines can
   prevent other output from being intermingled with it.

#. Micro_thread objects.  Each of these will have a re-usable C deferred
   object attached to it, since each micro_thread can only be suspended
   waiting for one thing at a time.  The current micro_thread would be 
stored
   within a C global variable, much like ``_PyThreadState_Current``.  If the
   Python programmer isn't interested in micro_threading, micro_threads 
can be
   safely ignored (like posix threads, you get one for free, but don't 
have to
   be aware of it).  If the programmer *is* interested in micro-threading,
   then s/he must create additional micro_threads.  Each micro-thread 
would be
   created to run a single Python function.  When that function returns, the
   micro-thread is finished.

   There are three usage scenarios, aided by three different functions to
   create micro-threads:

   #. Create a micro-thread to do something, without regard to the final
      value returned from *function*.  An example here would be a web server
      that has a top-level ``socket.accept`` loop that runs a
      ``handle_client`` function in a separate micro_thread on each new
      connection.  Once launched, the ``socket.accept`` thread is no longer
      interested in the ``handle_client`` threads.

      In this case, the normal return value of the ``handle_client`` 
function
      can be discarded.  But what should be done with exceptions that 
are not
      caught in the child threads?

      Therefore, this style of use would allow a top-level exception handler
      for the new thread::

          start_and_forget(function, *args,
                           exception_handler=traceback.print_exception,
                           **kws)

      The parent thread does not need to do any kind of *wait* after the 
child
      thread is complete.  It will either complete normally and go away
      silently (with any final return value ignored), or raise an uncaught
      exception, which is passed to the indicated exception_handler, and 
then
      go away without further ado.

   #. Create micro_threads to run multiple long-running *functions* in
      parallel where the final return value from each *function* is 
needed by
      the parent thread::

          thread = start_in_parallel(function, *args, **kws)

      In this case, the parent thread is expected to do a *thread.wait()*
      when it is ready for the return value of the function.  Thus, 
completed
      micro_threads will form zombie threads until their parents retrieve
      their final return values (much like unix processes).

      On doing the *wait*, an uncaught exception in the child 
micro_thread is
      re-raised in the parent micro_thread.

      It might be nice, for example, to have a ``parallel_map`` function 
that
      will create a micro_thread for each element of its *iterable* argument
      in order to run the mapping function on all of them in parallel 
and then
      return an iterable of the waited for results.

   #. In the above examples, the child micro_threads are completely
      independent of each other -- i.e., they don't communicate with each
      other except for child threads returning a final value to their 
parents.
     
      This final scenario uses *micro_pipes* to allow threads to 
cooperatively
      solve problems (much like unix pipes)::

          pipe = generate(function, *args, **kws)

      These micro_threads have a micro_pipe associated with them (called
      *stdout*).  When a micro_thread is finished it goes away silently (and
      the final return value from the *function* is ignored).

      The pipe looks like a normal Python iterator, but is designed to 
be read
      by a different micro-thread than the one generating the values.

      Uncaught exceptions in the micro_thread generating the values are
      propagated through the micro_pipe to the micro_pipe's reader.

#. Micro_pipes.  Micro_pipes are one-way pipes that allow synchronized
   communication between micro_threads.

   The protocol for the receiving side of the pipe is simply the standard
   Python iterator protocol.  Thus, for example, they can be directly used
   in ``for`` statements.

   The sending side has these methods:

   - ``put(object)`` to send *object* to the receiving side (retrieved with
     the ``__next__`` method).
   - ``take_from(iterable)`` to send a series of objects to the receiving
     side.
   - ``close()`` to cause a ``StopIteration`` on the ``__next__`` call.
     A ``put`` done after a ``close`` silently terminates the micro_thread
     doing the ``put`` (in case the receiving side closes the micro_pipe).

   Micro_pipes are automatically associated with micro_threads, making 
it less
   likely to hang the program:

   >>> pipe = micro_pipe()
   >>> next(pipe)  # hangs the program!  No micro_thread created to feed 
pipe...

   So each micro_thread may have a *stdout* micro_pipe assigned to them and
   may also be assigned a *stdin* micro_pipe (some other micro_thread's 
stdout
   micro_pipe).  When the micro_thread terminates, it automatically calls
   ``close`` on its stdin and stdout micro_pipes.

   To easily access the stdout micro_pipe of the current micro_thread, new
   ``put`` and ``take_from`` built-in functions are provided::

       put(object)
       take_from(iterable)

   In addition, the current built-in ``iter`` and ``next`` functions 
would be
   modified so that they may be called with no arguments.  In this case, 
they
   would use the current micro_thread's *stdin* pipe as their argument.

   Micro_pipes let us write generator functions in a new way by having the
   generator do ``put(object)`` rather than ``yield object``.  In this case,
   the generator function has no ``yield`` statement, so is not treated
   specially by the compiler.  Basically this means that calling a new-style
   generator does not automatically create a new micro_thread (sort of what
   calling an old-style generator does).

   The ``put(object)`` does the same thing as ``yield object``, but
   allows the generator to share the micro_pipe with other new-style
   generator functions (by simply calling them) and old-style generators (or
   any iterable) by calling ``take_from`` on them.  This lets the generator
   delegate to other generators without having to get involved with passing
   the results back to its caller.

   For example, a generator to output all the odd numbers from 1-n::

       def odd(n):
           take_from(range(1, n, 2))

   These "new-style" generators would have to be run in their own
   micro_thread:

   >>> pipe = generate(odd, 100)
   >>> # now pipe is an iterable representing the generator:
   >>> print tuple(pipe)

   The generator is then not restricted to having its own micro_thread.  It
   could also be used as a helper by other generators from the other
   generator's micro_thread without having to create additional 
micro-threads
   or doing "bucket brigades" to yield values from the helper back to the
   other generator's caller.  For example::

       def even(n):
           take_from(range(2, n, 2))

       def odd_even(n):
           odd(n)
           even(n)

   At this point ``generate`` could be called on any of these three 
generators
   (``odd``, ``even`` or ``odd_even``).

Specification of C Layer Enhancements
=====================================

This is where most of the work is to implement this PEP.  These are the
underlying mechanisms that make the whole thing "tick".

Basically, this is a C Deferred that micro-thread aware C functions deal 
with
to be put to sleep and avoid blocking; and a Reactor to wake the Deferreds
back up when the event occurs that they are waiting for.  This is very 
similar
in concept to the Twisted Deferred and Reactor, just done at the C level so
that Python programmers don't have to deal with them.

C Deferred
----------

``PyDeferred_CDeferred`` is written as a new exception type for use by the
C code to defer execution.  This is a subclass of ``NotImplementedError``.
Instances are not raised as a normal exception (e.g., with
``PyErr_SetObject``), but by calling ``PyNotifier_Defer`` (described in the
Notifier_ section, below).  This registers the ``PyDeferred_CDeferred``
associated with the currently running micro_thread as the current error 
object,
but also readies it for its primary job -- deferring execution.  As an
exception, it creates its own error message, if needed, which is
"Deferred execution not yet implemented by %s" % c_function_name.

``PyErr_ExceptionMatches`` may be used with these.  This allows them to be
treated as exceptions by non micro-threading aware (*unmodified*) C 
functions.

But these C deferred objects serve as special indicators that are treated
differently than normal exceptions by micro-threading aware (*modified*)
C code.  Modified C functions do this by calling ``PyDeferred_AddCallback``,
or explicitly checking ``PyErr_ExceptionMatches(PyDeferred_CDeferred)`` 
after
receiving an error return status from a called function.

``PyDeferred_CDeferred`` instances offer the following methods (in addition
to the normal exception methods):

- ``int PyDeferred_AddCallbackEx(PyObject *deferred, const char 
*caller_name,
  const char *called_name, PyObject *(*callback_fn)(PyObject 
*returned_object,
  void *state), void *state)``

  - The *caller_name* and *called_name* are case sensitive.  The 
*called_name*
    must match exactly the *caller_name* used by the called function when it
    dealt with this *deferred*.  If the names are different, the *deferred*
    knows that an intervening unmodified C function was called.  This is 
what
    triggers it to then act like an exception.

    The *called_name* must be ``NULL`` when called by the function that
    executed the ``PyNotifier_Defer`` to initiate the deferring process.

  - The *callback_fn* will be called with the ``PyObject`` of the results of
    the prior registered callback_fn.  An exception is passed to
    *callback_fn* by setting the exception and passing ``NULL`` (just like
    returning an exception from a C function).  In the case that the
    *deferred* initially accepts some *callback_fns* after a
    ``PyNotifier_Defer`` is done, and then later has to reject them (because
    of encountering the exception case, above), it will pass itself again,
    now acting like an exception, to all of these new callback_fns to allow
    them to clean up.  It then returns 0 to continue to be treated as an
    exception (see the explanation for ``PyDeferred_Callback``, below).
  - The *callback_fn* is always guaranteed to be called exactly once at some
    point in the future.  It will be passed the same *state* value as was
    passed with it to ``PyDeferred_AddCallback``.  It is up to the
    *callback_fn* to deal with the memory management of this *state* object.
  - The *callback_fn* may be ``NULL`` if no callback is required.  But in
    this case ``PyDeferred_AddCallback`` must still be called to notify the
    *deferred* that the C function is micro-threading aware.
  - This returns 0 if it fails (is acting like an exception), 1 otherwise.
    If it fails, the caller should do any needed clean up because the caller
    won't be resumed by the *deferred* (i.e., *callback_fn* will not be
    called).

- ``int PyDeferred_AddCallback(const char *caller_name, const char 
*called_name,
  PyObject *(*callback_fn)(PyObject *returned_object, void *state),
  void *state)``

  - Same as ``PyDeferred_AddCallbackEx``, except that the deferred object is
    taken from the *value* object returned by ``PyErr_Fetch``.  If the 
*type*
    returned by ``PyErr_Fetch`` is not ``PyDeferred_CDeferred``, 0 is 
returned.
    Thus, this function can be called after any exception and then other
    standard exception processing done if 0 is returned (including checking
    for other kinds of exceptions).

- ``int PyDeferred_IsExceptionEx(PyObject *deferred)``

  - Returns 1 if *deferred* is in exception mode, 0 otherwise.

- ``int PyDeferred_IsException(void)``

  - Same as ``PyDeferred_IsExceptionEx``, except that the deferred object is
    taken from the *value* object returned by ``PyErr_Fetch``.  If the 
*type*
    returned by ``PyErr_Fetch`` is not ``PyDeferred_CDeferred``, 1 is 
returned.
    Thus, this function can be called after any exception and then other
    standard exception processing done if 1 is returned (including checking
    for other kinds of exceptions).

- ``int PyDeferred_Callback(PyObject *deferred, PyObject *returned_object)``

  - This is called by the `reactor function`_ to resume execution of a
    micro_thread after the *deferred* has been scheduled with
    ``PyReactor_Schedule`` or ``PyReactor_ScheduleException``.
  - This calls the callback_fn sequence stored in *deferred* passing
    *returned_object* to the first registered callback_fn, and each
    callback_fn's returned ``PyObject`` to the next registered callback_fn.
  - To signal an exception to the callbacks, first set the error indicator
    (e.g. with ``PyErr_SetString``) and then call ``PyDeferred_Callback``
    passing ``NULL`` as the *returned_object* (just like returning ``NULL``
    from a C function to signal an exception).
  - If a callback_fn wants to defer execution, this same *deferred* object
    will be used by ``PyNotifier_Defer`` (since the callback_fn is 
running in
    the same micro_thread).  The *deferred* keeps the newly added 
callback_fns
    in the proper sequence relative the existing callback_fns that have not
    yet been executed (described below).  When *deferred* is returned from a
    callback_fn, no further callback_fns are called.

    Note that this check is also done on the starting *returned_object*, so
    that if this *deferred* exception is passed in, then none of its
    callback_fns are executed and it simply returns.

  - If a callback_fn defers, a final check is done to see if its name 
was the
    last one registered by a ``PyDeferred_AddCallback`` call.  If not, 
and if
    this *deferred* has not already been set into exception mode, the
    *deferred* sets itself into exception mode and raises itself through the
    entire callback_fn sequence.  This should end up terminating the
    micro_thread.

  - If a callback_fn starts to defer (by calling ``PyNotifier_Defer``) 
and then
    later raises some other exception, the *deferred* will know that 
it's been
    activated but not returned as the final error object by the callback_fn.
    In this case, the *deferred* raises a ``SystemError`` attaching the 
other
    exception to it as its ``__cause__`` and runs this through all new
    callback_fns that were added subsequent to the 
``PyNotifier_Defer``.  The
    ``SystemError`` exception is then cleared and the other exception
    reestablished (it will have the *deferred* as its ``__context__``).  The
    other exception is then passed to the remaining callback_fns to 
terminate
    the micro_thread.

  - If no callback_fn defers, then the micro_thread is finished executing.
    The results of the last callback_fn are treated as the final result 
of the
    micro_thread.  If the micro_thread has an ``exception_handler``, the
    ``exception_handler`` is used on the final exception (if there is 
one) and
    the micro_thread is deleted.

    If the micro_thread has no ``exception_handler``, the final return value
    (or exception) is stored in the micro_thread and the micro_thread is
    converted into a zombie state.  This will also result in a ``close``
    being done on the micro_thread's stdout micro_pipe.

  - Returns 0 on error, 1 otherwise.  Note that an error from the final
    callback_fn does not cause a 0 to be returned here.  Only if
    ``PyDeferred_Callback`` itself has a problem that it can't deal with is
    0 returned.

Each micro_thread has its own C deferred object associated with it.  This is
possible because each micro_thread may only be suspended for one thing at a
time.  This also allows us to re-use C deferreds and, through the following
trick, means that we don't need a lot of C deferred instances when a
micro_thread is deferred many times at different points in the call stack.

One peculiar thing about the stored callbacks, is that they're not really a
queue.  When the C deferred is first used and has no saved callbacks,
the callbacks are saved in straight FIFO manor.  Let's say that four
callbacks are saved in this order: ``D'``, ``C'``, ``B'``, ``A'`` (meaning
that ``A`` called ``B``, called ``C``, called ``D`` which deferred):

- after ``D'`` is added, the queue looks like: ``D'``
- after ``C'`` is added, the queue looks like: ``D'``, ``C'``
- after ``B'`` is added, the queue looks like: ``D'``, ``C'``, ``B'``
- after ``A'`` is added, the queue looks like: ``D'``, ``C'``, ``B'``, 
``A'``

Upon resumption, ``D'`` is called, then ``C'`` is called.  ``C'`` then calls
``E`` which calls ``F`` which now wants to defer execution again.  
``B'`` and
``A'`` are still in the deferred's callback queue.  When ``F'``, then ``E'``
then ``C''`` are pushed, they go in front of the callbacks still present
from the last defer:

- after ``F'`` is added, the queue looks like: ``F'``, ``B'``, ``A'``
- after ``E'`` is added, the queue looks like: ``F'``, ``E'``, ``B'``, 
``A'``
- after ``C''`` is added, the queue looks like: ``F'``, ``E'``, ``C''``,
  ``B'``, ``A'``

These callback functions are basically a reflection of the C stack at the
point the micro_thread is deferred.


Reactor Design
--------------

The Reactor design is divided into two levels:

- The top level `reactor function`_.  There is only one long running
  invocation of this function per standard Posix thread_.
- A list of Notifiers_.  Each of these knows how to check for a different
  type of external event, such as a file being ready for IO, a signal
  having been received, or a GUI/windows event.

.. _Notifiers: Notifier_


Reactor Function
''''''''''''''''

There is a reactor function instance for each Posix thread.  All instances
share the same set of ``NotifierList``, ``TimedWaitSeconds`` and
``EventCheckingThreshold`` parameters.

The master ``NotifierList`` is a list of classes that are instantiated when
the reactor function is created.  This list is maintained in descending
``PyNotifier_Priority`` order.

The reactor function pops (deferred, returned_object) pairs, doing
``PyDeferred_Callback`` on each, until either the ``EventCheckingThreshold``
number of deferreds have been popped, or there are no more deferreds 
scheduled.

It then runs its copy of the ``NotifierList`` to give each notifier_ a 
chance
to poll for its events.  If there are then still no deferreds scheduled, it
goes to each notifier in turn asking it to do a ``PyNotifier_TimedWait`` for
``TimedWaitSeconds`` until one returns 1.  Then it polls the
remaining notifiers again and goes back to running scheduled deferreds.

If there is only one notifier, a ``PyNotifier_WaitForever`` is used, rather
than first polling with ``PyNotifier_Poll`` and then 
``PyNotifier_TimedWait``.

If all but one notifier returns -1 on the initial poll pass (such that only
one notifier has any deferreds), a ``PyNotifier_WaitForever`` is used on 
that
notifier on the second pass rather than ``PyNotifier_TimedWait``.

If all notifiers return -1 on the initial poll pass and there are no 
deferreds
scheduled, the reactor function is done and returns to terminate its Posix
thread.

The reactor function also manages a list of timers for the notifiers.  It
calls ``PyNotifier_Timeout`` each time a timer pops.

The following functions use the reactor function for the current Posix 
thread.

- ``int PyReactor_Schedule(PyObject *deferred, PyObject *returned_object)``

  - Returns 0 on error, 1 otherwise.

- ``int PyReactor_ScheduleException(PyObject *deferred,
  PyObject *exc_type, PyObject *exc_value, PyObject *exc_traceback)``

  - Returns 0 on error, 1 otherwise.

- ``int PyReactor_Run(void)``

  - At least one ``PyReactor_Schedule`` must be done first, or
    ``PyReactor_Run`` will return immediately.
  - This only returns when there is nothing left to do.
  - Returns 0 on error, 1 otherwise.

- ``int PyReactor_SetTimer(PyObject *notifier, PyObject *deferred,
  double seconds)``

  - Returns 0 on error, 1 otherwise.

- ``int PyReactor_ClearTimer(PyObject *notifier, PyObject *deferred)``

  - Returns 0 on error, 1 otherwise.

These functions apply globally to all reactor functions (all Posix threads):

- ``int PyReactor_AddNotifier(PyObject *notifier_class)``

  - The *notifier_class* is added to the NotifierList in proper priority 
order.
  - The same NotifierList is used by all reactor functions (all Posix 
threads).
  - Returns 0 on error, 1 otherwise.

- ``int PyReactor_RemoveNotifier(PyObject *notifier_class)``

  - The *notifier_class* is removed from the NotifierList.
  - Returns 0 on error, 1 otherwise.  It is an error if the *notifier_class*
    was not in the NotifierList.

- ``int PyReactor_SetEventCheckingThreshold(long num_continues)``

  - Returns 0 on error, 1 otherwise.

- ``int PyReactor_SetTimedWaitSeconds(double seconds)``

  - Returns 0 on error, 1 otherwise.


Notifier
''''''''

Each notifier knows how to check for a different kind of event.  The 
notifiers
must release the GIL lock prior to suspending the Posix thread.

- ``int PyNotifier_Priority(PyObject *notifier_class)``
 
  - Returns the priority of this *notifier_class* (-1 for error).  Higher
    numbers have higher priorities.

- ``int PyNotifier_RegisterDeferred(PyObject *notifier, PyObject *deferred,
  PyObject *wait_reason, double max_wait_seconds)``

  - *Max_wait_seconds* of 0.0 means no time limit.  Otherwise, register
    *deferred* with ``PyReactor_SetTimer`` (above).
  - Adds *deferred* to the list of waiting objects, for *wait_reason*.
  - The meaning of *wait_reason* is determined by the notifier.  It can be
    used, for example, to indicate whether to wait for input or output on a
    file.
  - Returns 0 on error, 1 otherwise.

- ``void PyNotifier_Defer(PyObject *notifier, PyObject *wait_reason,
  double max_wait_seconds)``

  - Passes the deferred of the current micro_thread to
    ``PyNotifier_RegisterDeferred``, and then raises the deferred as an
    exception.  *Wait_reason* and *max_wait_seconds* are passed on to
    ``PyNotifier_RegisterDeferred``.
  - This function has no return value.  It always generates an exception.

- ``int PyNotifier_Poll(PyObject *notifier)``

  - Poll for events and schedule the appropriate ``PyDeferred_CDeferreds``.
    Do not cause the process to be put to sleep.  Return -1 if no deferreds
    are waiting for this events, 0 on error, 1 on success (whether or 
not any
    events were discovered).

- ``int PyNotifier_TimedWait(PyObject *notifier, double seconds)``

  - Wait for events and schedule the appropriate deferreds.  Do not 
cause the
    Posix thread to be put to sleep for more than the indicated number
    of *seconds*.  Return -2 if *notifier* is not capable of doing timed
    sleeps, -1 if no deferreds are waiting for events, 0 on error, 1 on
    success (whether or not any events were discovered).  Return a 1 if
    the wait was terminated due to the process having received a signal.
  - If *notifier* is not capable of doing timed waits, it should still do a
    poll and should still return -1 if no deferreds are waiting for events.

- ``int PyNotifier_WaitForever(PyObject *notifier)``

  - Suspend the process until an event occurs and schedule the appropriate
    deferreds.  The process may be put to sleep indefinitely.
    Return -1 if no deferreds are waiting for events, 0 on error, 1 on 
success
    (whether or not any ``PyDeferred_CDeferreds`` were scheduled).
    Return a 1 if the wait was terminated due to the process having received
    a signal.

- ``int PyNotifier_Timeout(PyObject *notifier, PyObject *deferred)``
 
  - Called by `reactor function`_ when the timer set by 
``PyReactor_SetTimer``
    expires.
  - Deregisters *deferred*.
  - Passes a ``TimeoutException`` to *deferred* using 
``PyDeferred_Callback``.
  - Return 0 on error, 1 otherwise.

- ``int PyNotifier_DeregisterDeferred(PyObject *notifier, PyObject 
*deferred,
  PyObject *returned_object)``

  - Deregisters *deferred*.
  - Passes *returned_object* to *deferred* using ``PyDeferred_Callback``.
  - *Returned_object* may be ``NULL`` to indicate an exception to the
    callbacks.
  - Returns 0 on error, 1 otherwise.


Open Questions
==============

#. How are tracebacks handled?
#. Do we:

   #. Treat each Python-to-Python call as a separate C call, with it's own
      callback_fn?
   #. Only register one callback_fn for each continuous string of
      Python-to-Python calls and then process them iteratively rather than
      recursively in the callback_fn (but not in the original calls)?  or
   #. Treat Python-to-Python calls iteratively both in the original calls
      and in the callback_fn?

#. How is process termination handled?
  
   - I guess we can keep a list of micro_threads and terminate each of them.
     There's a question of whether to allow the micro_threads to complete or
     to abort them mid-stream.  Kind of like a unix shutdown.  Maybe two 
kinds
     of process termination?

#. How does this impact the debugger/profiler/sys.settrace?
#. Should functions (C and Python) that may defer be indicated with some
   naming convention (e.g., ends in '_d') to make it easier for programmers
   to avoid them within their critical sections of code (in terms of
   synchronization)?
#. Do we really need to expose micro_pipes to the Python programmer as
   anything more than iterables, or can we just use the built-in ``put`` and
   ``take_from`` functions?


Rationale
=========


Impact on Other Python Implementations
--------------------------------------

The heart of this approach, the C deferred, reactor function and notifiers,
are not exposed to the Python level.  This leaves their implementation open
so that other implementations of Python (e.g., Jython_ [#jython-project]_,
IronPython_ [#ironpy]_ and PyPy_ [#pypy_project]_) are not constrained by
the choices made for CPython.

Also, the interfaces to the new Python-level objects (micro_threads,
micro_pipes) are kept to a minimum thus hiding design decisions made within
the underlying implementation so as not to unduly constrain other Python
implementations that wish to support compatible features.


Other Approaches
----------------

Here's a brief comparison to other approaches to micro-threading in Python:

- `Stackless Python`_ [#stackless]_
 
  - As near as I can tell, stackless went through two incarnations:
   
    #. The first incarnation involved an implementation of Frame 
continuations
       which were then used to provide the rest of the stackless 
functionality.
      
       - A new ``Py_UnwindToken`` was created to unwind the stack.  This is
         similar to the new ``PyDeferred_CDeferred`` proposed in this PEP,
         except that ``Py_UnwindToken`` is treated as a special case of a
         normal ``PyObject`` return value, while the 
``PyDeferred_CDeferred``
         is treated as a special case of a normal exception.

         It's not clear whether C functions are exposed to this special 
value.
         So either C functions can't be unwound, or unmodified C 
functions may
         behave strangely.  There is mention of trouble if a C function 
calls
         a Python function.  I also saw no mention of being able to defer
         execution rather than block the whole program.

         This PEP treats the requests to defer as special exceptions, which
         are already designed to unwind the C stack.

       - Another difference between the two styles of continuations is that
         the stackless continuation is designed to be able to be continued
         multiple times.  In other words, you can continue the execution of
         the program from the point the continuation was made as many times
         as you wish, passing different seed values each time.

         The ``PyDeferred_CDeferred`` described in this PEP (like the 
Twisted
         Deferred) is designed to be continued only once.

       - The stackless approach provides a Python-level continuation
         mechanism (at the Frame level) that only makes Python functions
         continuable.  It provides no way for C functions to register
         continuations so that C functions can be unwound from the stack
         and later continued (other than those related to the byte code
         interpreter).

         In contrast, this PEP proposes a C-level continuation mechanism
         very similar to the Twisted Deferred.  Each C function registers a
         callback to be run when the deferred is continued.  From this
         perspective, the byte code interpreter is just another C function.

    #. The second incarnation involved a way of hacking the underlying C
       stack to copy it and later restore it as a means of continuing the
       execution.

       - This doesn't appear to be portable to different CPU/C Compiler
         configurations.
       - This doesn't deal with other global state (global/static variables,
         file pointers, etc) that may also be used by this saved stack.
       - In contrast, this PEP uses a single C stack and makes no 
assumptions
         about the underlying C stack implementation.  It is completely
         portable to any CPU/C compiler configuration.

- `py.magic.greenlet: Lightweight concurrent programming`_ [#greenlets]_

  This takes its implementation from the second incarnation of stackless and
  copies the C stack for re-use.  It has the same portability questions that
  the second generation of stackless does.

  It does not include a reactor component, though one could be written 
for it.

- `Implementing "weightless threads" with Python generators`_ [#weightless]_

  - This requires you code each thread as generators.  The generator
    executes a ``yield`` to relinquish control.
  - It's not clear how this scales.  It seems that to pause in a lower
    Python function, it and all intermediate functions must be generators.

- python-safethread_ [#safethread]_

  - This is an alternate implementation to thread_ that adds monitors to
    mutable types, deadlock detection, improves exception propagation
    across threads and program finalization, and removes the GIL lock.  As
    such, it is not a "micro" threading approach, though by removing the GIL
    lock it may be able to better utilize multiple processor configurations
    than the approach proposed in this PEP.

- `Sandboxed Threads in Python`_ [#sandboxed-threads]_

  - Another alternate implementation to thread_, this one only shares
    immutable objects between threads, modifying the referencing counting
    system to avoid synchronization issues with the reference count for
    shared objects.  Again, not a "micro" threading approach, but perhaps
    also better with multiple processors.

.. _Jython: http://www.jython.org/Project/
.. _IronPython: 
http://www.codeplex.com/Wiki/View.aspx?ProjectName=IronPython
.. _PyPy: http://codespeak.net/pypy/dist/pypy/doc/home.html
.. _Implementing "weightless threads" with Python generators:
   http://www.ibm.com/developerworks/library/l-pythrd.html
.. _python-safethread: https://launchpad.net/python-safethread
.. _Sandboxed Threads in Python:
   http://mail.python.org/pipermail/python-dev/2005-October/057082.html
.. _Stackless Python: http://www.stackless.com/
.. _thread: http://docs.python.org/lib/module-thread.html
.. _threading: http://docs.python.org/lib/module-threading.html
.. _`py.magic.greenlet: Lightweight concurrent programming`:
   http://codespeak.net/py/dist/greenlet.html


Backwards Compatibility
=======================

This PEP doesn't break any existing code.  Existing code just won't take
advantage of any of the new features.

But there are two possible problem areas:

#. Python code uses micro-threading, but then causes an unmodified C 
function
   to call a modified C function which tries to defer execution.

   In this case an exception will be generated stating that this C function
   needs to be converted before the program will work.

#. Python code originally written in a single threaded environment is 
now used
   in a micro-threaded environment.  The old code was not written taking
   synchronization issues into account, which may cause problems if the old
   code calls a function which defers in the middle of a critical section.
   This could cause very strange behavior, but can't result in any C-level
   errors (e.g., segmentation violation).

   This old code would have to be fixed to run with the new features.  I
   expect that this will not be a frequent problem as these 
interruptions can
   only occur at a few places (where functions that defer are called).


References
==========

.. [#twisted-fn] Twisted, Twisted Matrix Labs
   (http://twistedmatrix.com/trac/)
.. [#c_api] Python/C API Reference Manual, Rossum
   (http://docs.python.org/api/api.html)
.. [#stackless] Stackless Python, Tismer
   (http://www.stackless.com/)
.. [#thread-module] thread -- Multiple threads of control
   (http://docs.python.org/lib/module-thread.html)
.. [#threading-module] threading -- Higher-level threading interface
   (http://docs.python.org/lib/module-threading.html)
.. [#jython-project] The Jython Project
   (http://www.jython.org/Project/)
.. [#ironpy] IronPython
   (http://www.codeplex.com/Wiki/View.aspx?ProjectName=IronPython)
.. [#pypy_project] PyPy[home]
   (http://codespeak.net/pypy/dist/pypy/doc/home.html)
.. [#greenlets] py.magic.greenlet: Lightweight concurrent programming
   (http://codespeak.net/py/dist/greenlet.html)
.. [#weightless] Charming Python: Implementing "weightless threads" with
   Python generators, Mertz
   (http://www.ibm.com/developerworks/library/l-pythrd.html)
.. [#safethread] Threading extensions to the Python Language,
   (https://launchpad.net/python-safethread)
.. [#sandboxed-threads] Sandboxed Threads in Python, Olsen
   (http://mail.python.org/pipermail/python-dev/2005-October/057082.html)


Copyright
=========

This document has been placed in the public domain.




From g.brandl at gmx.net  Mon Aug 25 22:27:55 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 25 Aug 2008 22:27:55 +0200
Subject: [Python-ideas] Inline 'raises' expression
In-Reply-To: <B640FFB56C3E42FF848479B430CB5E36@RaymondLaptop1>
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
	<B640FFB56C3E42FF848479B430CB5E36@RaymondLaptop1>
Message-ID: <g8v4eo$a0e$2@ger.gmane.org>

Raymond Hettinger schrieb:
> From: "Fredrik Johansson" <fredrik.johansson at gmail.com>
>> one could just write
>> 
>>    assert 1/0 raises ZeroDivisionError
> 
> +1
> 
> This would be a nice extension to the assertion syntax.

It certainly looks good, and is probably implemented quickly.
However, it introduces a new pseudo-keyword, like "as" was in the past,
as we certainly don't want to make "raises" a "full" one.

Georg



From brett at python.org  Mon Aug 25 23:25:23 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 25 Aug 2008 14:25:23 -0700
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
In-Reply-To: <48B2E256.6080008@gmail.com>
References: <48B2E256.6080008@gmail.com>
Message-ID: <bbaeab100808251425p1f0f8be0x6f7aa531bf5e67e5@mail.gmail.com>

On Mon, Aug 25, 2008 at 9:48 AM, Bruce Frederiksen <dangyogi at gmail.com> wrote:
> It was suggested that I rearrange the micro-threading PEP proposal to place
> the juicy Python stuff up front.
>
> So I've done this here.  And now that I see that people are back from the
> toils of creating 3.0b3 and starting to comment again on python-ideas, it
> seems like a good time to repost this!  (I guess my first post was really
> bad timing..., sorry about that!)
>

Actually, the toiling for the core developers has shifted to the
release candidates, so I still would not count on much response out of
them until we go final and are sure there is no emergency release.

-Brett


From greg.ewing at canterbury.ac.nz  Tue Aug 26 02:54:08 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 26 Aug 2008 12:54:08 +1200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808242329n4e29cd91h89df36b142269e7d@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<bbaeab100808241653v3a811dc1jd11ca1e654157206@mail.gmail.com>
	<d03afaa20808241925q35ba797ahc9bad0ae7fd18b90@mail.gmail.com>
	<bbaeab100808242139y24e56e1awb47771b0da13a75e@mail.gmail.com>
	<d03afaa20808242204h74222910nfe9768f2b65c68e0@mail.gmail.com>
	<c7450fba0808242310o47b5afd0q5efb501a79e7c4b8@mail.gmail.com>
	<d03afaa20808242329n4e29cd91h89df36b142269e7d@mail.gmail.com>
Message-ID: <48B35430.8090403@canterbury.ac.nz>

Russ Paielli wrote:

> The notion that Python programmers are too simple minded to adapt to a 
> simple notational convention seems a bit bizarre to me.

I'd just like to point out that accusing someone of
being too simple-minded to understand one's proposal
may not be the best way of getting them to listen to it.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Aug 26 03:02:19 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 26 Aug 2008 13:02:19 +1200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
Message-ID: <48B3561B.8000608@canterbury.ac.nz>

Bruce Leban wrote:

> If I could write:
> 
>   class foo:
>     def self.bar():
>       self.rebar(self.babar)
> 
> then the call to  object.bar() would match the declaration.

I would rather reserve that syntax for a possible future
enhancement to allow the target of a def statement to
be a more general lvalue.

It's been suggested e.g. that it would be handy to be
able to create tables of functions using things like

   def funcs[5](x, y):
     ...

If that were allowed, then def a.b() would logically
have the meaning of defining a function b as an
attribute of a.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Aug 26 03:16:23 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 26 Aug 2008 13:16:23 +1200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
Message-ID: <48B35967.80206@canterbury.ac.nz>

Brandon Mintern wrote:
> The more characters available, the
> more expressive the names can be,

Also the more confusing they can be.

It's bearable in Lisp because there are very few
characters that have special meaning to the parser,
and most symbols are set off by whitespace or
parentheses.

But in Python, most punctuation is syntactically
special, and names are set off by operators at
least as often as whitespace. So allowing just a
few randomly-chosen punctuation chars as part of
names would be confusing, I think.

I find that Ruby code with ! and ? in variable
names trips up my visual parser, for example. I
see something like

   set!

and it looks like the name 'set' followed by
a '!' operator.

-- 
Greg


From russ.paielli at gmail.com  Tue Aug 26 03:26:49 2008
From: russ.paielli at gmail.com (Russ Paielli)
Date: Mon, 25 Aug 2008 18:26:49 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <48B35430.8090403@canterbury.ac.nz>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<bbaeab100808241653v3a811dc1jd11ca1e654157206@mail.gmail.com>
	<d03afaa20808241925q35ba797ahc9bad0ae7fd18b90@mail.gmail.com>
	<bbaeab100808242139y24e56e1awb47771b0da13a75e@mail.gmail.com>
	<d03afaa20808242204h74222910nfe9768f2b65c68e0@mail.gmail.com>
	<c7450fba0808242310o47b5afd0q5efb501a79e7c4b8@mail.gmail.com>
	<d03afaa20808242329n4e29cd91h89df36b142269e7d@mail.gmail.com>
	<48B35430.8090403@canterbury.ac.nz>
Message-ID: <d03afaa20808251826o655f45f4j576e05178bf95f77@mail.gmail.com>

On Mon, Aug 25, 2008 at 5:54 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>wrote:

> Russ Paielli wrote:
>
>  The notion that Python programmers are too simple minded to adapt to a
>> simple notational convention seems a bit bizarre to me.
>>
>
> I'd just like to point out that accusing someone of
> being too simple-minded to understand one's proposal
> may not be the best way of getting them to listen to it.
>

I was suggesting exactly the opposite. I was suggesting that those who are
*against* my proposal seem to think that Python programmers are too
simple-minded to handle it.

--Russ
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080825/584b4780/attachment.html>

From russ.paielli at gmail.com  Tue Aug 26 03:41:06 2008
From: russ.paielli at gmail.com (Russ Paielli)
Date: Mon, 25 Aug 2008 18:41:06 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
Message-ID: <d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>

Just for fun, I made a copy of one of my Python files with hundreds of
occurrences of "self", and I replaced them all with "S". This streamlines
things significantly, and I think it looks good, but I suppose most Python
aficionados would be aghast. I'm thinking about doing it permanently for all
my files. I normally avoid single-character names, but I think this case
could be a reasonable exception. What would you think if you saw this in
"production" code?

--Russ


On Mon, Aug 25, 2008 at 3:35 AM, Brandon Mintern <bmintern at gmail.com> wrote:

> On Mon, Aug 25, 2008 at 3:45 AM, Bruce Leban <bruce at leapyear.org> wrote:
> > Jim Jewett wrote:
> > On Sun, Aug 24, 2008 at 6:26 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> >>
> >> If m is an attribute of type(s) (which is s.__class__), this shrinkage
> is
> >> a convenient abbreviation, not a requirement.  The above calls are the
> same
> >> as
> >> type(s).m(s.a,b,z) or type(s).m(s,a,b).  Or, if you prefer,
> >> fn = type(s).m # or s.__class__.m
> >> fn(s,a,b,z)
> >> If m is an attribute s, and s in not a class, the shrinkage is not
> >> available, and one must write s.m(s,a,b,z) or s.m(s.a,b).
> >>
> >> Terry Jan Reedy
> >
> > If I could write:
> >
> >   class foo:
> >     def self.bar():
> >       self.rebar(self.babar)
> >
> > then the call to  object.bar() would match the declaration.
>
> +1. I like this proposal: it's clean and sensible, and I think it
> looks a lot more clear, especially to someone who is new to Python.
>
> > Back to Russ's proposal: it would be better accomodated IMHO by allowing
> $
> > as a character in a variable name, just like _ is. Then, conventionally,
> > people could use $ as self:
> >
> >   def $.bar():
> >     $.rebar($.babar)
> >
> > and for whatever it's worth, I find $.bar easier to read then $bar as the
> > explicit dot reminds me it's doing an attribute get rather than looking
> like
> > a special variable name.
>
> +1 I'm in agreement here as well. The closest I would get to
> supporting the $-for-self. proposal is allowing $ in variable names.
> Having used Lisp before Python, I'm used to just about any character
> being allowed in identifier names. The more characters available, the
> more expressive the names can be, and I think it could only help
> things. The overhead of people having to get used to new symbols being
> used in variable names would be small in comparison to the flexibility
> offered.
>
> Brandon
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
http://RussP.us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080825/a87659d1/attachment.html>

From phd at phd.pp.ru  Tue Aug 26 08:43:22 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Tue, 26 Aug 2008 10:43:22 +0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
Message-ID: <20080826064322.GA19811@phd.pp.ru>

On Mon, Aug 25, 2008 at 06:41:06PM -0700, Russ Paielli wrote:
> Just for fun, I made a copy of one of my Python files with hundreds of
> occurrences of "self", and I replaced them all with "S". This streamlines
> things significantly, and I think it looks good, but I suppose most Python
> aficionados would be aghast. I'm thinking about doing it permanently for all
> my files. I normally avoid single-character names, but I think this case
> could be a reasonable exception. What would you think if you saw this in
> "production" code?

   I'd be against it because it violates a well-established convention.
I consider having a One True Style (a naming convention in this particular
case) is a Good Thing. I am accustomed to that naming convention; if I see
code with a different convention I'd have problems reading and
understanding it.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.


From bronger at physik.rwth-aachen.de  Tue Aug 26 08:58:16 2008
From: bronger at physik.rwth-aachen.de (Torsten Bronger)
Date: Tue, 26 Aug 2008 08:58:16 +0200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
Message-ID: <87ljykb6uv.fsf@physik.rwth-aachen.de>

Hall?chen!

Russ Paielli writes:

> Just for fun, I made a copy of one of my Python files with
> hundreds of occurrences of "self", and I replaced them all with
> "S". This streamlines things significantly, and I think it looks
> good, but I suppose most Python aficionados would be aghast. I'm
> thinking about doing it permanently for all my files. I normally
> avoid single-character names, but I think this case could be a
> reasonable exception. What would you think if you saw this in
> "production" code?

The "self" convention goes so far that my editor highlights it --
this may be true for other editors/HTML highlighters, too.  I'd even
frown upon any programmer who breaks this convention for code that
has the slightest chance to become public.

Besides, I don't consider "self"s cluttering up the code.  YMMV, but
I don't measure code legibility by the terseness of certain
identifies at all.

Tsch?,
Torsten.

-- 
Torsten Bronger, aquisgrana, europa vetus
                   Jabber ID: torsten.bronger at jabber.rwth-aachen.de



From solipsis at pitrou.net  Tue Aug 26 14:50:00 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 26 Aug 2008 12:50:00 +0000 (UTC)
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
References: <48B2E256.6080008@gmail.com>
Message-ID: <loom.20080826T120936-583@post.gmane.org>


Hello,

I haven't read everything in detail, but a few comments.

> The advantages to the Twisted approach over Posix threads are:
> 
> #. much less memory is required per thread

Yes, which also means better CPU cache utilization and thus potentially better
scalability.

> #. faster thread creation

Certainly.

> #. faster context switching (I'm guessing on this one, is this really true?)

Depends. The CPU overhead of context switching is probably lower (although
that's not sure in the case of Twisted, since the reactor is written in pure
Python). However, in a cooperative threading model (rather than the traditional
preemptive model), latencies tend to go up unless you have a lot of possible
switching points.

> #. synchronization between threads is easier because there is no preemption,
>    making it much easier to write critical sections of code.

This is definitely the primary advantage of the Twisted approach.

> Finally, once Python has this deferred mechanism in place at the C level,
> many things become quite easy at the Python level.  This includes full
> micro-threading, micro-pipes between micro-threads, new-style generators 
> that
> can delegate responsibility for generating values to called functions 
> without
> having to intervene between their caller and the called function, parallel
> execution constructs (``parallel_map``).

Probably. However, since the core of your proposal is itself far from trivial, I
suggest you concentrate on it in this PEP; the higher-level constructs can be
deferred ( :-)) to another PEP.

> #. An addition of non_blocking modes of accessing files, sockets, time.sleep
>    and other functions that may block.  It is not clear yet exactly what 
> these
>    will look like.  The possibilities are:
> 
>    - Add an argument to the object creation functions to specify blocking or
>      non-blocking.
>    - Add an operation to change the blocking mode after the object has been
>      created.
>    - Add new non-blocking versions of the methods on the objects that may
>      block (e.g., read_d/write_d/send_d/recv_d/sleep_d).
>    - Some combination of these.

Sounds ok. FWIW, the py3k IO stack is supposed to be ready for non-blocking IO,
but this possibility is almost completely untested as of yet.

>    It may also be useful to add a locking capability to files and sockets so
>    that code (like traceback.print_exception) that outputs several lines can
>    prevent other output from being intermingled with it.

I don't think it's critical.

> #. Micro_thread objects.  Each of these will have a re-usable C deferred
>    object attached to it, since each micro_thread can only be suspended
>    waiting for one thing at a time.  The current micro_thread would be 
> stored
>    within a C global variable, much like ``_PyThreadState_Current``.

By "global", you mean "thread-local", no? That is, there is (at most) one
currently running micro-thread per OS-level thread.

>    There are three usage scenarios, aided by three different functions to
>    create micro-threads:

I suggest you fold those usage scenarios into one simple primitive that launches
a single micro-thread and provides a way to wait for its result (using a
CDeferred I suppose?). Higher-level stuff ("start_in_parallel") does not seem
critical for the usefulness of the PEP.

>       This final scenario uses *micro_pipes* to allow threads to 
> cooperatively
>       solve problems (much like unix pipes)::

What is the added value of "micro pipes" compared to, e.g., a standard Python
list or deque? Are they non-blocking?

>    - ``close()`` to cause a ``StopIteration`` on the ``__next__`` call.
>      A ``put`` done after a ``close`` silently terminates the micro_thread
>      doing the ``put`` (in case the receiving side closes the micro_pipe).

Silencing this sounds like a bad idea.

>    So each micro_thread may have a *stdout* micro_pipe assigned to them and
>    may also be assigned a *stdin* micro_pipe (some other micro_thread's 
> stdout
>    micro_pipe).

Hmm, is it really necessary? Shouldn't micro-threads just create their own pipes
when they need them? The stdin/stdout analogy is only meaningful in certain
types of workloads.

> ``PyDeferred_CDeferred`` is written as a new exception type for use by the
> C code to defer execution.  This is a subclass of ``NotImplementedError``.
> Instances are not raised as a normal exception (e.g., with
> ``PyErr_SetObject``), but by calling ``PyNotifier_Defer`` (described in the
> Notifier_ section, below).  This registers the ``PyDeferred_CDeferred``
> associated with the currently running micro_thread as the current error 
> object,

I'm not sure I understand this right. Does this mean there is a single,
pre-constructed CDeferred object for each micro-thread? If yes, then this
deviates slightly from the Twisted model where many deferreds can be created
dynamically, chained together etc.

> One peculiar thing about the stored callbacks, is that they're not really a
> queue.  When the C deferred is first used and has no saved callbacks,
> the callbacks are saved in straight FIFO manor.  Let's say that four
> callbacks are saved in this order: ``D'``, ``C'``, ``B'``, ``A'`` (meaning
> that ``A`` called ``B``, called ``C``, called ``D`` which deferred):

In this example, can you give the C pseudo-code and the equivalent Twisted
Python (pseudo-)code?

(I haven't read the Reactor part so I won't comment on it)

> #. How is process termination handled?

Raising SystemExit (or another BaseException-derived exception, e.g. ThreadExit)
in all micro-threads sounds reasonable.

> #. How does this impact the debugger/profiler/sys.settrace?

:-)


Last point: you should try to get some Twisted guys involved in the writing of
the PEP if you want it to succeed.

Regards

Antoine.




From russ.paielli at gmail.com  Wed Aug 27 02:02:39 2008
From: russ.paielli at gmail.com (Russ Paielli)
Date: Tue, 26 Aug 2008 17:02:39 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <87ljykb6uv.fsf@physik.rwth-aachen.de>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
	<87ljykb6uv.fsf@physik.rwth-aachen.de>
Message-ID: <d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>

My editor, Xemacs, also highlights the word "self." I personally find that a
bit annoying, and perhaps that is part of the reason I don't appreciate the
word. When I write an algorithm, I don't want to see some artifact
highlighted, as if it is somehow the most important part of the algorithm. I
like seeing keywords highlighted, but I don't want to see "self" highlighted
because it is only a convention. Maybe I'm alone on that.

As I wrote earlier, I copied one of my larger Python files and replaced each
occurrence of "self" with "S". That was approximately 350 occurrences. I
also replaced each occurrence of "cls" with "C", which was another 50 or so
occurrences. The file size was reduced by 2.8%, many wrapped lines no longer
need to wrap, and I like the way it looks.

I understand that people like to have "one true way," but I think Python
programmers are adaptable enough to handle this one. When you read code, you
certainly need to read the argument names anyway, so the fact that "S" is
being used instead of "self" should be obvious. The only minor disadvantage
I can think of is that searching on a single character can be problematic,
but if you are using an IDE, that should not be an issue (and I can't
remember ever searching on "self" anyway).

One of these days, when I get a chance, I may write an informational PEP
recommending the use of "S" as an acceptable replacement for "self", and "C"
as an accptable replacement for "cls". I hope that doesn't blow any fuses.

Have a nice day.

--Russ

On Mon, Aug 25, 2008 at 11:58 PM, Torsten Bronger <
bronger at physik.rwth-aachen.de> wrote:

> Hall?chen!
>
> Russ Paielli writes:
>
> > Just for fun, I made a copy of one of my Python files with
> > hundreds of occurrences of "self", and I replaced them all with
> > "S". This streamlines things significantly, and I think it looks
> > good, but I suppose most Python aficionados would be aghast. I'm
> > thinking about doing it permanently for all my files. I normally
> > avoid single-character names, but I think this case could be a
> > reasonable exception. What would you think if you saw this in
> > "production" code?
>
> The "self" convention goes so far that my editor highlights it --
> this may be true for other editors/HTML highlighters, too.  I'd even
> frown upon any programmer who breaks this convention for code that
> has the slightest chance to become public.
>
> Besides, I don't consider "self"s cluttering up the code.  YMMV, but
> I don't measure code legibility by the terseness of certain
> identifies at all.
>
> Tsch?,
> Torsten.
>
> --
> Torsten Bronger, aquisgrana, europa vetus
>                   Jabber ID: torsten.bronger at jabber.rwth-aachen.de
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
http://RussP.us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080826/65843dc0/attachment.html>

From cesare.dimauro at a-tono.com  Wed Aug 27 09:11:04 2008
From: cesare.dimauro at a-tono.com (Cesare Di Mauro)
Date: Wed, 27 Aug 2008 09:11:04 +0200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
	<87ljykb6uv.fsf@physik.rwth-aachen.de>
	<d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
Message-ID: <op.ugi98qn403jqhe@cesareprova.org>

On 27 aug 2008 at 02:02:39, Russ Paielli <russ.paielli at gmail.com> wrote:

> My editor, Xemacs, also highlights the word "self." I personally find that a
> bit annoying, and perhaps that is part of the reason I don't appreciate the
> word. When I write an algorithm, I don't want to see some artifact
> highlighted, as if it is somehow the most important part of the algorithm. I
> like seeing keywords highlighted, but I don't want to see "self" highlighted
> because it is only a convention. Maybe I'm alone on that.
>
> As I wrote earlier, I copied one of my larger Python files and replaced each
> occurrence of "self" with "S". That was approximately 350 occurrences. I
> also replaced each occurrence of "cls" with "C", which was another 50 or so
> occurrences. The file size was reduced by 2.8%, many wrapped lines no longer
> need to wrap, and I like the way it looks.
>
> I understand that people like to have "one true way," but I think Python
> programmers are adaptable enough to handle this one. When you read code, you
> certainly need to read the argument names anyway, so the fact that "S" is
> being used instead of "self" should be obvious. The only minor disadvantage
> I can think of is that searching on a single character can be problematic,
> but if you are using an IDE, that should not be an issue (and I can't
> remember ever searching on "self" anyway).
>
> One of these days, when I get a chance, I may write an informational PEP
> recommending the use of "S" as an acceptable replacement for "self", and "C"
> as an accptable replacement for "cls". I hope that doesn't blow any fuses.
>
> Have a nice day.
>
> --Russ

I absolutely disagree. It's the same thing that happens when we use one
character identifiers: the code becomes less readable.

Identifiers must be self-explicative.

Cheers,
Cesare


From gnewsg at gmail.com  Wed Aug 27 14:50:54 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Wed, 27 Aug 2008 05:50:54 -0700 (PDT)
Subject: [Python-ideas] Inline 'raises' expression
In-Reply-To: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
Message-ID: <6eef22d2-d254-47b5-8a3d-f3c8b487d174@8g2000hse.googlegroups.com>

On 24 Ago, 12:43, "Fredrik Johansson" <fredrik.johans... at gmail.com>
wrote:
> Hi,
>
> It happens that I'm just interested in whether an expression raises an
> exception, not the return value. This might look something like
>
> ? ? try:
> ? ? ? ? check_status()
> ? ? except IOError:
> ? ? ? ? cleanup()
>
> which could be written more simply as
>
> ? ? if check_status() raises IOError:
> ? ? ? ? cleanup()
>
> Also, instead of the inconvenient
>
> ? ? self.assertRaises(ZeroDivisionError, lambda: 1/0)
>
> one could just write
>
> ? ? assert 1/0 raises ZeroDivisionError
>
> Something like this would especially be useful for those of us who
> aren't fans of the unittest framework. Alternatively, just the
> assert--raises form could be permitted.
>
> Thoughts?
>
> Fredrik
> _______________________________________________
> Python-ideas mailing list
> Python-id... at python.orghttp://mail.python.org/mailman/listinfo/python-ideas

+1, I really like it.


--- Giampaolo
http://code.google.com/p/pyftpdlib/


From bjourne at gmail.com  Wed Aug 27 15:52:49 2008
From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=)
Date: Wed, 27 Aug 2008 15:52:49 +0200
Subject: [Python-ideas] Inline 'raises' expression
In-Reply-To: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
Message-ID: <740c3aec0808270652k793ed3e0kab4cec75a70146e8@mail.gmail.com>

2008/8/24 Fredrik Johansson <fredrik.johansson at gmail.com>:
> Hi,
>
> It happens that I'm just interested in whether an expression raises an
> exception, not the return value. This might look something like
>
>    try:
>        check_status()
>    except IOError:
>        cleanup()

Seems like a bad example to me. Wouldn't you rather write:

try:
    check_status()
finally:
    cleanup()

??


-- 
mvh Bj?rn


From dangyogi at gmail.com  Wed Aug 27 17:50:22 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Wed, 27 Aug 2008 11:50:22 -0400
Subject: [Python-ideas] micro-threading: C level in python pseudo code
Message-ID: <7d702edf0808270850s19edebf2rf892aaaa17a921fb@mail.gmail.com>

I've written up the C level code for this PEP in a python pseudo code
to hopefully make it more clear.  I have left the explanations out to
keep this shorter.  Refer to the PEP for the explanations.

I will also post the python level code for the PEP as a separate post.

-bruce

------------- cut here -------------
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: C_level.txt
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/cad2d72d/attachment.txt>

From dangyogi at gmail.com  Wed Aug 27 17:55:28 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Wed, 27 Aug 2008 11:55:28 -0400
Subject: [Python-ideas] micro-threading: Python level pseudo code
Message-ID: <7d702edf0808270855i1e131f08sf624ab548b3c4dd@mail.gmail.com>

Here is the Python level pseudo code (for the micro_threads themselves
and the micro_pipes).

-bruce

On Mon, Aug 25, 2008 at 12:48 PM, Bruce Frederiksen <dangyogi at gmail.com> wrote:
> [...]
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Python_level.txt
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/8beca750/attachment.txt>

From ziade.tarek at gmail.com  Wed Aug 27 18:51:41 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 27 Aug 2008 18:51:41 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
Message-ID: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>

Hello

There's a pattern I am doing all the time: filtering out some elements of a
list, and cleaning them in the same move.

For example, if I have a multi line text, where I want to:

- keep non empty lines
- clean non empty lines

I am doing:

    >>> text = """
    ... this is a multi-line text\t
    ...
    ... \t\twith
    ...
    ... muliple lines."""

    >>> [l.strip() for l in text.split('\n') if l.strip() != '']
    ['this is a multi-line text', 'with', 'muliple lines.']

It is not optimal, because I call strip() twice. I could use ifilter then
imap or even use a real loop, but I
want my simple, concise, list comprehension !  And I couldn't find a simple
way to express it.

The pattern can be generically resumed like this :

   [transform(e) for e in seq if some_test(transform(e))]

So what about using the 'as' keyword to extend lists comprehensions, and
to avoid calling transform() twice ?

Could be:

   [transform(e) as transformed for e in seq if some_test(transformed)]

In my use case I would simply have to write;:

  [l.strip() as stripped for l in text.split('\n') if stripped != '']

Which seems to me clear and concise.

Regards,
Tarek

-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/97fbd3b4/attachment.html>

From lorgandon at gmail.com  Wed Aug 27 19:00:32 2008
From: lorgandon at gmail.com (Imri Goldberg)
Date: Wed, 27 Aug 2008 20:00:32 +0300
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
Message-ID: <fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>

+1 from me, I find myself doing that all the time as well.

On Wed, Aug 27, 2008 at 7:51 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> Hello
>
> There's a pattern I am doing all the time: filtering out some elements of a
> list, and cleaning them in the same move.
>
> For example, if I have a multi line text, where I want to:
>
> - keep non empty lines
> - clean non empty lines
>
> I am doing:
>
>     >>> text = """
>     ... this is a multi-line text\t
>     ...
>     ... \t\twith
>     ...
>     ... muliple lines."""
>
>     >>> [l.strip() for l in text.split('\n') if l.strip() != '']
>     ['this is a multi-line text', 'with', 'muliple lines.']
>
> It is not optimal, because I call strip() twice. I could use ifilter then
> imap or even use a real loop, but I
> want my simple, concise, list comprehension !  And I couldn't find a simple
> way to express it.
>
> The pattern can be generically resumed like this :
>
>    [transform(e) for e in seq if some_test(transform(e))]
>
> So what about using the 'as' keyword to extend lists comprehensions, and
> to avoid calling transform() twice ?
>
> Could be:
>
>    [transform(e) as transformed for e in seq if some_test(transformed)]
>
> In my use case I would simply have to write;:
>
>   [l.strip() as stripped for l in text.split('\n') if stripped != '']
>
> Which seems to me clear and concise.
>
> Regards,
> Tarek
>
> --
> Tarek Ziad? | Association AfPy | www.afpy.org
> Blog FR | http://programmation-python.org
> Blog EN | http://tarekziade.wordpress.com/
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


-- 
Imri Goldberg
--------------------------------------
www.algorithm.co.il/blogs/
www.imri.co.il
--------------------------------------
-- insert signature here ----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/2a4173f6/attachment.html>

From russ.paielli at gmail.com  Wed Aug 27 19:21:54 2008
From: russ.paielli at gmail.com (Russ Paielli)
Date: Wed, 27 Aug 2008 10:21:54 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <op.ugi98qn403jqhe@cesareprova.org>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
	<87ljykb6uv.fsf@physik.rwth-aachen.de>
	<d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
	<op.ugi98qn403jqhe@cesareprova.org>
Message-ID: <d03afaa20808271021t265c5aeeo5e9b2e0b649c40bc@mail.gmail.com>

Normally I agree with you on the single-character identifiers, but I
consider this case an exception. Lets not forget that C++ and Java use
zero-character identifiers for "self", and I don't hear a din of complaints
that it harms readability.

My code does not appear to me to be any less readable to me when I
substitute "S" for "self". The fact that it is a capital letter helps a bit
here, I think. The fact that it usually appears as "S.attr" rather than just
"S" also makes it stand out clearly.

I just don't see the problem with having an "approved" shorthand form for
self when it does not require any changes in the language itself.

--Russ


On Wed, Aug 27, 2008 at 12:11 AM, Cesare Di Mauro <cesare.dimauro at a-tono.com
> wrote:

> On 27 aug 2008 at 02:02:39, Russ Paielli <russ.paielli at gmail.com> wrote:
>
> > My editor, Xemacs, also highlights the word "self." I personally find
> that a
> > bit annoying, and perhaps that is part of the reason I don't appreciate
> the
> > word. When I write an algorithm, I don't want to see some artifact
> > highlighted, as if it is somehow the most important part of the
> algorithm. I
> > like seeing keywords highlighted, but I don't want to see "self"
> highlighted
> > because it is only a convention. Maybe I'm alone on that.
> >
> > As I wrote earlier, I copied one of my larger Python files and replaced
> each
> > occurrence of "self" with "S". That was approximately 350 occurrences. I
> > also replaced each occurrence of "cls" with "C", which was another 50 or
> so
> > occurrences. The file size was reduced by 2.8%, many wrapped lines no
> longer
> > need to wrap, and I like the way it looks.
> >
> > I understand that people like to have "one true way," but I think Python
> > programmers are adaptable enough to handle this one. When you read code,
> you
> > certainly need to read the argument names anyway, so the fact that "S" is
> > being used instead of "self" should be obvious. The only minor
> disadvantage
> > I can think of is that searching on a single character can be
> problematic,
> > but if you are using an IDE, that should not be an issue (and I can't
> > remember ever searching on "self" anyway).
> >
> > One of these days, when I get a chance, I may write an informational PEP
> > recommending the use of "S" as an acceptable replacement for "self", and
> "C"
> > as an accptable replacement for "cls". I hope that doesn't blow any
> fuses.
> >
> > Have a nice day.
> >
> > --Russ
>
> I absolutely disagree. It's the same thing that happens when we use one
> character identifiers: the code becomes less readable.
>
> Identifiers must be self-explicative.
>
> Cheers,
> Cesare
>



-- 
http://RussP.us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/73c6723a/attachment.html>

From bruce at leapyear.org  Wed Aug 27 19:30:29 2008
From: bruce at leapyear.org (Bruce Leban)
Date: Wed, 27 Aug 2008 10:30:29 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>
Message-ID: <cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>

[t for t in [t.strip() for t in text.split('\n')] if t != '']


On Wed, Aug 27, 2008 at 10:00 AM, Imri Goldberg <lorgandon at gmail.com> wrote:

> +1 from me, I find myself doing that all the time as well.
>
> On Wed, Aug 27, 2008 at 7:51 PM, Tarek Ziad? <ziade.tarek at gmail.com>wrote:
>
>> Hello
>>
>> There's a pattern I am doing all the time: filtering out some elements of
>> a list, and cleaning them in the same move.
>>
>> For example, if I have a multi line text, where I want to:
>>
>> - keep non empty lines
>> - clean non empty lines
>>
>> I am doing:
>>
>>     >>> text = """
>>     ... this is a multi-line text\t
>>     ...
>>     ... \t\twith
>>     ...
>>     ... muliple lines."""
>>
>>     >>> [l.strip() for l in text.split('\n') if l.strip() != '']
>>     ['this is a multi-line text', 'with', 'muliple lines.']
>>
>> It is not optimal, because I call strip() twice. I could use ifilter then
>> imap or even use a real loop, but I
>> want my simple, concise, list comprehension !  And I couldn't find a
>> simple way to express it.
>>
>> The pattern can be generically resumed like this :
>>
>>    [transform(e) for e in seq if some_test(transform(e))]
>>
>> So what about using the 'as' keyword to extend lists comprehensions, and
>> to avoid calling transform() twice ?
>>
>> Could be:
>>
>>    [transform(e) as transformed for e in seq if some_test(transformed)]
>>
>> In my use case I would simply have to write;:
>>
>>   [l.strip() as stripped for l in text.split('\n') if stripped != '']
>>
>> Which seems to me clear and concise.
>>
>> Regards,
>> Tarek
>>
>> --
>> Tarek Ziad? | Association AfPy | www.afpy.org
>> Blog FR | http://programmation-python.org
>> Blog EN | http://tarekziade.wordpress.com/
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>>
>
>
> --
> Imri Goldberg
> --------------------------------------
> www.algorithm.co.il/blogs/
> www.imri.co.il
> --------------------------------------
> -- insert signature here ----
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/8d4e2327/attachment.html>

From ziade.tarek at gmail.com  Wed Aug 27 19:41:17 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 27 Aug 2008 19:41:17 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>
	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
Message-ID: <94bdd2610808271041m6db798f8uc4b6df48305e6cad@mail.gmail.com>

On Wed, Aug 27, 2008 at 7:30 PM, Bruce Leban <bruce at leapyear.org> wrote:

> [t for t in [t.strip() for t in text.split('\n')] if t != '']
>

Yes, and other ways could be found with imap/map, but I would find this
simpler and more natural :

[t.strip() as s for t in text.split('\n') if s != '']

Regards
Tarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/e0108aea/attachment.html>

From george.sakkis at gmail.com  Wed Aug 27 20:05:01 2008
From: george.sakkis at gmail.com (George Sakkis)
Date: Wed, 27 Aug 2008 14:05:01 -0400
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
Message-ID: <91ad5bf80808271105w1d2ec988p33a9f4d3071411d8@mail.gmail.com>

On Wed, Aug 27, 2008 at 12:51 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> Hello
>
> There's a pattern I am doing all the time: filtering out some elements of a
> list, and cleaning them in the same move.
>
> For example, if I have a multi line text, where I want to:
>
> - keep non empty lines
> - clean non empty lines
>
> I am doing:
>
>     >>> text = """
>     ... this is a multi-line text\t
>     ...
>     ... \t\twith
>     ...
>     ... muliple lines."""
>
>     >>> [l.strip() for l in text.split('\n') if l.strip() != '']
>     ['this is a multi-line text', 'with', 'muliple lines.']
>
> It is not optimal, because I call strip() twice. I could use ifilter then
> imap or even use a real loop, but I
> want my simple, concise, list comprehension !  And I couldn't find a simple
> way to express it.
>
> The pattern can be generically resumed like this :
>
>    [transform(e) for e in seq if some_test(transform(e))]
>
> So what about using the 'as' keyword to extend lists comprehensions, and
> to avoid calling transform() twice ?
>
> Could be:
>
>    [transform(e) as transformed for e in seq if some_test(transformed)]
>

-1; not general enough to justify the extra overhead (both in human mental
effort and compiler changes), given that the current alternatives (esp.
genexps) are already quite readable and more flexible.

George
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/26c85c27/attachment.html>

From dangyogi at gmail.com  Wed Aug 27 20:12:38 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Wed, 27 Aug 2008 14:12:38 -0400
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
In-Reply-To: <loom.20080826T120936-583@post.gmane.org>
References: <48B2E256.6080008@gmail.com>
	<loom.20080826T120936-583@post.gmane.org>
Message-ID: <48B59916.1000803@gmail.com>

Thank you for your response.  I have written up a python-style pseudo 
code for both the C level code and the Python level code and posted them 
as two separate posts.  I changed the subject somewhat on them, so they 
don't show up on the same thread...  :-(

Here is some specific feedback on your questions.

Antoine Pitrou wrote:
> Probably. However, since the core of your proposal is itself far from trivial, I
> suggest you concentrate on it in this PEP; the higher-level constructs can be
> deferred ( :-)) to another PEP.
>   
I had considered that.  But the core by itself accomplishes nothing, 
except to serve as a foundation for some kind of higher-level 
constructs, so I put them together.  I guess having separate PEPs allows 
them to evolve more independently.  (I'm new to this PEP process).

If I split them, so I keep posting updated versions on python-ideas?  Or 
do I just accumulate the changes offline and post the completed PEP much 
later?
>   
>> #. An addition of non_blocking modes of accessing files, sockets, time.sleep
>>    and other functions that may block.  It is not clear yet exactly what 
>> these
>>    will look like.  The possibilities are:
>>
>>    - Add an argument to the object creation functions to specify blocking or
>>      non-blocking.
>>    - Add an operation to change the blocking mode after the object has been
>>      created.
>>    - Add new non-blocking versions of the methods on the objects that may
>>      block (e.g., read_d/write_d/send_d/recv_d/sleep_d).
>>    - Some combination of these.
>>     
>
> Sounds ok. FWIW, the py3k IO stack is supposed to be ready for non-blocking IO,
> but this possibility is almost completely untested as of yet.
>   
Good to know.  I'll have to look at this.
>   
>> #. Micro_thread objects.  Each of these will have a re-usable C deferred
>>    object attached to it, since each micro_thread can only be suspended
>>    waiting for one thing at a time.  The current micro_thread would be 
>> stored
>>    within a C global variable, much like ``_PyThreadState_Current``.
>>     
>
> By "global", you mean "thread-local", no? That is, there is (at most) one
> currently running micro-thread per OS-level thread.
>   
Yes!
>   
>>    There are three usage scenarios, aided by three different functions to
>>    create micro-threads:
>>     
>
> I suggest you fold those usage scenarios into one simple primitive that launches
> a single micro-thread and provides a way to wait for its result (using a
> CDeferred I suppose?). Higher-level stuff ("start_in_parallel") does not seem
> critical for the usefulness of the PEP.
>   
I have a single micro_thread class with a couple of optional arguments 
that affects it operation, so you may be right.  I have included the 
higher-level stuff ("start_in_parallel") in the Python level pseudo code 
to give everybody a feel of what's involved.
>   
>>       This final scenario uses *micro_pipes* to allow threads to 
>> cooperatively
>>       solve problems (much like unix pipes)::
>>     
>
> What is the added value of "micro pipes" compared to, e.g., a standard Python
> list or deque? Are they non-blocking?
>   
Micro_pipes connect two micro_threads, much like unix pipes join two 
unix processes.  Each thread will suspend if the other thread isn't 
ready.  The micro_pipes use the C_deferreds to suspend the thread and 
allow other threads to run.

So micro_pipes don't store a sequence of values (like lists or deques), 
but pass individual values on from one thread to another.  The 
implementation proposed in the Python level pseudo code only stores one 
value and will block the writer when it tries to write a second value 
before the reader has read the first value.  This buffer size of one 
could be expanded, but I've been working on the premise that this should 
be kept as simple as possible for a first out, and then allowed to grow 
after more experience is gained with it.  I've seen many software 
projects (and I'm guilty of this myself) where they include all kinds of 
stuff that really isn't that useful.  And, once released, these things 
are hard to take back.  So I'm consciously trying to keep the first out 
to a bare minimum of features that can grow later.
>   
>>    - ``close()`` to cause a ``StopIteration`` on the ``__next__`` call.
>>      A ``put`` done after a ``close`` silently terminates the micro_thread
>>      doing the ``put`` (in case the receiving side closes the micro_pipe).
>>     
>
> Silencing this sounds like a bad idea.
>   
Yes, I think "silently" means raising a MicroThreadExit exception in the 
``put`` and then silently ignoring it when it is finally re-raised by 
the top function of the thread (thus, terminating the thread, but 
allowing clean code to run on the way down).
>   
>>    So each micro_thread may have a *stdout* micro_pipe assigned to them and
>>    may also be assigned a *stdin* micro_pipe (some other micro_thread's 
>> stdout
>>    micro_pipe).
>>     
>
> Hmm, is it really necessary? Shouldn't micro-threads just create their own pipes
> when they need them? The stdin/stdout analogy is only meaningful in certain
> types of workloads.
>   
They seem necessary to handle exception situations.  For example, when a 
reader thread on a pipe dies with an exception, how is the write thread 
notified?  What mechanism knows that this pipe was being read by the 
errant thread so that it will never be read from again?  Lacking some 
kind of mechanism like this may mean that the writer thread is suspended 
forever.  And the same applies in reverse if the writer thread dies.  
The reader is left hanging forever.  So the pipes need to be "attached" 
to the threads so that an exception in one thread can also affect other 
interested threads.
>   
>> ``PyDeferred_CDeferred`` is written as a new exception type for use by the
>> C code to defer execution.  This is a subclass of ``NotImplementedError``.
>> Instances are not raised as a normal exception (e.g., with
>> ``PyErr_SetObject``), but by calling ``PyNotifier_Defer`` (described in the
>> Notifier_ section, below).  This registers the ``PyDeferred_CDeferred``
>> associated with the currently running micro_thread as the current error 
>> object,
>>     
>
> I'm not sure I understand this right. Does this mean there is a single,
> pre-constructed CDeferred object for each micro-thread? If yes, then this
> deviates slightly from the Twisted model where many deferreds can be created
> dynamically, chained together etc.
>   
Yes, a single deferred for each micro-thread.  And yes, this differs 
some from the Twisted model.  But, again, this helps to "connect the 
dots" between threads for exception propagation.  I think that it will 
also give slightly better performance because fewer memory allocations 
are required.
>   
>> One peculiar thing about the stored callbacks, is that they're not really a
>> queue.  When the C deferred is first used and has no saved callbacks,
>> the callbacks are saved in straight FIFO manor.  Let's say that four
>> callbacks are saved in this order: ``D'``, ``C'``, ``B'``, ``A'`` (meaning
>> that ``A`` called ``B``, called ``C``, called ``D`` which deferred):
>>     
>
> In this example, can you give the C pseudo-code and the equivalent Twisted
> Python (pseudo-)code?
>   
Do you mean the pseudo code of the deferred implementation, or the 
pseudo code for using the deferreds?
>
> Last point: you should try to get some Twisted guys involved in the writing of
> the PEP if you want it to succeed.
>   
Good suggestion!  I was hoping that some might show up here, but ...  I 
guess I need to go looking for them!

Thanks!

-bruce


From bwinton at latte.ca  Wed Aug 27 20:12:10 2008
From: bwinton at latte.ca (Blake Winton)
Date: Wed, 27 Aug 2008 14:12:10 -0400
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <91ad5bf80808271105w1d2ec988p33a9f4d3071411d8@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<91ad5bf80808271105w1d2ec988p33a9f4d3071411d8@mail.gmail.com>
Message-ID: <48B598FA.8070307@latte.ca>

George Sakkis wrote:
> 
> On Wed, Aug 27, 2008 at 12:51 PM, Tarek Ziad? <ziade.tarek at gmail.com 
> <mailto:ziade.tarek at gmail.com>> wrote:
>     The pattern can be generically resumed like this :
>        [transform(e) for e in seq if some_test(transform(e))]
>     So what about using the 'as' keyword to extend lists comprehensions, and
>     to avoid calling transform() twice ?
>      
>     Could be:
>        [transform(e) as transformed for e in seq if some_test(transformed)]
> 
> -1; not general enough to justify the extra overhead (both in human 
> mental effort and compiler changes), given that the current alternatives 
> (esp. genexps) are already quite readable and more flexible.

-1; It's not clear to me that transform(e) is what's going into the 
list.  And what if I want to test on two different values (or two 
different transforms on the same value), which one is put into the 
comprehension?

Later,
Blake.


From brett at python.org  Wed Aug 27 20:19:43 2008
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Aug 2008 11:19:43 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
Message-ID: <bbaeab100808271119p3ce847f3l5b608aa73838697a@mail.gmail.com>

On Wed, Aug 27, 2008 at 9:51 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> Hello
>
> There's a pattern I am doing all the time: filtering out some elements of a
> list, and cleaning them in the same move.
>
> For example, if I have a multi line text, where I want to:
>
> - keep non empty lines
> - clean non empty lines
>
> I am doing:
>
>     >>> text = """
>     ... this is a multi-line text\t
>     ...
>     ... \t\twith
>     ...
>     ... muliple lines."""
>
>     >>> [l.strip() for l in text.split('\n') if l.strip() != '']
>     ['this is a multi-line text', 'with', 'muliple lines.']
>
> It is not optimal, because I call strip() twice. I could use ifilter then
> imap or even use a real loop, but I
> want my simple, concise, list comprehension !  And I couldn't find a simple
> way to express it.
>
> The pattern can be generically resumed like this :
>
>    [transform(e) for e in seq if some_test(transform(e))]
>
> So what about using the 'as' keyword to extend lists comprehensions, and
> to avoid calling transform() twice ?
>

Not worth it since while you might want your "simple. concise list
comprehension", listcomps are not meant to generalize for all possible
situations in 'for' loops that can lead to a new list. Is it really
that much harder to write::

 list_ = []
 for line in text.splitlines():
   line = line.strip():
   if line:
     list_.append(line)

?

And if you really want your one-liner, you can avoid your duplicate
call to str.strip() in two different ways::

  [line.strip() for line in text.splitlines() if line and not line.isspace()]

or::

  filter(None, (line.strip() for line in text.splitlines()))

I don't see enough benefit to clutter listcomps with more syntax and
to provide another alternative to this pattern to add to the multiple
ones that already exist.

-Brett

From jimjjewett at gmail.com  Wed Aug 27 20:19:55 2008
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 27 Aug 2008 14:19:55 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <g8uk3r$9sf$1@ger.gmane.org>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<g8uk3r$9sf$1@ger.gmane.org>
Message-ID: <fb6fbf560808271119qdcdac3bh4385bf25e0b12e2e@mail.gmail.com>

On Mon, Aug 25, 2008 at 11:46 AM, Terry Reedy > 3.0 uses the standard
Unicode syntax for identifiers:
> "
> identifier  ::=  id_start id_continue*
>
> id_start    ::=  <all characters in general categories Lu, Ll, Lt, Lm, Lo,
> Nl, the underscore, and characters with the Other_ID_Start property>


Are you sure?

IIRC, the unicode consortion themselves recommend the xid_start and
xid_continue properties instead.

(One of the quirks of unicode is that it will gladly shoot itself in
the foot to support backwards compatibility, and this is one of those
cases.  Characters that really shouldn't be used for identifiers got
mixed in by mistake.  There are even more restrictive proposals for
security, but those were deemed excessive.)

-jJ


From bjourne at gmail.com  Wed Aug 27 20:21:09 2008
From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=)
Date: Wed, 27 Aug 2008 20:21:09 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
Message-ID: <740c3aec0808271121q3f1f68e4w8a8d2a4e0332f8d2@mail.gmail.com>

2008/8/27 Tarek Ziad? <ziade.tarek at gmail.com>:
>   [l.strip() as stripped for l in text.split('\n') if stripped != '']

Two linkes are better than one:

stripped = [l.strip() for l in text.split('\n')]
[l for l in stripped if l != '']

You are performing two different operations on the list after all. And
more readable.


-- 
mvh Bj?rn


From scott+python-ideas at scottdial.com  Wed Aug 27 19:54:19 2008
From: scott+python-ideas at scottdial.com (Scott Dial)
Date: Wed, 27 Aug 2008 13:54:19 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808271021t265c5aeeo5e9b2e0b649c40bc@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>	<g8t1on$7ne$1@ger.gmane.org>	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>	<87ljykb6uv.fsf@physik.rwth-aachen.de>	<d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>	<op.ugi98qn403jqhe@cesareprova.org>
	<d03afaa20808271021t265c5aeeo5e9b2e0b649c40bc@mail.gmail.com>
Message-ID: <48B594CB.3050008@scottdial.com>

Russ Paielli wrote:
> Normally I agree with you on the single-character identifiers, but I
> consider this case an exception. Lets not forget that C++ and Java use
> zero-character identifiers for "self", and I don't hear a din of
> complaints that it harms readability.

(Not) from who? I think in general Pythonistas complain that the
"zero-character identifiers" harm readability. Someone already said this
in this thread:

Andy Toulouse wrote:
> I don't like programming in languages that don't make as clear a
> distinction between local variables and instance variables.

In the rare cases that I am writing Java or C++ these days, I absolutely
use "this" explicitly despite the lack of a requirement to do so. It's
simply more readable in the long-term.

> My code does not appear to me to be any less readable to me when I
> substitute "S" for "self". The fact that it is a capital letter helps a
> bit here, I think. The fact that it usually appears as "S.attr" rather
> than just "S" also makes it stand out clearly.
> 
> I just don't see the problem with having an "approved" shorthand form
> for self when it does not require any changes in the language itself.

Feel free to write code like this for yourself. I don't see anyone
"approving" any sort of "shorthand form" ever. The only official stance
on coding conventions is intended to guide the writing of code for the
stdlib. I find it very unlikely anyone would accept "S" in place of
"self" within the stdlib. Furthermore, I think the likelihood of having
another developer working with your code will sharply go down if you
stray from the guidelines for the stdlib. But to paraphrase what someone
else in this thread already said: what you do in private is your business.

-Scott

P.S. Top-posting is inappropriate for this list and makes it more
difficult to follow long discussions (like this one).

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu


From jspicklemire at gmail.com  Wed Aug 27 22:29:57 2008
From: jspicklemire at gmail.com (Jerry Spicklemire)
Date: Wed, 27 Aug 2008 16:29:57 -0400
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
Message-ID: <4a038ea30808271329p4dc7a5fdyc3273969e0b892d2@mail.gmail.com>

Bruce wrote:

"The micro_pipes use the C_deferreds to suspend the thread
and allow other threads to run."

http://mail.python.org/pipermail/python-ideas/2008-August/001848.html


However, in the PEP you mention, among the short list of
disadvantages:

"since there is no preemption, a long running micro-thread
will starve other micro-threads"

http://mail.python.org/pipermail/python-ideas/2008-August/001825.html


Are these two point contradictory, or am I simply
misunderstanding, as usual.

Tanks,
Jerry S.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/f8a8c58f/attachment.html>

From g.brandl at gmx.net  Thu Aug 28 00:25:42 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 28 Aug 2008 00:25:42 +0200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <fb6fbf560808271119qdcdac3bh4385bf25e0b12e2e@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>	<g8t1on$7ne$1@ger.gmane.org>	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>	<g8uk3r$9sf$1@ger.gmane.org>
	<fb6fbf560808271119qdcdac3bh4385bf25e0b12e2e@mail.gmail.com>
Message-ID: <g94k22$9rf$1@ger.gmane.org>

Jim Jewett schrieb:
> On Mon, Aug 25, 2008 at 11:46 AM, Terry Reedy > 3.0 uses the standard
> Unicode syntax for identifiers:
>> "
>> identifier  ::=  id_start id_continue*
>>
>> id_start    ::=  <all characters in general categories Lu, Ll, Lt, Lm, Lo,
>> Nl, the underscore, and characters with the Other_ID_Start property>
> 
> 
> Are you sure?
> 
> IIRC, the unicode consortion themselves recommend the xid_start and
> xid_continue properties instead.

I thought we were already using them.

Georg



From g.brandl at gmx.net  Thu Aug 28 00:27:19 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 28 Aug 2008 00:27:19 +0200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808271021t265c5aeeo5e9b2e0b649c40bc@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>	<g8t1on$7ne$1@ger.gmane.org>	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>	<87ljykb6uv.fsf@physik.rwth-aachen.de>	<d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>	<op.ugi98qn403jqhe@cesareprova.org>
	<d03afaa20808271021t265c5aeeo5e9b2e0b649c40bc@mail.gmail.com>
Message-ID: <g94k54$9rf$2@ger.gmane.org>

Russ Paielli schrieb:
> Normally I agree with you on the single-character identifiers, but I
> consider this case an exception. Lets not forget that C++ and Java use
> zero-character identifiers for "self", and I don't hear a din of
> complaints that it harms readability.

Yes, because most have coding styles that mandate hungarian prefixes on
such identifiers. And "mInfo" isn't really more readable than "self.info".

Georg



From g.brandl at gmx.net  Thu Aug 28 00:28:51 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 28 Aug 2008 00:28:51 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <bbaeab100808271119p3ce847f3l5b608aa73838697a@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<bbaeab100808271119p3ce847f3l5b608aa73838697a@mail.gmail.com>
Message-ID: <g94k7v$9rf$3@ger.gmane.org>

Brett Cannon schrieb:

> And if you really want your one-liner, you can avoid your duplicate
> call to str.strip() in two different ways::
> 
>   [line.strip() for line in text.splitlines() if line and not line.isspace()]
> 
> or::
> 
>   filter(None, (line.strip() for line in text.splitlines()))

And thank god we can finally write

map(str.strip, text.splitlines())

again in Py3k.

Georg



From tjreedy at udel.edu  Thu Aug 28 01:17:54 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 27 Aug 2008 19:17:54 -0400
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
Message-ID: <g94nb4$jn9$1@ger.gmane.org>



Tarek Ziad? wrote:

>    [transform(e) as transformed for e in seq if some_test(transformed)]
> 
> In my use case I would simply have to write;:
> 
>   [l.strip() as stripped for l in text.split('\n') if stripped != '']
> 
> Which seems to me clear and concise.

-1 For me it is backward and confusing, whereas

lines = []
for l in test:
   l = l.strip()
   if l: lines.append(l)

is clear as could be.

To me, the drive to replace all for loops with list comps is 
mis-directed.  There is no end to the clauses people could propose to 
add: while, when, whatever, until, unless. mapped_to, transformed_by, 
and so on.  The result would soon by something quite different from 
Python as we know it.

tjr



From cvrebert at gmail.com  Thu Aug 28 01:29:52 2008
From: cvrebert at gmail.com (Chris Rebert)
Date: Wed, 27 Aug 2008 16:29:52 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <g94nb4$jn9$1@ger.gmane.org>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<g94nb4$jn9$1@ger.gmane.org>
Message-ID: <47c890dc0808271629v3b5f5713id517567d7a68ae99@mail.gmail.com>

On Wed, Aug 27, 2008 at 4:17 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>
>
> Tarek Ziad? wrote:
>
>>   [transform(e) as transformed for e in seq if some_test(transformed)]
>>
>> In my use case I would simply have to write;:
>>
>>  [l.strip() as stripped for l in text.split('\n') if stripped != '']
>>
>> Which seems to me clear and concise.
>
> -1 For me it is backward and confusing, whereas
>
> lines = []
> for l in test:
>  l = l.strip()
>  if l: lines.append(l)
>
> is clear as could be.
>
> To me, the drive to replace all for loops with list comps is mis-directed.
>  There is no end to the clauses people could propose to add: while, when,
> whatever, until, unless. mapped_to, transformed_by, and so on.  The result
> would soon by something quite different from Python as we know it.

Indeed, probably something approaching Common Lisp's overcomplicated
"loop" macro:
http://www.unixuser.org/~euske/doc/cl/loop.html

- Chris
========
Follow the path of the Iguana...
Rebertia: http://rebertia.com
Blog: http://blog.rebertia.com

>
> tjr
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>

From greg.ewing at canterbury.ac.nz  Thu Aug 28 02:38:44 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 28 Aug 2008 12:38:44 +1200
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808271021t265c5aeeo5e9b2e0b649c40bc@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
	<87ljykb6uv.fsf@physik.rwth-aachen.de>
	<d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
	<op.ugi98qn403jqhe@cesareprova.org>
	<d03afaa20808271021t265c5aeeo5e9b2e0b649c40bc@mail.gmail.com>
Message-ID: <48B5F394.9070004@canterbury.ac.nz>

Russ Paielli wrote:
> Lets not forget that C++ and Java use 
> zero-character identifiers for "self", and I don't hear a din of 
> complaints that it harms readability.

I'm not so sure about that. Frequently people adopt
conventions such as always writing this->x, or naming
all instance variables with an "m_" prefix, suggesting
that zero characters is considered a problem by many.

-- 
Greg


From george.sakkis at gmail.com  Thu Aug 28 03:10:10 2008
From: george.sakkis at gmail.com (George Sakkis)
Date: Wed, 27 Aug 2008 21:10:10 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <48B5F394.9070004@canterbury.ac.nz>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
	<87ljykb6uv.fsf@physik.rwth-aachen.de>
	<d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
	<op.ugi98qn403jqhe@cesareprova.org>
	<d03afaa20808271021t265c5aeeo5e9b2e0b649c40bc@mail.gmail.com>
	<48B5F394.9070004@canterbury.ac.nz>
Message-ID: <91ad5bf80808271810i67175528w59d11a397749c91b@mail.gmail.com>

On Wed, Aug 27, 2008 at 8:38 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>wrote:

Russ Paielli wrote:
>
>> Lets not forget that C++ and Java use zero-character identifiers for
>> "self", and I don't hear a din of complaints that it harms readability.
>>
>
> I'm not so sure about that. Frequently people adopt
> conventions such as always writing this->x, or naming
> all instance variables with an "m_" prefix, suggesting
> that zero characters is considered a problem by many.


And that's despite the fact that they have more assistance from IDEs for
autocomplete, highlighting, etc.

George
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/2507e082/attachment.html>

From dangyogi at gmail.com  Thu Aug 28 03:42:43 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Wed, 27 Aug 2008 21:42:43 -0400
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
In-Reply-To: <4a038ea30808271329p4dc7a5fdyc3273969e0b892d2@mail.gmail.com>
References: <4a038ea30808271329p4dc7a5fdyc3273969e0b892d2@mail.gmail.com>
Message-ID: <48B60293.2050609@gmail.com>

In this case, you're misunderstanding :-).

What I meant by a "long running" micro_thread starving other 
micro-threads, was a micro_thread that doesn't do anything that would 
cause it to be suspended (e.g., I/O, sleep).

For example, calculating the number PI to 2000 digits will starve other 
micro_threads.

Each time a micro_thread does something which causes it to be suspended 
(like a file.read that needs to access the disk, a socket.recv or a 
time.sleep), other micro_threads may run.  So the first micro_thread 
doesn't cause the whole os-thread (generally the whole Python program) 
to be suspended, like it does now.

But if one micro_thread uses the CPU for a long time without doing any 
I/O, then other micro_threads are starved because micro_threads are 
non-preemptive (unlike os-threads).

I'm adding a "breath" function that allows responsible micro_threads to 
"come up for air" periodically, to give other micro_threads a chance to 
run.  But nothing forces a micro_thread to cooperate like this...

I hopes this helps to clear things up!

-bruce


Jerry Spicklemire wrote:
> Bruce wrote:
>
> "The micro_pipes use the C_deferreds to suspend the thread
> and allow other threads to run."
>
> http://mail.python.org/pipermail/python-ideas/2008-August/001848.html
>
>
> However, in the PEP you mention, among the short list of
> disadvantages:
>
> "since there is no preemption, a long running micro-thread 
> will starve other micro-threads"
>
> http://mail.python.org/pipermail/python-ideas/2008-August/001825.html
>
>
>
> Are these two point contradictory, or am I simply 
> misunderstanding, as usual.
>   


From jacobolus at gmail.com  Thu Aug 28 05:39:45 2008
From: jacobolus at gmail.com (Jacob Rus)
Date: Wed, 27 Aug 2008 20:39:45 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>	<g8t1on$7ne$1@ger.gmane.org>	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>	<87ljykb6uv.fsf@physik.rwth-aachen.de>
	<d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
Message-ID: <g956m3$l6m$1@ger.gmane.org>

Russ Paielli wrote:
> My editor, Xemacs, also highlights the word "self." I personally find that a
> bit annoying, and perhaps that is part of the reason I don't appreciate the
> word. When I write an algorithm, I don't want to see some artifact
> highlighted, as if it is somehow the most important part of the algorithm. I
> like seeing keywords highlighted, but I don't want to see "self" highlighted
> because it is only a convention. Maybe I'm alone on that.

Change your editor such that the highlighting de-emphasizes the word 
self (by making it closer to the background color or whatever; this is 
what I do myself), and you should be all set then.

-Jacob



From ntoronto at cs.byu.edu  Thu Aug 28 05:38:35 2008
From: ntoronto at cs.byu.edu (Neil Toronto)
Date: Wed, 27 Aug 2008 21:38:35 -0600
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <47c890dc0808271629v3b5f5713id517567d7a68ae99@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>	<g94nb4$jn9$1@ger.gmane.org>
	<47c890dc0808271629v3b5f5713id517567d7a68ae99@mail.gmail.com>
Message-ID: <48B61DBB.3090309@cs.byu.edu>

Chris Rebert wrote:
> On Wed, Aug 27, 2008 at 4:17 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>>
>> To me, the drive to replace all for loops with list comps is mis-directed.
>>  There is no end to the clauses people could propose to add: while, when,
>> whatever, until, unless. mapped_to, transformed_by, and so on.  The result
>> would soon by something quite different from Python as we know it.
> 
> Indeed, probably something approaching Common Lisp's overcomplicated
> "loop" macro:
> http://www.unixuser.org/~euske/doc/cl/loop.html
> 
> - Chris

Doesn't have to be that bad. I'd personally like a way to make a loop 
expression that returns a list, with two operators that stuff new things 
on and update old ones. That's most likely a reflection of my 
expression-based-but-not-purely style, though.

One thing I've noticed in this discussion is that people seem to want 
the "as" keyword to operate as a general expression-assignment operator. 
Is that worth adding to the language? There would be no possibility of 
getting it mixed up with "==", and the word as an infix operator is 
sufficiently ugly that people would prefer "=" in general.

Neil



From andrew at atoulou.se  Thu Aug 28 06:53:40 2008
From: andrew at atoulou.se (Andrew Akira Toulouse)
Date: Wed, 27 Aug 2008 21:53:40 -0700
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
In-Reply-To: <48B60293.2050609@gmail.com>
References: <4a038ea30808271329p4dc7a5fdyc3273969e0b892d2@mail.gmail.com>
	<48B60293.2050609@gmail.com>
Message-ID: <c7450fba0808272153n3c45becbs15e64c7e6b819bdd@mail.gmail.com>

Forgive the pedantry: do you mean "breathe", rather than "breath"?

On Wed, Aug 27, 2008 at 6:42 PM, Bruce Frederiksen <dangyogi at gmail.com>wrote:

> In this case, you're misunderstanding :-).
>
> What I meant by a "long running" micro_thread starving other micro-threads,
> was a micro_thread that doesn't do anything that would cause it to be
> suspended (e.g., I/O, sleep).
>
> For example, calculating the number PI to 2000 digits will starve other
> micro_threads.
>
> Each time a micro_thread does something which causes it to be suspended
> (like a file.read that needs to access the disk, a socket.recv or a
> time.sleep), other micro_threads may run.  So the first micro_thread doesn't
> cause the whole os-thread (generally the whole Python program) to be
> suspended, like it does now.
>
> But if one micro_thread uses the CPU for a long time without doing any I/O,
> then other micro_threads are starved because micro_threads are
> non-preemptive (unlike os-threads).
>
> I'm adding a "breath" function that allows responsible micro_threads to
> "come up for air" periodically, to give other micro_threads a chance to run.
>  But nothing forces a micro_thread to cooperate like this...
>
> I hopes this helps to clear things up!
>
> -bruce
>
>
>
> Jerry Spicklemire wrote:
>
>> Bruce wrote:
>>
>> "The micro_pipes use the C_deferreds to suspend the thread
>> and allow other threads to run."
>>
>> http://mail.python.org/pipermail/python-ideas/2008-August/001848.html
>>
>>
>> However, in the PEP you mention, among the short list of
>> disadvantages:
>>
>> "since there is no preemption, a long running micro-thread will starve
>> other micro-threads"
>>
>> http://mail.python.org/pipermail/python-ideas/2008-August/001825.html
>>
>>
>>
>> Are these two point contradictory, or am I simply misunderstanding, as
>> usual.
>>
>>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/100d094f/attachment.html>

From bruce at leapyear.org  Thu Aug 28 07:07:51 2008
From: bruce at leapyear.org (Bruce Leban)
Date: Wed, 27 Aug 2008 22:07:51 -0700
Subject: [Python-ideas] Variations on a loop
Message-ID: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>

There are several different blocks of code you could tack onto a loop (I've
deliberately chosen somewhat unusual words to express these here):

    for x in items:
        # body
    interstitially:
        # things to do between loop iteration
        # (executed after each iteration in the loop when there is a next
value)
    subsequently:
        # things to do after the last element of the loop is processed
        # (when the loop is not exited by break)
    contrariwise:
        # things to do if the list was empty

For example:

    result = ""
    for x in items:
        result += str(x)
    interstitially:
        result += ", "
    contrariwise:
        result = "no data"

When I first learned that Python had an 'else' clause on loops, I assumed it
meant 'contrariwise'. I was surprised that it actually meant 'subsequently'.


To be more clear, contrariwise is essentially equivalent to:

    empty = True
    for x in items:
        empty = False
        # body
    if empty:
        # do contrariwise code

and interstitially is essentially equivalent to:

    first = True
    for x2 in items:
        if not first:
            # do interstitial code
            first = False
        x = x2
        # body

I think these are common/useful paradigms. I'm curious what others think.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080827/52db181d/attachment.html>

From cvrebert at gmail.com  Thu Aug 28 07:34:00 2008
From: cvrebert at gmail.com (Chris Rebert)
Date: Wed, 27 Aug 2008 22:34:00 -0700
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
Message-ID: <47c890dc0808272234n26522793i2fa9c8b53bd8e6f5@mail.gmail.com>

On Wed, Aug 27, 2008 at 10:07 PM, Bruce Leban <bruce at leapyear.org> wrote:
> There are several different blocks of code you could tack onto a loop (I've
> deliberately chosen somewhat unusual words to express these here):
>
>     for x in items:
>         # body
>     interstitially:
>         # things to do between loop iteration
>         # (executed after each iteration in the loop when there is a next
> value)
>     subsequently:
>         # things to do after the last element of the loop is processed
>         # (when the loop is not exited by break)
>     contrariwise:
>         # things to do if the list was empty
>
> For example:
>
>     result = ""
>     for x in items:
>         result += str(x)
>     interstitially:
>         result += ", "
>     contrariwise:
>         result = "no data"
>
> When I first learned that Python had an 'else' clause on loops, I assumed it
> meant 'contrariwise'. I was surprised that it actually meant 'subsequently'.
>
> To be more clear, contrariwise is essentially equivalent to:
>
>     empty = True
>     for x in items:
>         empty = False
>         # body
>     if empty:
>         # do contrariwise code
>
> and interstitially is essentially equivalent to:
>
>     first = True
>     for x2 in items:
>         if not first:
>             # do interstitial code
>             first = False
>         x = x2
>         # body

Just for comparison, (assuming my Perl6 book is still accurate),
"interstitially" is equivalent to "NEXT" blocks in Perl6 and
"continue" blocks in Perl5.
I don't think that's necessarily a good thing, but anyway...

- Chris

========
Follow the path of the Iguana...
Rebertia: http://rebertia.com
Blog: http://blog.rebertia.com

>
> I think these are common/useful paradigms. I'm curious what others think.
>
> --- Bruce
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


From cesare.dimauro at a-tono.com  Thu Aug 28 10:56:52 2008
From: cesare.dimauro at a-tono.com (Cesare Di Mauro)
Date: Thu, 28 Aug 2008 10:56:52 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
Message-ID: <op.ugk9s2ud03jqhe@cesareprova.org>

A solution could be this:

[stripped for l in text.split('\n') with l.strip() as stripped if stripped != '']

so that you can keep both values (l and l.strip()) too.

Cheers,
Cesare

On 27 agu 2008 at 18:51:41, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

> Hello
>
> There's a pattern I am doing all the time: filtering out some elements of a
> list, and cleaning them in the same move.
>
> For example, if I have a multi line text, where I want to:
>
> - keep non empty lines
> - clean non empty lines
>
> I am doing:
>
>     >>> text = """
>     ... this is a multi-line text\t
>     ...
>     ... \t\twith
>     ...
>     ... muliple lines."""
>
>     >>> [l.strip() for l in text.split('\n') if l.strip() != '']
>     ['this is a multi-line text', 'with', 'muliple lines.']
>
> It is not optimal, because I call strip() twice. I could use ifilter then
> imap or even use a real loop, but I
> want my simple, concise, list comprehension !  And I couldn't find a simple
> way to express it.
>
> The pattern can be generically resumed like this :
>
>    [transform(e) for e in seq if some_test(transform(e))]
>
> So what about using the 'as' keyword to extend lists comprehensions, and
> to avoid calling transform() twice ?
>
> Could be:
>
>    [transform(e) as transformed for e in seq if some_test(transformed)]
>
> In my use case I would simply have to write;:
>
>   [l.strip() as stripped for l in text.split('\n') if stripped != '']
>
> Which seems to me clear and concise.
>
> Regards,
> Tarek
>



-- 
Dott. Cesare Di Mauro
A-Tono S.r.l.
T.: (+39)095-7365314
Information in this email is confidential and may be privileged. It is intended for the addresses only.
If you have received it in error, please notify the sender immediately and delete it from your system. You should not otherwise copy it, retransmit it or use or disclose its content to anyone.
Thank you for your co-operation.


From qrczak at knm.org.pl  Thu Aug 28 11:04:41 2008
From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk)
Date: Thu, 28 Aug 2008 11:04:41 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <op.ugk9s2ud03jqhe@cesareprova.org>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<op.ugk9s2ud03jqhe@cesareprova.org>
Message-ID: <3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>

2008/8/28 Cesare Di Mauro <cesare.dimauro at a-tono.com>:

> A solution could be this:
>
> [stripped for l in text.split('\n') with l.strip() as stripped if stripped != '']
>
> so that you can keep both values (l and l.strip()) too.

In Haskell this would be (I translate only the list comprehension
structure, leaving expressions in Python syntax):

[stripped | l <- text.split('\n'), let stripped = l.strip(), stripped != '']

Python borrowed 2 out of 3 kinds of list comprehension constructs.

-- 
Marcin Kowalczyk
qrczak at knm.org.pl
http://qrnik.knm.org.pl/~qrczak/


From solipsis at pitrou.net  Thu Aug 28 15:13:22 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 28 Aug 2008 13:13:22 +0000 (UTC)
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
References: <48B2E256.6080008@gmail.com>
	<loom.20080826T120936-583@post.gmane.org>
	<48B59916.1000803@gmail.com>
Message-ID: <loom.20080828T125048-744@post.gmane.org>


Hi,

> I had considered that.  But the core by itself accomplishes nothing, 
> except to serve as a foundation for some kind of higher-level 
> constructs, so I put them together.  I guess having separate PEPs allows 
> them to evolve more independently.  (I'm new to this PEP process).

Having separate PEPs also makes it easier to discuss the issues piecewise rather
than a whole big chunk of additions.

> If I split them, so I keep posting updated versions on python-ideas?  Or 
> do I just accumulate the changes offline and post the completed PEP much 
> later?

I think it's better to post updated versions, at least as long as there seems to
be some interest.

> So micro_pipes don't store a sequence of values (like lists or deques), 
> but pass individual values on from one thread to another.  The 
> implementation proposed in the Python level pseudo code only stores one 
> value and will block the writer when it tries to write a second value 
> before the reader has read the first value.  This buffer size of one 
> could be expanded, but I've been working on the premise that this should 
> be kept as simple as possible for a first out, and then allowed to grow 
> after more experience is gained with it.

Yes, it's the kind of things that can be discussed as part of the implementation
rather than as part of the spec itself.

> They seem necessary to handle exception situations.  For example, when a 
> reader thread on a pipe dies with an exception, how is the write thread 
> notified?  What mechanism knows that this pipe was being read by the 
> errant thread so that it will never be read from again?

A pipe could be divided into two half-objects: the receiving end and the sending
end, each independently managed by the standard reference counting mechanism,
and referencing each other through weakrefs. That way, when e.g. the last
reference to the receiving end dies, the sending end will notice that and can
raise an exception when a thread tries to write to it.

In any case, I don't think creating standard "stdin" and "stdout" pipes for each
thread makes things any easier. You still have to handle the case of whatever
non-stdin/stdout pipes get created.

> > In this example, can you give the C pseudo-code and the equivalent Twisted
> > Python (pseudo-)code?
> >   
> Do you mean the pseudo code of the deferred implementation, or the 
> pseudo code for using the deferreds?

I mean the pseudo code for using the deferreds in the particular example which
is outlined. Also, if possible, the corresponding Twisted code, to see where and
how the two idioms diverge.

You can of course give pseudo-code for the implementation as well, but I think
discussing the implementation is prematurate if the API hasn't been discussed
first :-)

> Good suggestion!  I was hoping that some might show up here, but ...  I 
> guess I need to go looking for them!

A Twisted developer told me that he had tried to read your PEP (the first
version) but found it difficult to understand.

Regards

Antoine.




From josiah.carlson at gmail.com  Thu Aug 28 16:28:50 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 28 Aug 2008 07:28:50 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>
	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
Message-ID: <e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>

-1 on the feature, I use the compound expression as below, only have
the internal item be a generator expression to reduce peak memory
usage.  Also, the != condition is unnecessary.

[t for t in (t.strip() for t in text.split('\n')) if t]

Overloading 'as', 'with', etc., when there are simple expressions that
already do the *exact* same thing is silly.

Never mind that not everything needs to be done in a 1-liner or list
comprehension.

 - Josiah

On Wed, Aug 27, 2008 at 10:30 AM, Bruce Leban <bruce at leapyear.org> wrote:
> [t for t in [t.strip() for t in text.split('\n')] if t != '']
>
>
> On Wed, Aug 27, 2008 at 10:00 AM, Imri Goldberg <lorgandon at gmail.com> wrote:
>>
>> +1 from me, I find myself doing that all the time as well.
>>
>> On Wed, Aug 27, 2008 at 7:51 PM, Tarek Ziad? <ziade.tarek at gmail.com>
>> wrote:
>>>
>>> Hello
>>>
>>> There's a pattern I am doing all the time: filtering out some elements of
>>> a list, and cleaning them in the same move.
>>>
>>> For example, if I have a multi line text, where I want to:
>>>
>>> - keep non empty lines
>>> - clean non empty lines
>>>
>>> I am doing:
>>>
>>>     >>> text = """
>>>     ... this is a multi-line text\t
>>>     ...
>>>     ... \t\twith
>>>     ...
>>>     ... muliple lines."""
>>>
>>>     >>> [l.strip() for l in text.split('\n') if l.strip() != '']
>>>     ['this is a multi-line text', 'with', 'muliple lines.']
>>>
>>> It is not optimal, because I call strip() twice. I could use ifilter then
>>> imap or even use a real loop, but I
>>> want my simple, concise, list comprehension !  And I couldn't find a
>>> simple way to express it.
>>>
>>> The pattern can be generically resumed like this :
>>>
>>>    [transform(e) for e in seq if some_test(transform(e))]
>>>
>>> So what about using the 'as' keyword to extend lists comprehensions, and
>>> to avoid calling transform() twice ?
>>>
>>> Could be:
>>>
>>>    [transform(e) as transformed for e in seq if some_test(transformed)]
>>>
>>> In my use case I would simply have to write;:
>>>
>>>   [l.strip() as stripped for l in text.split('\n') if stripped != '']
>>>
>>> Which seems to me clear and concise.
>>>
>>> Regards,
>>> Tarek
>>>
>>> --
>>> Tarek Ziad? | Association AfPy | www.afpy.org
>>> Blog FR | http://programmation-python.org
>>> Blog EN | http://tarekziade.wordpress.com/
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> http://mail.python.org/mailman/listinfo/python-ideas
>>>
>>
>>
>>
>> --
>> Imri Goldberg
>> --------------------------------------
>> www.algorithm.co.il/blogs/
>> www.imri.co.il
>> --------------------------------------
>> -- insert signature here ----
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


From josiah.carlson at gmail.com  Thu Aug 28 16:28:50 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 28 Aug 2008 07:28:50 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>
	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
Message-ID: <e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>

-1 on the feature, I use the compound expression as below, only have
the internal item be a generator expression to reduce peak memory
usage.  Also, the != condition is unnecessary.

[t for t in (t.strip() for t in text.split('\n')) if t]

Overloading 'as', 'with', etc., when there are simple expressions that
already do the *exact* same thing is silly.

Never mind that not everything needs to be done in a 1-liner or list
comprehension.

 - Josiah

On Wed, Aug 27, 2008 at 10:30 AM, Bruce Leban <bruce at leapyear.org> wrote:
> [t for t in [t.strip() for t in text.split('\n')] if t != '']
>
>
> On Wed, Aug 27, 2008 at 10:00 AM, Imri Goldberg <lorgandon at gmail.com> wrote:
>>
>> +1 from me, I find myself doing that all the time as well.
>>
>> On Wed, Aug 27, 2008 at 7:51 PM, Tarek Ziad? <ziade.tarek at gmail.com>
>> wrote:
>>>
>>> Hello
>>>
>>> There's a pattern I am doing all the time: filtering out some elements of
>>> a list, and cleaning them in the same move.
>>>
>>> For example, if I have a multi line text, where I want to:
>>>
>>> - keep non empty lines
>>> - clean non empty lines
>>>
>>> I am doing:
>>>
>>>     >>> text = """
>>>     ... this is a multi-line text\t
>>>     ...
>>>     ... \t\twith
>>>     ...
>>>     ... muliple lines."""
>>>
>>>     >>> [l.strip() for l in text.split('\n') if l.strip() != '']
>>>     ['this is a multi-line text', 'with', 'muliple lines.']
>>>
>>> It is not optimal, because I call strip() twice. I could use ifilter then
>>> imap or even use a real loop, but I
>>> want my simple, concise, list comprehension !  And I couldn't find a
>>> simple way to express it.
>>>
>>> The pattern can be generically resumed like this :
>>>
>>>    [transform(e) for e in seq if some_test(transform(e))]
>>>
>>> So what about using the 'as' keyword to extend lists comprehensions, and
>>> to avoid calling transform() twice ?
>>>
>>> Could be:
>>>
>>>    [transform(e) as transformed for e in seq if some_test(transformed)]
>>>
>>> In my use case I would simply have to write;:
>>>
>>>   [l.strip() as stripped for l in text.split('\n') if stripped != '']
>>>
>>> Which seems to me clear and concise.
>>>
>>> Regards,
>>> Tarek
>>>
>>> --
>>> Tarek Ziad? | Association AfPy | www.afpy.org
>>> Blog FR | http://programmation-python.org
>>> Blog EN | http://tarekziade.wordpress.com/
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> http://mail.python.org/mailman/listinfo/python-ideas
>>>
>>
>>
>>
>> --
>> Imri Goldberg
>> --------------------------------------
>> www.algorithm.co.il/blogs/
>> www.imri.co.il
>> --------------------------------------
>> -- insert signature here ----
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


From dangyogi at gmail.com  Thu Aug 28 18:05:18 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Thu, 28 Aug 2008 12:05:18 -0400
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
In-Reply-To: <c7450fba0808272153n3c45becbs15e64c7e6b819bdd@mail.gmail.com>
References: <4a038ea30808271329p4dc7a5fdyc3273969e0b892d2@mail.gmail.com>	
	<48B60293.2050609@gmail.com>
	<c7450fba0808272153n3c45becbs15e64c7e6b819bdd@mail.gmail.com>
Message-ID: <48B6CCBE.8040402@gmail.com>

Yes!

Thanks!

-bruce

Andrew Akira Toulouse wrote:
> Forgive the pedantry: do you mean "breathe", rather than "breath"?
>
> On Wed, Aug 27, 2008 at 6:42 PM, Bruce Frederiksen <dangyogi at gmail.com 
> <mailto:dangyogi at gmail.com>> wrote:
>
>     In this case, you're misunderstanding :-).
>
>     What I meant by a "long running" micro_thread starving other
>     micro-threads, was a micro_thread that doesn't do anything that
>     would cause it to be suspended (e.g., I/O, sleep).
>
>     For example, calculating the number PI to 2000 digits will starve
>     other micro_threads.
>
>     Each time a micro_thread does something which causes it to be
>     suspended (like a file.read that needs to access the disk, a
>     socket.recv or a time.sleep), other micro_threads may run.  So the
>     first micro_thread doesn't cause the whole os-thread (generally
>     the whole Python program) to be suspended, like it does now.
>
>     But if one micro_thread uses the CPU for a long time without doing
>     any I/O, then other micro_threads are starved because
>     micro_threads are non-preemptive (unlike os-threads).
>
>     I'm adding a "breath" function that allows responsible
>     micro_threads to "come up for air" periodically, to give other
>     micro_threads a chance to run.  But nothing forces a micro_thread
>     to cooperate like this...
>
>     I hopes this helps to clear things up!
>
>     -bruce
>



From grosser.meister.morti at gmx.net  Thu Aug 28 18:20:05 2008
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Thu, 28 Aug 2008 18:20:05 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>	<op.ugk9s2ud03jqhe@cesareprova.org>
	<3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>
Message-ID: <48B6D035.6080804@gmx.net>

Marcin 'Qrczak' Kowalczyk schrieb:
> 2008/8/28 Cesare Di Mauro <cesare.dimauro at a-tono.com>:
>
>> A solution could be this:
>>
>> [stripped for l in text.split('\n') with l.strip() as stripped if stripped != '']
>>
>> so that you can keep both values (l and l.strip()) too.
>
> In Haskell this would be (I translate only the list comprehension
> structure, leaving expressions in Python syntax):
>
> [stripped | l <- text.split('\n'), let stripped = l.strip(), stripped != '']
>
> Python borrowed 2 out of 3 kinds of list comprehension constructs.
>

so maybe?

[stripped for l in text.split('\n') if stripped != '' let stripped = l.strip()]

but this would introduce a new keyword.


From rrr at ronadam.com  Thu Aug 28 18:14:30 2008
From: rrr at ronadam.com (Ron Adam)
Date: Thu, 28 Aug 2008 11:14:30 -0500
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
	<e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>
Message-ID: <48B6CEE6.5090606@ronadam.com>



Josiah Carlson wrote:
> -1 on the feature, I use the compound expression as below, only have
> the internal item be a generator expression to reduce peak memory
> usage.  Also, the != condition is unnecessary.
> 
> [t for t in (t.strip() for t in text.split('\n')) if t]



If split was a bit smarter...

     text.split(pattern=(' *\n+ *'))


;-)


It can be done with the re module, but I need to look that up each time I 
use it since I don't use it enough to remember all it's subtleties.

Ron


From josiah.carlson at gmail.com  Thu Aug 28 18:59:38 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 28 Aug 2008 09:59:38 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <48B6D035.6080804@gmx.net>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<op.ugk9s2ud03jqhe@cesareprova.org>
	<3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>
	<48B6D035.6080804@gmx.net>
Message-ID: <e6511dbf0808280959l5cd4cf32wd8167eb9e52b1099@mail.gmail.com>

That's just as bad as the original.  Please stop offering new syntax.

 - Josiah

On Thu, Aug 28, 2008 at 9:20 AM, Mathias Panzenb?ck
<grosser.meister.morti at gmx.net> wrote:
> Marcin 'Qrczak' Kowalczyk schrieb:
>> 2008/8/28 Cesare Di Mauro <cesare.dimauro at a-tono.com>:
>>
>>> A solution could be this:
>>>
>>> [stripped for l in text.split('\n') with l.strip() as stripped if stripped != '']
>>>
>>> so that you can keep both values (l and l.strip()) too.
>>
>> In Haskell this would be (I translate only the list comprehension
>> structure, leaving expressions in Python syntax):
>>
>> [stripped | l <- text.split('\n'), let stripped = l.strip(), stripped != '']
>>
>> Python borrowed 2 out of 3 kinds of list comprehension constructs.
>>
>
> so maybe?
>
> [stripped for l in text.split('\n') if stripped != '' let stripped = l.strip()]
>
> but this would introduce a new keyword.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From josiah.carlson at gmail.com  Thu Aug 28 19:12:00 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 28 Aug 2008 10:12:00 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <48B6CEE6.5090606@ronadam.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>
	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
	<e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>
	<48B6CEE6.5090606@ronadam.com>
Message-ID: <e6511dbf0808281012u23086822we643b46f2f087c04@mail.gmail.com>

The split/strip stuff is simple on purpose; it's fast.  Tossing in
regular expressions handling is a great way to slow down the general
case, never mind if you actually want to split on the passed literal
string.

 - Josiah

On Thu, Aug 28, 2008 at 9:14 AM, Ron Adam <rrr at ronadam.com> wrote:
>
>
> Josiah Carlson wrote:
>>
>> -1 on the feature, I use the compound expression as below, only have
>> the internal item be a generator expression to reduce peak memory
>> usage.  Also, the != condition is unnecessary.
>>
>> [t for t in (t.strip() for t in text.split('\n')) if t]
>
>
>
> If split was a bit smarter...
>
>    text.split(pattern=(' *\n+ *'))
>
>
> ;-)
>
>
> It can be done with the re module, but I need to look that up each time I
> use it since I don't use it enough to remember all it's subtleties.
>
> Ron
>


From rrr at ronadam.com  Thu Aug 28 19:41:20 2008
From: rrr at ronadam.com (Ron Adam)
Date: Thu, 28 Aug 2008 12:41:20 -0500
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <e6511dbf0808281012u23086822we643b46f2f087c04@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>	
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>	
	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>	
	<e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>	
	<48B6CEE6.5090606@ronadam.com>
	<e6511dbf0808281012u23086822we643b46f2f087c04@mail.gmail.com>
Message-ID: <48B6E340.2010205@ronadam.com>



Josiah Carlson wrote:
> The split/strip stuff is simple on purpose; it's fast.  Tossing in
> regular expressions handling is a great way to slow down the general
> case, never mind if you actually want to split on the passed literal
> string.

That's easy to remedy without making it overly complex.

 >>    text.split_pattern(' *\n+ *')


Cheers,
   Ron



>  - Josiah
> 
> On Thu, Aug 28, 2008 at 9:14 AM, Ron Adam <rrr at ronadam.com> wrote:
>>
>> Josiah Carlson wrote:
>>> -1 on the feature, I use the compound expression as below, only have
>>> the internal item be a generator expression to reduce peak memory
>>> usage.  Also, the != condition is unnecessary.
>>>
>>> [t for t in (t.strip() for t in text.split('\n')) if t]
>>
>>
>> If split was a bit smarter...
>>
>>    text.split(pattern=(' *\n+ *'))
>>
>>
>> ;-)
>>
>>
>> It can be done with the re module, but I need to look that up each time I
>> use it since I don't use it enough to remember all it's subtleties.
>>
>> Ron
>>
> 


From veloso at verylowsodium.com  Thu Aug 28 20:40:12 2008
From: veloso at verylowsodium.com (Greg Falcon)
Date: Thu, 28 Aug 2008 14:40:12 -0400
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <48B6E340.2010205@ronadam.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>
	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
	<e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>
	<48B6CEE6.5090606@ronadam.com>
	<e6511dbf0808281012u23086822we643b46f2f087c04@mail.gmail.com>
	<48B6E340.2010205@ronadam.com>
Message-ID: <3cdcefb80808281140m5092afedj3e102f8df2f5e6fd@mail.gmail.com>

On Thu, Aug 28, 2008 at 1:41 PM, Ron Adam <rrr at ronadam.com> wrote:
> Josiah Carlson wrote:
>>
>> The split/strip stuff is simple on purpose; it's fast.  Tossing in
>> regular expressions handling is a great way to slow down the general
>> case, never mind if you actually want to split on the passed literal
>> string.
>
> That's easy to remedy without making it overly complex.
>
>>>    text.split_pattern(' *\n+ *')

We're well off the original subject now, but what you propose can
today be written

re.split(' *\n+ *', text)

which I'd argue is just as easy to remember as your proposed new
method, and in fact reads better, as it's much easier to guess that
regex matching is going on.

Greg F


From grosser.meister.morti at gmx.net  Thu Aug 28 22:16:56 2008
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Thu, 28 Aug 2008 22:16:56 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <e6511dbf0808280959l5cd4cf32wd8167eb9e52b1099@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>	
	<op.ugk9s2ud03jqhe@cesareprova.org>	
	<3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>	
	<48B6D035.6080804@gmx.net>
	<e6511dbf0808280959l5cd4cf32wd8167eb9e52b1099@mail.gmail.com>
Message-ID: <48B707B8.4060001@gmx.net>

Josiah Carlson schrieb:
> That's just as bad as the original.

I know. I wasn't serious (I should have made that clear).

> Please stop offering new syntax.

Do you mean on this particular issue or ever again? So you don't want any new
syntax in python ever again??


From tjreedy at udel.edu  Thu Aug 28 22:37:20 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 28 Aug 2008 16:37:20 -0400
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <48B6D035.6080804@gmx.net>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>	<op.ugk9s2ud03jqhe@cesareprova.org>	<3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>
	<48B6D035.6080804@gmx.net>
Message-ID: <g9729v$esa$1@ger.gmane.org>



Mathias Panzenb?ck wrote:
> Marcin 'Qrczak' Kowalczyk schrieb:
>> 2008/8/28 Cesare Di Mauro <cesare.dimauro at a-tono.com>:
>> [stripped | l <- text.split('\n'), let stripped = l.strip(), stripped != '']
>>
>> Python borrowed 2 out of 3 kinds of list comprehension constructs.
>>
> 
> so maybe?
> 
> [stripped for l in text.split('\n') if stripped != '' let stripped = l.strip()]

To parallel the Haskell-ish example, this should be

[stripped for l in text.split('/n') stripped as l.strip() if stripped != '']

but the clause has 'as' in the middle instead of at the beginning, 
making it hard to parse.  Haskell used commas

[stripped for l in text.split('/n'), stripped as l.strip(), if stripped 
!= '']

but I think this would conflict with Python's other comma usage.  Most 
feasible, I think, would be

[stripped for l in text.split('/n') with stripped as l.strip() if 
stripped != '']

This corresponds to the multi-statement for loop version

_=[]
for l in text.split('\n'):
   stripped = l.strip()
   if stripped != '':
     _.append(stripped)

with 'stripped = l.strip()' replaced by 'with stripped as l.strip()'. 
If other use cases were presented that could not be more easily written 
otherwise, as with the re.split() version, I might at least be neutral 
on this.

Terry Jan Reedy





From rrr at ronadam.com  Fri Aug 29 02:27:18 2008
From: rrr at ronadam.com (Ron Adam)
Date: Thu, 28 Aug 2008 19:27:18 -0500
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <3cdcefb80808281140m5092afedj3e102f8df2f5e6fd@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>	
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>	
	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>	
	<e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>	
	<48B6CEE6.5090606@ronadam.com>	
	<e6511dbf0808281012u23086822we643b46f2f087c04@mail.gmail.com>	
	<48B6E340.2010205@ronadam.com>
	<3cdcefb80808281140m5092afedj3e102f8df2f5e6fd@mail.gmail.com>
Message-ID: <48B74266.3080704@ronadam.com>



Greg Falcon wrote:
> On Thu, Aug 28, 2008 at 1:41 PM, Ron Adam <rrr at ronadam.com> wrote:
>> Josiah Carlson wrote:
>>> The split/strip stuff is simple on purpose; it's fast.  Tossing in
>>> regular expressions handling is a great way to slow down the general
>>> case, never mind if you actually want to split on the passed literal
>>> string.
>> That's easy to remedy without making it overly complex.
>>
>>>>    text.split_pattern(' *\n+ *')
> 
> We're well off the original subject now, but what you propose can
> today be written
> 
> re.split(' *\n+ *', text)

Yep.


> which I'd argue is just as easy to remember as your proposed new
> method, and in fact reads better, as it's much easier to guess that
> regex matching is going on.

Who's arguing?  ;-)

The main point is that extending the core syntax isn't needed.  Extending 
the library should be considered first.

Cheers,
   Ron


From leif.walsh at gmail.com  Fri Aug 29 03:16:20 2008
From: leif.walsh at gmail.com (Leif Walsh)
Date: Thu, 28 Aug 2008 21:16:20 -0400
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <48B74266.3080704@ronadam.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<fa980a6f0808271000q50da0b62ud47fab4ee4dbdef8@mail.gmail.com>
	<cf5b87740808271030o5506fd04n8ec061cf985bfcb4@mail.gmail.com>
	<e6511dbf0808280728q3769e6d4nbc11c7ac89126dbc@mail.gmail.com>
	<48B6CEE6.5090606@ronadam.com>
	<e6511dbf0808281012u23086822we643b46f2f087c04@mail.gmail.com>
	<48B6E340.2010205@ronadam.com>
	<3cdcefb80808281140m5092afedj3e102f8df2f5e6fd@mail.gmail.com>
	<48B74266.3080704@ronadam.com>
Message-ID: <cc7430500808281816k5f638351j1d4c14090a03fe1f@mail.gmail.com>

-1 on the first post.

> [l.strip() as stripped for l in text.split('\n') if stripped != '']

This is equivalent, with respect to semantics as well as efficiency, to:

[s for s in (l.strip() for l in text.split('\n')) if s != '']

And, even easier to read:

stripped = (l.strip() for l in text.split('\n'))
non_null_lines = [s for s in stripped if s]  # let's use the implicit
truth value too

It seems to me that there's too much being asked for here when the
tools are already all available.  Generator expressions/list
comprehensions seem to be getting a lot of attention from python-ideas
lately, when it's really making the code _harder_ to read, and not
even easier to write, because of all the extra thinking required.  In
fact, most of these things are better done with straight-out looping
and yielding:

for line in text.split('\n'):
  stripped = l.strip()
  if stripped:
    yield stripped

Now _that_ I can read!

-- 
Cheers,
Leif


From josiah.carlson at gmail.com  Fri Aug 29 06:21:14 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 28 Aug 2008 21:21:14 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <48B707B8.4060001@gmx.net>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<op.ugk9s2ud03jqhe@cesareprova.org>
	<3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>
	<48B6D035.6080804@gmx.net>
	<e6511dbf0808280959l5cd4cf32wd8167eb9e52b1099@mail.gmail.com>
	<48B707B8.4060001@gmx.net>
Message-ID: <e6511dbf0808282121r17497dbdi19a4f6163a585d13@mail.gmail.com>

On Thu, Aug 28, 2008 at 1:16 PM, Mathias Panzenb?ck
<grosser.meister.morti at gmx.net> wrote:
> Josiah Carlson schrieb:
>> That's just as bad as the original.
>
> I know. I wasn't serious (I should have made that clear).
>
>> Please stop offering new syntax.
>
> Do you mean on this particular issue or ever again? So you don't want any new
> syntax in python ever again??

This issue.  It's just like the "group by" operator that was proposed
months ago and rejected.

Keep it simple.  Additional syntax typically burdens the user with
additional mental overhead.  Decorators are *still* causing
significant consternation among relatively new users who haven't seen
them before, as have variable leakage out of for loops and list
comprehensions.

 - Josiah


From andrew at atoulou.se  Fri Aug 29 07:50:36 2008
From: andrew at atoulou.se (Andrew Akira Toulouse)
Date: Thu, 28 Aug 2008 22:50:36 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <e6511dbf0808282121r17497dbdi19a4f6163a585d13@mail.gmail.com>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<op.ugk9s2ud03jqhe@cesareprova.org>
	<3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>
	<48B6D035.6080804@gmx.net>
	<e6511dbf0808280959l5cd4cf32wd8167eb9e52b1099@mail.gmail.com>
	<48B707B8.4060001@gmx.net>
	<e6511dbf0808282121r17497dbdi19a4f6163a585d13@mail.gmail.com>
Message-ID: <c7450fba0808282250x406c44e5u882b0d3f6de0dff2@mail.gmail.com>

On Thu, Aug 28, 2008 at 9:21 PM, Josiah Carlson <josiah.carlson at gmail.com>wrote:

> On Thu, Aug 28, 2008 at 1:16 PM, Mathias Panzenb?ck
> <grosser.meister.morti at gmx.net> wrote:
> > Josiah Carlson schrieb:
> >> That's just as bad as the original.
> >
> > I know. I wasn't serious (I should have made that clear).
> >
> >> Please stop offering new syntax.
> >
> > Do you mean on this particular issue or ever again? So you don't want any
> new
> > syntax in python ever again??
>
> This issue.  It's just like the "group by" operator that was proposed
> months ago and rejected.
>

Also, I already suggested this exact thing (with and as in list
comprehensions) a few weeks ago, and it was rejected.

--Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080828/2f4183db/attachment.html>

From cesare.dimauro at a-tono.com  Fri Aug 29 08:33:10 2008
From: cesare.dimauro at a-tono.com (Cesare Di Mauro)
Date: Fri, 29 Aug 2008 08:33:10 +0200
Subject: [Python-ideas] More power in list comprehensions with the 'as'
 keyword
In-Reply-To: <g9729v$esa$1@ger.gmane.org>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<op.ugk9s2ud03jqhe@cesareprova.org>
	<3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>
	<48B6D035.6080804@gmx.net> <g9729v$esa$1@ger.gmane.org>
Message-ID: <op.ugmxtkcj03jqhe@cesareprova.org>

On 28 agu 2008 at 22:37:20, Terry Reedy <tjreedy at udel.edu> wrote:

> To parallel the Haskell-ish example, this should be
>
> [stripped for l in text.split('/n') stripped as l.strip() if stripped != '']
>
> but the clause has 'as' in the middle instead of at the beginning,
> making it hard to parse.  Haskell used commas
>
> [stripped for l in text.split('/n'), stripped as l.strip(), if stripped
> != '']
>
> but I think this would conflict with Python's other comma usage.  Most
> feasible, I think, would be
>
> [stripped for l in text.split('/n') with stripped as l.strip() if
> stripped != '']
>
> This corresponds to the multi-statement for loop version
>
> _=[]
> for l in text.split('\n'):
>    stripped = l.strip()
>    if stripped != '':
>      _.append(stripped)
>
> with 'stripped = l.strip()' replaced by 'with stripped as l.strip()'.
> If other use cases were presented that could not be more easily written
> otherwise, as with the re.split() version, I might at least be neutral
> on this.
>
> Terry Jan Reedy

We already a "with Expression as Identifier" syntax that is well known
and used in Python: why use something different?

[stripped for l in text.split('\n') with l.strip() as stripped if stripped != '']

will be a much better syntax to parse and acquire for a typical
pythonista. ;)

Cheers,
Cesare



From josiah.carlson at gmail.com  Fri Aug 29 08:43:31 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 28 Aug 2008 23:43:31 -0700
Subject: [Python-ideas] More power in list comprehensions with the 'as'
	keyword
In-Reply-To: <op.ugmxtkcj03jqhe@cesareprova.org>
References: <94bdd2610808270951g2c191469g97e5008775c3de01@mail.gmail.com>
	<op.ugk9s2ud03jqhe@cesareprova.org>
	<3f4107910808280204jc24447bjaa52151bbd1f6d0b@mail.gmail.com>
	<48B6D035.6080804@gmx.net> <g9729v$esa$1@ger.gmane.org>
	<op.ugmxtkcj03jqhe@cesareprova.org>
Message-ID: <e6511dbf0808282343w292a94ffr10f67745dc9ef54a@mail.gmail.com>

On Thu, Aug 28, 2008 at 11:33 PM, Cesare Di Mauro
<cesare.dimauro at a-tono.com> wrote:
> On 28 agu 2008 at 22:37:20, Terry Reedy <tjreedy at udel.edu> wrote:
>
>> To parallel the Haskell-ish example, this should be
>>
>> [stripped for l in text.split('/n') stripped as l.strip() if stripped != '']
>>
>> but the clause has 'as' in the middle instead of at the beginning,
>> making it hard to parse.  Haskell used commas
>>
>> [stripped for l in text.split('/n'), stripped as l.strip(), if stripped
>> != '']
>>
>> but I think this would conflict with Python's other comma usage.  Most
>> feasible, I think, would be
>>
>> [stripped for l in text.split('/n') with stripped as l.strip() if
>> stripped != '']
>>
>> This corresponds to the multi-statement for loop version
>>
>> _=[]
>> for l in text.split('\n'):
>>    stripped = l.strip()
>>    if stripped != '':
>>      _.append(stripped)
>>
>> with 'stripped = l.strip()' replaced by 'with stripped as l.strip()'.
>> If other use cases were presented that could not be more easily written
>> otherwise, as with the re.split() version, I might at least be neutral
>> on this.
>>
>> Terry Jan Reedy
>
> We already a "with Expression as Identifier" syntax that is well known
> and used in Python: why use something different?
>
> [stripped for l in text.split('\n') with l.strip() as stripped if stripped != '']
>
> will be a much better syntax to parse and acquire for a typical
> pythonista. ;)

Just because it exists, doesn't mean that it's "well known and used".
Also, don't conflate the need to handle context management (locking,
closing files, etc.) with the false perceived need to add temporary
assignments in list comprehensions and generator expressions.

 - Josiah


From dangyogi at gmail.com  Fri Aug 29 14:35:45 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Fri, 29 Aug 2008 08:35:45 -0400
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
In-Reply-To: <loom.20080828T125048-744@post.gmane.org>
References: <48B2E256.6080008@gmail.com>	<loom.20080826T120936-583@post.gmane.org>	<48B59916.1000803@gmail.com>
	<loom.20080828T125048-744@post.gmane.org>
Message-ID: <48B7ED21.8000605@gmail.com>

Antoine Pitrou wrote:
> Having separate PEPs also makes it easier to discuss the issues piecewise rather
> than a whole big chunk of additions.
>   
I am preparing 3 new PEPs now.  One for the C level, one for 
micro-threads which provides some basic capabilities, and one for 
micro-pipes which I expect will be the one that will change the most and 
take the longest to put to bed.  I'm also concentrating on APIs rather 
than implementations and will include examples of using these APIs.
>> They seem necessary to handle exception situations.  For example, when a 
>> reader thread on a pipe dies with an exception, how is the write thread 
>> notified?  What mechanism knows that this pipe was being read by the 
>> errant thread so that it will never be read from again?
>>     
>
> A pipe could be divided into two half-objects: the receiving end and the sending
> end, each independently managed by the standard reference counting mechanism,
> and referencing each other through weakrefs. That way, when e.g. the last
> reference to the receiving end dies, the sending end will notice that and can
> raise an exception when a thread tries to write to it.
>   
I don't think that this would be very portable to other flavors of 
python (jython/ironpython/pypy) that don't use reference counting.  It 
makes the thread termination dependent on the implementation of the 
garbage collector.
> In any case, I don't think creating standard "stdin" and "stdout" pipes for each
> thread makes things any easier. You still have to handle the case of whatever
> non-stdin/stdout pipes get created.
>   
I imagine that the python programmer would not be allowed to create 
micro-pipes directly.  I'm thinking that there would be a num_stdout (= 
0) parameter on the micro_thread constructor that creates that many 
micro_pipes, and then the python programmer can connect these to the 
stdin of other micro-threads.  The reason for this is so that the 
underlying implementation always knows how the dots are connected so 
that it can provide sensible exception/abort semantics.

I'm working on another project that uses generators a _lot_ and there 
are problems there because 'for' loops don't call 'close' on generators 
to clean things up.  I've also hit problems where the code works fine on 
CPython, but fails on jython and ironpython because I'm relying on the 
reference counting to immediately collect abandoned generators and run 
their 'finally' clauses.
>   
>> Good suggestion!  I was hoping that some might show up here, but ...  I 
>> guess I need to go looking for them!
>>     
>
> A Twisted developer told me that he had tried to read your PEP (the first
> version) but found it difficult to understand.
>   
I hope that the new versions will be easier to follow!

Thanks!

-bruce


From dangyogi at gmail.com  Fri Aug 29 14:49:32 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Fri, 29 Aug 2008 08:49:32 -0400
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
Message-ID: <48B7F05C.6090405@gmail.com>

Bruce Leban wrote:
>
> For example:
>
>     result = ""
>     for x in items:
>         result += str(x)
>     interstitially:
>         result += ", "
>     contrariwise:
>         result = "no data"
We already have statements that only apply within loops (break and 
continue), how about some expressions that only apply in for loops: 
'last' and 'empty':

for x in items:
    result += str(x)
    if not last:
        result += ', '
else:
    if empty:
        result = "no data"

-bruce (not to be confused with Bruce :-)


From grosser.meister.morti at gmx.net  Fri Aug 29 15:09:28 2008
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Fri, 29 Aug 2008 15:09:28 +0200
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
In-Reply-To: <48B7ED21.8000605@gmail.com>
References: <48B2E256.6080008@gmail.com>	<loom.20080826T120936-583@post.gmane.org>	<48B59916.1000803@gmail.com>	<loom.20080828T125048-744@post.gmane.org>
	<48B7ED21.8000605@gmail.com>
Message-ID: <48B7F508.1080003@gmx.net>

Bruce Frederiksen schrieb:
>
> I'm working on another project that uses generators a _lot_ and there
> are problems there because 'for' loops don't call 'close' on generators
> to clean things up.  I've also hit problems where the code works fine on
> CPython, but fails on jython and ironpython because I'm relying on the
> reference counting to immediately collect abandoned generators and run
> their 'finally' clauses.
>

You can use "with closing" to ensure this. However, it bloats the code a tiny bit.

from __future__ import with_statement
from itertools  import islice
from contextlib import closing

def gen():
	try:
		print "before"
		yield 1
		print "between"
		yield 2
		print "after"
	except GeneratorExit:
		print "exit"
	finally:
		print "finally"

>>> with closing(gen()) as g:
...	for x in islice(g,1):
...		print x
...	
before
1
exit
finally


From grosser.meister.morti at gmx.net  Fri Aug 29 15:19:59 2008
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Fri, 29 Aug 2008 15:19:59 +0200
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <48B7F05C.6090405@gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com>
Message-ID: <48B7F77F.1030303@gmx.net>

Bruce Frederiksen wrote:
> Bruce Leban wrote:
>>
>> For example:
>>
>>     result = ""
>>     for x in items:
>>         result += str(x)
>>     interstitially:
>>         result += ", "
>>     contrariwise:
>>         result = "no data"
> We already have statements that only apply within loops (break and
> continue), how about some expressions that only apply in for loops:
> 'last' and 'empty':
>
> for x in items:
>    result += str(x)
>    if not last:
>        result += ', '
> else:
>    if empty:
>        result = "no data"
>
> -bruce (not to be confused with Bruce :-)


I don't know. I don't like either. What about named loops/loop objects? Using
this you have no need for new keywords:

myloop = for x in items:
	result += str(x)
	if not myloop.last:
		result += ', '

if myloop.empty:
	result = 'no data'

However, how to calculate the last property for generators where it might depend
on some side effects if it's really the last item?

Other things would be possible, too:

a = for x in xs:
	b = for y in ys:
		c = for z in zs:
			if cond(z):
				break b
			if cond(y):
				continue a

However, this would increase clutter and might be a bad idea for the same reason
goto is a bad idea. I also would be possible to pass the loop object to another
function which then might call continue/break which changes the controlflow
completely and makes the code very hard to read. So I guess this is a bad idea.

	-panzi


From bruce at leapyear.org  Fri Aug 29 19:26:05 2008
From: bruce at leapyear.org (Bruce Leban)
Date: Fri, 29 Aug 2008 10:26:05 -0700
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <48B7F77F.1030303@gmx.net>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com> <48B7F77F.1030303@gmx.net>
Message-ID: <cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>

On Fri, Aug 29, 2008 at 6:19 AM, Mathias Panzenb?ck <
grosser.meister.morti at gmx.net> wrote:

> Bruce Frederiksen wrote:
> > We already have statements that only apply within loops (break and
> > continue), how about some expressions that only apply in for loops:
> > 'last' and 'empty':
>
> I don't know. I don't like either. What about named loops/loop objects?
> Using
> this you have no need for new keywords:
>
> myloop = for x in items:
>        result += str(x)
>        if not myloop.last:
>                result += ', '
> if myloop.empty:
>        result = 'no data'
>

This is interesting. I don't like the fact that you have the loop variable
scope extending past the end of the loop and it's not clear what we can do
with that. It occurs to me that we can do something like this without
changes to the language but it's clumsy:

for x in funky(items):
    result += str(x.value)
    if not x.last():
        result += ', '
else:
    if x == funky_empty:
        result = 'no data'


> However, how to calculate the last property for generators where it might
> depend
> on some side effects if it's really the last item?
>

 Note that I wrote x.last() not x.last: when you check last, it pre-fetches
the next value. That's not an issue in the above code but could be in other
cases, so you have to be careful when you use it.

This could also provide other things like automatic counting of loop
iterations:

for x in funky(items):
    foo(x.index, x.value)

It wouldn't provide the ability to do nested breaks and continues which I'd
like to see.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080829/fd311c73/attachment.html>

From dangyogi at gmail.com  Fri Aug 29 19:53:01 2008
From: dangyogi at gmail.com (Bruce Frederiksen)
Date: Fri, 29 Aug 2008 13:53:01 -0400
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>	
	<48B7F05C.6090405@gmail.com> <48B7F77F.1030303@gmx.net>
	<cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>
Message-ID: <48B8377D.7060605@gmail.com>

Bruce Leban wrote:
>
>
> On Fri, Aug 29, 2008 at 6:19 AM, Mathias Panzenb?ck 
> <grosser.meister.morti at gmx.net <mailto:grosser.meister.morti at gmx.net>> 
> wrote:
>
>     Bruce Frederiksen wrote:
>     > We already have statements that only apply within loops (break and
>     > continue), how about some expressions that only apply in for loops:
>     > 'last' and 'empty':
>
>     I don't know. I don't like either. What about named loops/loop
>     objects? Using
>     this you have no need for new keywords:
>
>     myloop = for x in items:
>            result += str(x)
>            if not myloop.last:
>                    result += ', '
>     if myloop.empty:
>            result = 'no data'
>
>
> This is interesting. I don't like the fact that you have the loop 
> variable scope extending past the end of the loop and it's not clear 
> what we can do with that. It occurs to me that we can do something 
> like this without changes to the language but it's clumsy:
>
> for x in funky(items):
>     result += str(x.value)
>     if not x.last():
>         result += ', '
> else:
>     if x == funky_empty:
>         result = 'no data'
>
So x.first seems to make more sense than x.last.  You could also name 
the loops like:

outer = funky(items1)
for x in outer:
    middle = funky(items2)
    for y in middle:
        inner = funky(items3)
        for z in inner:
            middle.continue_()

It seems to implement this, funky.continue_ and funky.break_ would have 
to raise exceptions.  If the for loop honored the 'throw' method (of 
generators), it seems like this might be made to work...

-bruce


From dwblas at gmail.com  Fri Aug 29 22:47:15 2008
From: dwblas at gmail.com (David Blaschke)
Date: Fri, 29 Aug 2008 13:47:15 -0700
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <48B8377D.7060605@gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com> <48B7F77F.1030303@gmx.net>
	<cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>
	<48B8377D.7060605@gmail.com>
Message-ID: <7e69a0400808291347j2692f43cu9d7b07bc75b53835@mail.gmail.com>

This idea
elif x == funky_empty:
       result = 'no data'

Can be accomplished with
if not len(items):
       result = 'no data'
which means that part of the proposal can be scratched IMHO.


From bruce at leapyear.org  Sat Aug 30 02:25:35 2008
From: bruce at leapyear.org (Bruce Leban)
Date: Fri, 29 Aug 2008 17:25:35 -0700
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <7e69a0400808291347j2692f43cu9d7b07bc75b53835@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com> <48B7F77F.1030303@gmx.net>
	<cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>
	<48B8377D.7060605@gmail.com>
	<7e69a0400808291347j2692f43cu9d7b07bc75b53835@mail.gmail.com>
Message-ID: <cf5b87740808291725v4f0aa3b2u77fcbde5281df652@mail.gmail.com>

On Fri, Aug 29, 2008 at 1:47 PM, David Blaschke <dwblas at gmail.com> wrote:

> This idea
> elif x == funky_empty:
>       result = 'no data'
>
> Can be accomplished with
> if not len(items):
>       result = 'no data'
> which means that part of the proposal can be scratched IMHO.
>

Not quite: len(items) requires iterating across the entire list which may
not be necessary or desirable. Plus you may need an intermediate variable to
store it. I think that's inferior.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080829/9ac32da7/attachment.html>

From steven.bethard at gmail.com  Sat Aug 30 04:17:38 2008
From: steven.bethard at gmail.com (Steven Bethard)
Date: Fri, 29 Aug 2008 20:17:38 -0600
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <cf5b87740808291725v4f0aa3b2u77fcbde5281df652@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com> <48B7F77F.1030303@gmx.net>
	<cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>
	<48B8377D.7060605@gmail.com>
	<7e69a0400808291347j2692f43cu9d7b07bc75b53835@mail.gmail.com>
	<cf5b87740808291725v4f0aa3b2u77fcbde5281df652@mail.gmail.com>
Message-ID: <d11dcfba0808291917s5bf0a6d7jcf76bf2e046f0749@mail.gmail.com>

On Fri, Aug 29, 2008 at 6:25 PM, Bruce Leban <bruce at leapyear.org> wrote:
>
> On Fri, Aug 29, 2008 at 1:47 PM, David Blaschke <dwblas at gmail.com> wrote:
>>
>> This idea
>> elif x == funky_empty:
>>       result = 'no data'
>>
>> Can be accomplished with
>> if not len(items):
>>       result = 'no data'
>> which means that part of the proposal can be scratched IMHO.
>
> Not quite: len(items) requires iterating across the entire list which may
> not be necessary or desirable. Plus you may need an intermediate variable to
> store it. I think that's inferior.

If len(items) requires iterating across the entire list, then you're
not using a Python list object, you're using someone else's (poorly
implemented) list data structure. Python lists give len(obj) in O(1).

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
 --- Bucky Katt, Get Fuzzy


From collinw at gmail.com  Sat Aug 30 04:50:35 2008
From: collinw at gmail.com (Collin Winter)
Date: Fri, 29 Aug 2008 19:50:35 -0700
Subject: [Python-ideas] Inline 'raises' expression
In-Reply-To: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
References: <3d0cebfb0808240343i719416bci630df64397f9a1a1@mail.gmail.com>
Message-ID: <43aa6ff70808291950k67252d5ax20f63ce458cd92dc@mail.gmail.com>

On Sun, Aug 24, 2008 at 3:43 AM, Fredrik Johansson
<fredrik.johansson at gmail.com> wrote:
> It happens that I'm just interested in whether an expression raises an
> exception, not the return value. This might look something like
>
>    try:
>        check_status()
>    except IOError:
>        cleanup()
>
> which could be written more simply as
>
>    if check_status() raises IOError:
>        cleanup()

If you need to add handling for another type of exception, you have to
totally restructure your code. It's no longer a matter of adding
another except statement to the chain.

Also, what do you do if you need to use the exception instance? Are
you proposing (by implication) the additional syntax "if
check_status() raises IOError as foo:"? What If I want multiple
statements to be guarded by the try block? Again, I have to switch
mechanisms.

> Also, instead of the inconvenient
>
>    self.assertRaises(ZeroDivisionError, lambda: 1/0)
>
> one could just write
>
>    assert 1/0 raises ZeroDivisionError

I rarely find myself only caring about the type of the exception. For
anything complex, I want to make some assertion about the
attributes/structure of the raised exception, for example, regexing
the exception's str() value. If I want to change from only caring
about the type of the exception (the syntax you propose above) to
caring about the contents/structure of the exception instance, I have
to completely change which mechanism I'm using.

With this proposal I can't gracefully go from doing something simple
to doing something complex. -1

Collin Winter


From collinw at gmail.com  Sat Aug 30 05:58:35 2008
From: collinw at gmail.com (Collin Winter)
Date: Fri, 29 Aug 2008 20:58:35 -0700
Subject: [Python-ideas] micro-threading PEP proposal (long) -- take 2!
In-Reply-To: <48B2E256.6080008@gmail.com>
References: <48B2E256.6080008@gmail.com>
Message-ID: <43aa6ff70808292058n6e9ca60ckbb26d41a7df3321@mail.gmail.com>

On Mon, Aug 25, 2008 at 9:48 AM, Bruce Frederiksen <dangyogi at gmail.com> wrote:
[snip]
> I don't imagine that this PEP represents an /easy/ way to solve this
> problem, but do imagine that it is the /right/ way to solve it.  Other
> similar proposals have been made in past years that looked at easier ways
> out.  These have all been rejected.  But I don't think that there are really
> any easy ways out that are robust solutions, and so I offer this one.  If I
> am wrong, and the reason that the prior proposals were rejected is due to a
> lack of need, rather than a lack of robustness, then this proposal should
> also be rejected.  This might be the case if, for example, all Python
> programs end up being unavoidably CPU bound so that micro-threading would
> provide little real benefit.

It all depends on what you're doing. If you're waiting on a lot of
RPCs to complete and doing light-weight operations to process the
responses, then you're probably fine with micro-threads (unless, of
course, those RPC responses are themselves pretty big and require a
lot of deserialization work, in which case, micro-threads will hurt
more than they help).

> Motivation
> ==========
>
> The popularity of the Twisted project has demonstrated the need for a
> micro-threading alternative to the standard Posix thread_ [#thread-module]_
> and threading_ [#threading-module]_ packages.

It in no way demonstrates that. I would say that popularity of Twisted
indicates that "a micro-threading alternative to the
standard...threading packages" can survive and indeed thrive outside
of the standard library. If you feel that Twisted's popularity does
indeed demonstrate something in this area, please back up that
assertion.

> Micro-threading allows large
> numbers (1000's) of simultaneous connections to Python servers, as well
> as fan-outs to large numbers of downstream connections.
>
> The advantages to the Twisted approach over Posix threads are:
>
> #. much less memory is required per thread
> #. faster thread creation
> #. faster context switching (I'm guessing on this one, is this really true?)

That you don't know is, frankly, not reassuring.

> #. synchronization between threads is easier because there is no preemption,
>  making it much easier to write critical sections of code.
>
> The disadvantages are:
>
> #. the Python developer must write his/her program in an event driven style
> #. the approach can not be used with standard Python code that wasn't
>  written in this event driven style
> #. the approach does not take advantage of multiple processor architectures
> #. since there is no preemption, a long running micro-thread will starve
>  other micro-threads

By long-running, you mean "non-yielding", right? Don't CPU-intensive
operations generally fall into this category? Combined with the first
two disadvantages, this means that a developer using this system has
to vet all libraries they might want to use (and all libraries in
their transitive dependency closure), looking for places that might
destabilize the ability of micro-threads to cooperatively yield. That
sounds like an incredibly error-prone and painstaking waste of
developer time.

> This PEP attempts to retain all of the advantages that Twisted has
> demonstrated,

Please don't assume that everyone reading your PEP is familiar with Twisted.

> and to resolve the first two disadvantages to make the
> advantages accessible to all Python programs, including legacy programs
> not written in the Twisted style.  This should make it very easy for legacy
> programs like WSGI apps, Django and TurboGears to reap the benefits of
> Twisted.

So you say, but I see nothing in this entire PEP (and I'll freely
admit I started skimming it after page five or so) that specifically
references these disadvantages or demonstrates how they're being
solved.

> This PEP does not address the last two disadvantages, and thus also has
> these disadvantages itself.

Starvation is a pretty big disadvantage to simply gloss over.

>  In addition, the current built-in ``iter`` and ``next`` functions would be
>  modified so that they may be called with no arguments.  In this case, they
>  would use the current micro_thread's *stdin* pipe as their argument.

I don't understand this. Please explain in more detail why adding this
new (and unexpected) functionality to iter() and next() is desirable
as opposed to adding new functions/methods.

> C Deferred
> ----------
>
> ``PyDeferred_CDeferred`` is written as a new exception type for use by the
> C code to defer execution.  This is a subclass of ``NotImplementedError``.

Why is this a subclass of NotImplementedError and not a direct
subclass of Exception? This is an odd choice of parent class.

> Instances are not raised as a normal exception (e.g., with
> ``PyErr_SetObject``), but by calling ``PyNotifier_Defer`` (described in the
> Notifier_ section, below).  This registers the ``PyDeferred_CDeferred``
> associated with the currently running micro_thread as the current error
> object,
> but also readies it for its primary job -- deferring execution.  As an
> exception, it creates its own error message, if needed, which is
> "Deferred execution not yet implemented by %s" % c_function_name.

And what happens if I use PyErr_SetObject() instead of this new
function? Is a TypeError raised?

> ``PyErr_ExceptionMatches`` may be used with these.  This allows them to be
> treated as exceptions by non micro-threading aware (*unmodified*) C
> functions.

So it's possible for non-micro-threading aware code to simply swallow
these new exceptions? That seems...unwise.

Collin Winter


From josiah.carlson at gmail.com  Sat Aug 30 07:20:50 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Fri, 29 Aug 2008 22:20:50 -0700
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<d03afaa20808251841o1263125cv181664312495763e@mail.gmail.com>
	<87ljykb6uv.fsf@physik.rwth-aachen.de>
	<d03afaa20808261702q2e5ea67ja53c6dffed8a2686@mail.gmail.com>
Message-ID: <e6511dbf0808292220s1a9a4175gf029d92009e48cc0@mail.gmail.com>

On Tue, Aug 26, 2008 at 5:02 PM, Russ Paielli <russ.paielli at gmail.com> wrote:
> My editor, Xemacs, also highlights the word "self." I personally find that a
> bit annoying, and perhaps that is part of the reason I don't appreciate the
> word. When I write an algorithm, I don't want to see some artifact
> highlighted, as if it is somehow the most important part of the algorithm. I
> like seeing keywords highlighted, but I don't want to see "self" highlighted
> because it is only a convention. Maybe I'm alone on that.
>
> As I wrote earlier, I copied one of my larger Python files and replaced each
> occurrence of "self" with "S". That was approximately 350 occurrences. I
> also replaced each occurrence of "cls" with "C", which was another 50 or so
> occurrences. The file size was reduced by 2.8%, many wrapped lines no longer
> need to wrap, and I like the way it looks.
>
> I understand that people like to have "one true way," but I think Python
> programmers are adaptable enough to handle this one. When you read code, you
> certainly need to read the argument names anyway, so the fact that "S" is
> being used instead of "self" should be obvious. The only minor disadvantage
> I can think of is that searching on a single character can be problematic,
> but if you are using an IDE, that should not be an issue (and I can't
> remember ever searching on "self" anyway).
>
> One of these days, when I get a chance, I may write an informational PEP
> recommending the use of "S" as an acceptable replacement for "self", and "C"
> as an accptable replacement for "cls". I hope that doesn't blow any fuses.

You can submit anything you want.  You know what Guido is going to say
with regards to self -> S and cls -> C as a proposed change to the PEP
8 style guide?  Something along the lines of "no".  He may even be
that polite.

What you do in your own code (that you write and maintain) is your own
business.  If S/C makes you happy: use it.  But don't be surprised
if/when someone starts working on that same code they start asking
(perfectly valid) questions about why you chose to go against the PEP
8 style guide.

 - Josiah


From bruce at leapyear.org  Sat Aug 30 08:49:55 2008
From: bruce at leapyear.org (Bruce Leban)
Date: Fri, 29 Aug 2008 23:49:55 -0700
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <d11dcfba0808291917s5bf0a6d7jcf76bf2e046f0749@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com> <48B7F77F.1030303@gmx.net>
	<cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>
	<48B8377D.7060605@gmail.com>
	<7e69a0400808291347j2692f43cu9d7b07bc75b53835@mail.gmail.com>
	<cf5b87740808291725v4f0aa3b2u77fcbde5281df652@mail.gmail.com>
	<d11dcfba0808291917s5bf0a6d7jcf76bf2e046f0749@mail.gmail.com>
Message-ID: <cf5b87740808292349w522b2361ya76514b5da075b1@mail.gmail.com>

On Fri, Aug 29, 2008 at 7:17 PM, Steven Bethard <steven.bethard at gmail.com>wrote:

> On Fri, Aug 29, 2008 at 6:25 PM, Bruce Leban <bruce at leapyear.org> wrote:
> > Not quite: len(items) requires iterating across the entire list which may
> > not be necessary or desirable. Plus you may need an intermediate variable
> to
> > store it. I think that's inferior.
>
> If len(items) requires iterating across the entire list, then you're
> not using a Python list object, you're using someone else's (poorly
> implemented) list data structure. Python lists give len(obj) in O(1).
>

items need not be a literal list. If you can compute this
   len([t for t in range(100) if foo(t)]):
without iterating over the list, then there's a Turing award waiting for
you. :-) (Yes, if you know what foo is perhaps you can optimize that. I'm
not counting that.) This is also why you may need an intermediate variable:
you don't want to compute that list more than once because foo(t) may have a
side effect.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080829/02106cbd/attachment.html>

From carl at carlsensei.com  Sat Aug 30 11:48:10 2008
From: carl at carlsensei.com (Carl Johnson)
Date: Fri, 29 Aug 2008 23:48:10 -1000
Subject: [Python-ideas] Variations on a loop
Message-ID: <D44BB9BA-6C6F-49D8-BDBB-85A0688A9ACF@carlsensei.com>

Here's one way to test if the loop you just came out of was empty:

     for item in range(0):
         print item
     else:
         if "item" not in locals():
             print "Empty list."

Obviously, this only works if "item" has not been used in the current  
scope before. If anyone knows of a more efficient way to check if a  
variable name has been used besides using locals(), let me know.

Another way to do it is if you know that some particular value, such  
as None, will never appear in your list-like-object you can set item  
to that value in advance:

     item = None
     for item in range(0):
         print item
     else:
         if item is None:
             print "Empty list."

Again, this has the limitation that it only works if you can guaranty  
that your guard value won't be used by the loop. The advantage is that  
you don't have to create locals() and then do a __contains__ on it,  
which presumably takes longer than a simple assignment and identity  
check.

-- Carl Johnson


From g.brandl at gmx.net  Sat Aug 30 15:11:00 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 30 Aug 2008 15:11:00 +0200
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <cf5b87740808291725v4f0aa3b2u77fcbde5281df652@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>	<48B7F05C.6090405@gmail.com>
	<48B7F77F.1030303@gmx.net>	<cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>	<48B8377D.7060605@gmail.com>	<7e69a0400808291347j2692f43cu9d7b07bc75b53835@mail.gmail.com>
	<cf5b87740808291725v4f0aa3b2u77fcbde5281df652@mail.gmail.com>
Message-ID: <g9bgts$8l6$1@ger.gmane.org>

Bruce Leban schrieb:
> 
> On Fri, Aug 29, 2008 at 1:47 PM, David Blaschke
> <dwblas at gmail.com
> <mailto:dwblas at gmail.com>> wrote:
> 
>     This idea
>     elif x == funky_empty:
>           result = 'no data'
> 
>     Can be accomplished with
>     if not len(items):
>           result = 'no data'
>     which means that part of the proposal can be scratched IMHO.
> 
> 
> Not quite: len(items) requires iterating across the entire list which
> may not be necessary or desirable. Plus you may need an intermediate
> variable to store it. I think that's inferior.

This is moot anyway: in general, iterators have no __len__().

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.



From arnodel at googlemail.com  Sat Aug 30 17:38:52 2008
From: arnodel at googlemail.com (Arnaud Delobelle)
Date: Sat, 30 Aug 2008 16:38:52 +0100
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <48B7F05C.6090405@gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com>
Message-ID: <71DC8CCF-625A-4091-AE14-2ADAAEA2F4A7@googlemail.com>


On 29 Aug 2008, at 13:49, Bruce Frederiksen wrote:

> Bruce Leban wrote:
>>
>> For example:
>>
>>    result = ""
>>    for x in items:
>>        result += str(x)
>>    interstitially:
>>        result += ", "
>>    contrariwise:
>>        result = "no data"
> We already have statements that only apply within loops (break and  
> continue), how about some expressions that only apply in for loops:  
> 'last' and 'empty':
>
> for x in items:
>   result += str(x)
>   if not last:
>       result += ', '
> else:
>   if empty:
>       result = "no data"
>
> -bruce (not to be confused with Bruce :-)

note that this can be achieved without 'empty'

for x in items:
     result += str(x)
     if last:
         break
     else:
         result += ', '
else:
     result = "no data"

Something like this can be implemented in Python with the following  
helper function:

def flaglast(iterable):
     it = iter(iterable)
     for prev in it:
         break
     else:
         return
     for item in it:
         yield prev, False
         prev = item
     yield prev, True

E.g.

 >>> list(flaglast([1,2,3]))
[(1, False), (2, False), (3, True)]

Then you can write your code snippet as:

for x, last in flaglast(items):
     result += str(x)
     if last:
         break
     else:
         result += ', '
else:
     result = "no data"

-- 
Arnaud



From steven.bethard at gmail.com  Sat Aug 30 18:05:44 2008
From: steven.bethard at gmail.com (Steven Bethard)
Date: Sat, 30 Aug 2008 10:05:44 -0600
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <cf5b87740808292349w522b2361ya76514b5da075b1@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com> <48B7F77F.1030303@gmx.net>
	<cf5b87740808291026y77beeb2fy4c66aa999692361a@mail.gmail.com>
	<48B8377D.7060605@gmail.com>
	<7e69a0400808291347j2692f43cu9d7b07bc75b53835@mail.gmail.com>
	<cf5b87740808291725v4f0aa3b2u77fcbde5281df652@mail.gmail.com>
	<d11dcfba0808291917s5bf0a6d7jcf76bf2e046f0749@mail.gmail.com>
	<cf5b87740808292349w522b2361ya76514b5da075b1@mail.gmail.com>
Message-ID: <d11dcfba0808300905x297c5f5dkf9bc0d1b7c5b06d4@mail.gmail.com>

On Sat, Aug 30, 2008 at 12:49 AM, Bruce Leban <bruce at leapyear.org> wrote:
>
> On Fri, Aug 29, 2008 at 7:17 PM, Steven Bethard <steven.bethard at gmail.com>
> wrote:
>>
>> On Fri, Aug 29, 2008 at 6:25 PM, Bruce Leban <bruce at leapyear.org> wrote:
>> > Not quite: len(items) requires iterating across the entire list which
>> > may
>> > not be necessary or desirable. Plus you may need an intermediate
>> > variable to
>> > store it. I think that's inferior.
>>
>> If len(items) requires iterating across the entire list, then you're
>> not using a Python list object, you're using someone else's (poorly
>> implemented) list data structure. Python lists give len(obj) in O(1).
>
> items need not be a literal list. If you can compute this
>    len([t for t in range(100) if foo(t)]):
> without iterating over the list, then there's a Turing award waiting for
> you. :-)

See what you said above: "len(items) requires iterating across the
list". The list comprehension is iterating over the items, not the
call to len(). So I *can* compute "len(items)" without iterating over
the list.

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
 --- Bucky Katt, Get Fuzzy


From arnodel at googlemail.com  Sat Aug 30 19:56:01 2008
From: arnodel at googlemail.com (Arnaud Delobelle)
Date: Sat, 30 Aug 2008 18:56:01 +0100
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <71DC8CCF-625A-4091-AE14-2ADAAEA2F4A7@googlemail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com>
	<71DC8CCF-625A-4091-AE14-2ADAAEA2F4A7@googlemail.com>
Message-ID: <F977CA40-3D99-4892-BD54-117A02955CF2@googlemail.com>


On 30 Aug 2008, at 16:38, Arnaud Delobelle wrote:

>
> On 29 Aug 2008, at 13:49, Bruce Frederiksen wrote:
>
>> Bruce Leban wrote:
>>>
>>> For example:
>>>
>>>   result = ""
>>>   for x in items:
>>>       result += str(x)
>>>   interstitially:
>>>       result += ", "
>>>   contrariwise:
>>>       result = "no data"
>> We already have statements that only apply within loops (break and  
>> continue), how about some expressions that only apply in for loops:  
>> 'last' and 'empty':
>>
>> for x in items:
>>  result += str(x)
>>  if not last:
>>      result += ', '
>> else:
>>  if empty:
>>      result = "no data"
>>
>> -bruce (not to be confused with Bruce :-)
>
> note that this can be achieved without 'empty'
>
> for x in items:
>    result += str(x)
>    if last:
>        break
>    else:
>        result += ', '
> else:
>    result = "no data"
>
> Something like this can be implemented in Python with the following  
> helper function:
>
> def flaglast(iterable):
>    it = iter(iterable)
>    for prev in it:
>        break
>    else:
>        return
>    for item in it:
>        yield prev, False
>        prev = item
>    yield prev, True
>
> E.g.
>
> >>> list(flaglast([1,2,3]))
> [(1, False), (2, False), (3, True)]
>
> Then you can write your code snippet as:
>
> for x, last in flaglast(items):
>    result += str(x)
>    if last:
>        break
>    else:
>        result += ', '
> else:
>    result = "no data"
>
> -- 
> Arnaud
>

Also having a 'last' local to a loop implies a lot of overhead in case  
one is looping over an iterable whose length is not known - in essence  
the overhead in my flaglast function.

Finally, my function works gracefully with nested loops:

for i, i_last in flaglast(items):
     for j, j_last in flaglast(jtems):
         ...

without the need for a 'named loop', about which I feel very  
circumspect.

-- 
Arnaud



From bruce at leapyear.org  Sat Aug 30 20:32:35 2008
From: bruce at leapyear.org (Bruce Leban)
Date: Sat, 30 Aug 2008 11:32:35 -0700
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <F977CA40-3D99-4892-BD54-117A02955CF2@googlemail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com>
	<71DC8CCF-625A-4091-AE14-2ADAAEA2F4A7@googlemail.com>
	<F977CA40-3D99-4892-BD54-117A02955CF2@googlemail.com>
Message-ID: <cf5b87740808301132k2514a741x7833c0abcefe780a@mail.gmail.com>

On Sat, Aug 30, 2008 at 10:56 AM, Arnaud Delobelle
<arnodel at googlemail.com>wrote:

>
> Finally, my function works gracefully with nested loops:
>
> for i, i_last in flaglast(items):
>    for j, j_last in flaglast(jtems):
>        ...
>
> without the need for a 'named loop', about which I feel very circumspect.


Unfortunately, your function has to get the N+1st item before the body
processes the Nth element so if getting the elements has side effects and
the body might break, then this changes the behavior of the loop. With an
explicit call to an <is this the last> function, we know exactly when it
will peek ahead in the list.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080830/245077fd/attachment.html>

From arnodel at googlemail.com  Sat Aug 30 20:59:38 2008
From: arnodel at googlemail.com (Arnaud Delobelle)
Date: Sat, 30 Aug 2008 19:59:38 +0100
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <cf5b87740808301132k2514a741x7833c0abcefe780a@mail.gmail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com>
	<71DC8CCF-625A-4091-AE14-2ADAAEA2F4A7@googlemail.com>
	<F977CA40-3D99-4892-BD54-117A02955CF2@googlemail.com>
	<cf5b87740808301132k2514a741x7833c0abcefe780a@mail.gmail.com>
Message-ID: <49CDADD6-357D-469B-882F-2604691CF4C9@googlemail.com>


On 30 Aug 2008, at 19:32, Bruce Leban wrote:

>
>
> On Sat, Aug 30, 2008 at 10:56 AM, Arnaud Delobelle <arnodel at googlemail.com 
> > wrote:
>
> Finally, my function works gracefully with nested loops:
>
> for i, i_last in flaglast(items):
>    for j, j_last in flaglast(jtems):
>        ...
>
> without the need for a 'named loop', about which I feel very  
> circumspect.
>
> Unfortunately, your function has to get the N+1st item before the  
> body processes the Nth element so if getting the elements has side  
> effects and the body might break, then this changes the behavior of  
> the loop. With an explicit call to an <is this the last> function,  
> we know exactly when it will peek ahead in the list.
>
> --- Bruce

How do you propose to find out if the item is the last without getting  
the next one?

-- 
Arnaud



From arnodel at googlemail.com  Sat Aug 30 21:37:34 2008
From: arnodel at googlemail.com (Arnaud Delobelle)
Date: Sat, 30 Aug 2008 20:37:34 +0100
Subject: [Python-ideas] Variations on a loop
In-Reply-To: <49CDADD6-357D-469B-882F-2604691CF4C9@googlemail.com>
References: <cf5b87740808272207q19028e96h251febe984afc1e8@mail.gmail.com>
	<48B7F05C.6090405@gmail.com>
	<71DC8CCF-625A-4091-AE14-2ADAAEA2F4A7@googlemail.com>
	<F977CA40-3D99-4892-BD54-117A02955CF2@googlemail.com>
	<cf5b87740808301132k2514a741x7833c0abcefe780a@mail.gmail.com>
	<49CDADD6-357D-469B-882F-2604691CF4C9@googlemail.com>
Message-ID: <8C258CF8-7EE7-4B86-B89E-445D35125788@googlemail.com>


On 30 Aug 2008, at 19:59, Arnaud Delobelle wrote:

>
> On 30 Aug 2008, at 19:32, Bruce Leban wrote:
>
>>
>>
>> On Sat, Aug 30, 2008 at 10:56 AM, Arnaud Delobelle <arnodel at googlemail.com 
>> > wrote:
>>
>> Finally, my function works gracefully with nested loops:
>>
>> for i, i_last in flaglast(items):
>>   for j, j_last in flaglast(jtems):
>>       ...
>>
>> without the need for a 'named loop', about which I feel very  
>> circumspect.
>>
>> Unfortunately, your function has to get the N+1st item before the  
>> body processes the Nth element so if getting the elements has side  
>> effects and the body might break, then this changes the behavior of  
>> the loop. With an explicit call to an <is this the last> function,  
>> we know exactly when it will peek ahead in the list.
>>
>> --- Bruce
>
> How do you propose to find out if the item is the last without  
> getting the next one?
>
> -- 
> Arnaud
>

Sorry I hadn't read your reply closely enough.  I agree, but I was  
only reproducing the functionality of some code in the thread without  
the need for new syntax.

-- 
Arnaud



From jimjjewett at gmail.com  Sun Aug 31 23:57:39 2008
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sun, 31 Aug 2008 17:57:39 -0400
Subject: [Python-ideas] PEP: Shorthand Symbol for "self"
In-Reply-To: <g94k22$9rf$1@ger.gmane.org>
References: <d03afaa20808241543i39786790jcb67ee06413cde0d@mail.gmail.com>
	<fb6fbf560808241717s35dfeacbg5169b7ec00e7a896@mail.gmail.com>
	<g8t1on$7ne$1@ger.gmane.org>
	<cf5b87740808250045m1ffdb3f7he20e236a458157e@mail.gmail.com>
	<4c0fccce0808250335y46b7bbfbg50044815fba868bf@mail.gmail.com>
	<g8uk3r$9sf$1@ger.gmane.org>
	<fb6fbf560808271119qdcdac3bh4385bf25e0b12e2e@mail.gmail.com>
	<g94k22$9rf$1@ger.gmane.org>
Message-ID: <fb6fbf560808311457q797ed655s43e81a8b772b1082@mail.gmail.com>

On Wed, Aug 27, 2008 at 6:25 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> Jim Jewett schrieb:
>> On Mon, Aug 25, 2008 at 11:46 AM, Terry Reedy > 3.0 uses the standard
>> Unicode syntax for identifiers:
>>> "
>>> identifier  ::=  id_start id_continue*
>>>
>>> id_start    ::=  <all characters in general categories Lu, Ll, Lt, Lm, Lo,
>>> Nl, the underscore, and characters with the Other_ID_Start property>

>> Are you sure?

>> IIRC, the unicode consortion themselves recommend the xid_start and
>> xid_continue properties instead.

> I thought we were already using them.

PEP 3131 says we should.  Terry's email said we don't.  I can't find
the section of code that would tell me directly.

-jJ