From mwm at mired.org  Tue Nov  1 00:19:01 2011
From: mwm at mired.org (Mike Meyer)
Date: Mon, 31 Oct 2011 16:19:01 -0700
Subject: [Python-ideas] Concurrent safety?
In-Reply-To: <CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
Message-ID: <CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>

On Mon, Oct 31, 2011 at 3:58 PM, Bruce Leban <bruce at leapyear.org> wrote:

> On Sun, Oct 30, 2011 at 8:11 PM, Mike Meyer <mwm at mired.org> wrote:
>
>> Any attempt to mutate an object that isn't currently locked will raise
>> an exception. Possibly ValueError, possibly a new exception class just
>> for this purpose. This includes rebinding attributes of objects that
>> aren't locked.
>>
>
> Do you mean that at any time attempting to mutate an unlocked object
> throws an exception?


Yes, that's the idea.  There are some exceptions, but you have to
explicitly work around that restriction.


> That would mean that all of my current code is broken.


Pretty much, yes. It's like adding garbage collection and removing
alloc*/free. It's going to break a *lot* of code. That's why I said "not in
3. and possibly never in cPython."



> Do you mean, that inside the control of 'locking', you can't mutate an
> unlocked object? That still breaks lots of code that is safe. You can't use
> itertools.cycle anymore until that's updated in a completely unnecessary
> way:
>
> def cycle(iterable):
>     # cycle('ABCD') --> A B C D A B C D A B C D ...
>     saved = []
>     for element in iterable:
>         yield element
>         saved.append(element)  *# throws an exception when called on a locked iterable*
>     while saved:
>         for element in saved:
>               yield element
>
>
According to what I wrote, yes, it does.Since the list being mutated is
only visible inside the function, it doesn't need to be. It might be
possible to figure out that this is the case at compile time and thus allow
the code to run unmodified. But that's 1) hard, 2) will miss some cases, 3)
seems like a corner case. This proposal would break enough code that not
breaking this case doesn't seem to be worth the effort. That's a question
that needs to be answered.



> I think the semantics of this need to be tightened up.
>

That's why I brought it up. I'm trying to get more eyes on the issue.


> Furthermore, merely *reading* an object that isn't locked can cause
> problems. This code is not thread-safe:
>
>     if element in dictionary: return dictionary[element]
>
> so you have to decide how much safety you want and what cost we're willing
> to pay for this.
>

You're right - it's not thread safe. However, it also doesn't suffer from
the problem I'm trying to deal with, where you mutate an object in a way
that leaves things broken, but won't be detected at that point. If it
breaks because someone mutates the object underneath it, it'll throw an
exception at that point. I know you can construct cases where that isn't
so. Maybe we need two types of locking - one that allows readers, and one
that doesn't.  I could live with that, as you'd still have to consider the
issue where you mutate the object.

   <mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111031/549ea9a2/attachment.html>

From ron3200 at gmail.com  Tue Nov  1 01:22:57 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 31 Oct 2011 19:22:57 -0500
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <4EAF0DEE.1020500@canterbury.ac.nz>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<4EA94C53.2060209@pearwood.info>
	<20111027183208.GH20970@pantoffel-wg.de>
	<4EA9AB03.8070302@stoneleaf.us>
	<CAJ6cK1aUy76tY5Q6HPTPoBaJ2h4T2hv-NVSJw31BkvUt_FGOUA@mail.gmail.com>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
Message-ID: <1320106977.6637.50.camel@Gutsy>

On Tue, 2011-11-01 at 10:06 +1300, Greg Ewing wrote:
> Ron Adam wrote:
> > If we put some strict requirements on the idea.
> > 
> >     1. Only have a *SINGLE* exception type as being resumable.
> > 
> >     2. That exception should *NEVER* occur naturally.
> > 
> >     3. Only allow continuing after it's *EXPLICITLY RAISED* by a
> >        raised statement.
> > 
> > All of the problem issues go away with those requirements in place, and
> > you only have the issue of how to actually write the patch.  Earlier
> > discussions indicated, it might not be that hard to do.
> 
> I'm not familiar with these earlier discussions. Did they go
> as far as sketching a feasible implementation? It's all very
> well to propose things like this, but the devil is very much
> in the details.

Yeah, there isn't very much about the details, but I think it is worth
looking into as it would pretty much does exactly what is needed. (IMHO)

Here is some of the things I was able to find.  But as I said, the
discussions didn't get very far.  I'm hoping that the idea has more
merit in the smaller 'restricted' context of coroutines rather than for
general exception handling.

Here is a very old 1994 thread that is interesting reading. (Long but
worth while.)

http://groups.google.com/group/comp.lang.python/browse_thread/thread/674a821ed7003b69/2aa10cabcbcc4acb?q=python+continuations+restarts#2aa10cabcbcc4acb


There was a reference to allow 'raise' to send out a reference, rather
than an exception. That may be an interesting way to do this.


Some more that didn't go anywhere...

http://bytes.com/topic/python/answers/46053-resume-after-exception

http://bytes.com/topic/python/answers/36650-exception-feature-creep-entering-normal-flow-after-exception-raised

http://mail.python.org/pipermail/python-list/2010-December/1261919.html

I can't seem to find where I found the "It wouldn't be too hard to do
part.".  But if a final restriction of only working with generators at
first is added, it may make it easier as they can already be suspended.


Here is a python implementation for lisp style restarts.  I haven't
studied it yet, but it may show a way.  (I'm going to look at this in
more detail tonight.)

http://pypi.python.org/pypi/withrestart/0.2.6


Not sure if these would help, but they may be of interest on a more
theoretical level.

http://okmij.org/ftp/continuations/generators.html
http://lambda-the-ultimate.org/node/1544



Wikipedia has this on it's exception handling page ...

"""
Restarts separate mechanism from policy

Condition handling moreover provides a separation of mechanism from
policy. Restarts provide various possible mechanisms for recovering from
error, but do not select which mechanism is appropriate in a given
situation. That is the province of the condition handler, which (since
it is located in higher-level code) has access to a broader view.
"""

In the case of coroutines, the error's are the suspension points, and
the error handler is the scheduler that switches between them.  The
context is a bit different but I believe the concept is still
applicable.

Hope some of this is helpful.

Cheers,
   Ron
































From bruce at leapyear.org  Tue Nov  1 02:07:19 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Mon, 31 Oct 2011 18:07:19 -0700
Subject: [Python-ideas] Concurrent safety?
In-Reply-To: <CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
Message-ID: <CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>

On Mon, Oct 31, 2011 at 4:19 PM, Mike Meyer <mwm at mired.org> wrote:

> On Mon, Oct 31, 2011 at 3:58 PM, Bruce Leban <bruce at leapyear.org> wrote:
>
>> On Sun, Oct 30, 2011 at 8:11 PM, Mike Meyer <mwm at mired.org> wrote:
>>
>>> Any attempt to mutate an object that isn't currently locked will raise
>>> an exception. Possibly ValueError, possibly a new exception class just
>>> for this purpose. This includes rebinding attributes of objects that
>>> aren't locked.
>>>
>>
>> Do you mean that at any time attempting to mutate an unlocked object
>> throws an exception?
>
>
> Yes, that's the idea.  There are some exceptions, but you have to
> explicitly work around that restriction.
>
>
>> That would mean that all of my current code is broken.
>
>
> Pretty much, yes. It's like adding garbage collection and removing
> alloc*/free. It's going to break a *lot* of code. That's why I said "not in
> 3. and possibly never in cPython."
>

In order to make concurrent code slightly safer, you're going to break all
existing programs that don't use concurrency. That seems to me like a new
language, not Python. You've been on this list long enough to see the
attention that's paid to backward compatibility.


> Furthermore, merely *reading* an object that isn't locked can cause
>> problems. This code is not thread-safe:
>>
>>     if element in dictionary: return dictionary[element]
>>
>> so you have to decide how much safety you want and what cost we're
>> willing to pay for this.
>>
>
> You're right - it's not thread safe. However, it also doesn't suffer from
> the problem I'm trying to deal with, where you mutate an object in a way
> that leaves things broken, but won't be detected at that point. If it
> breaks because someone mutates the object underneath it, it'll throw an
> exception at that point. I know you can construct cases where that isn't
> so.
>

I think the cases where non-thread-safe code won't throw an exception are
numerous, for example, the equally trivial:

    if element not in dictionary: dictionary[element] = 0

heck even this is not safe:

    dictionary[element] +=1

If you're going to tackle thread safety, it should address more of the
problem. These bugs are in many ways worse than mutating "an object in a
way that leaves things broken, but won't be detected at that point." The
above bugs may *never* be detected. I've come across bugs like that that
were in code for many years before I found them (and I'm sure that's
happened to others on this list as well).

The first thing to do is identify the problems you want to solve and make
sure that the problems are well understood. Then design some solutions.
Starting with a bad solution to a fraction of the problem isn't a good
start.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111031/46e0dffe/attachment.html>

From ncoghlan at gmail.com  Tue Nov  1 02:14:44 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 1 Nov 2011 11:14:44 +1000
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
	protocol
In-Reply-To: <1320106977.6637.50.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<4EA94C53.2060209@pearwood.info>
	<20111027183208.GH20970@pantoffel-wg.de>
	<4EA9AB03.8070302@stoneleaf.us>
	<CAJ6cK1aUy76tY5Q6HPTPoBaJ2h4T2hv-NVSJw31BkvUt_FGOUA@mail.gmail.com>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
Message-ID: <CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>

On Tue, Nov 1, 2011 at 10:22 AM, Ron Adam <ron3200 at gmail.com> wrote:
> On Tue, 2011-11-01 at 10:06 +1300, Greg Ewing wrote:
>> Ron Adam wrote:
>> > If we put some strict requirements on the idea.
>> >
>> > ? ? 1. Only have a *SINGLE* exception type as being resumable.
>> >
>> > ? ? 2. That exception should *NEVER* occur naturally.
>> >
>> > ? ? 3. Only allow continuing after it's *EXPLICITLY RAISED* by a
>> > ? ? ? ?raised statement.
>> >
>> > All of the problem issues go away with those requirements in place, and
>> > you only have the issue of how to actually write the patch. ?Earlier
>> > discussions indicated, it might not be that hard to do.
>>
>> I'm not familiar with these earlier discussions. Did they go
>> as far as sketching a feasible implementation? It's all very
>> well to propose things like this, but the devil is very much
>> in the details.
>
> Yeah, there isn't very much about the details, but I think it is worth
> looking into as it would pretty much does exactly what is needed. (IMHO)

It gave me another thought on an existing utility worth exploring in
this context: pdb's post-mortem capabilities.

Now, those *don't* implement coroutines (when you do a postmortem, you
end up in an emulation of the eval loop, not the eval loop itself).
However, that exception instance *does* contain the full frame stack,
all the way down to where the exception was thrown. Figuring out what
hooks you would need in the core eval loop in order to reinstate an
exception's frame stack as the "real" frame stack might be an
interesting exercise.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From anacrolix at gmail.com  Tue Nov  1 04:47:53 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Tue, 1 Nov 2011 14:47:53 +1100
Subject: [Python-ideas] Concurrent safety?
In-Reply-To: <CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
Message-ID: <CAB4yi1NmhcBNwpF6TQhrP5MbAXNO2+W00saNgwUJZWo0fuHnVA@mail.gmail.com>

I wouldn't say it necessarily has to be a new language. All too many
new languages have arisen when the problem was actually with some
specific part of an implementation, or could have been fixed with a
careful iteration.

It might be slightly off-topic, but one of Python's strengths in my
eyes has been very careful incremental improvements. Prior to 2.5
Python was of no interest to me due to various concerns that have now
been fixed. Indeed the language is significantly more consistent and
usable than it was in the past.

The last thing I (and many others) would want to see is yet another
language created, because an an existing excellent language's
implementation couldn't keep up with the times, or surmount an
obstacle. Innovations like PyPy, multiprocessing, and the currently
debated coroutines are what give Python a chance in the future.

I might be beating a dead dog, but the last thing Python should do is
roll over and die because "it's too hard". Just wanted to point out
that language != implementation.

On Tue, Nov 1, 2011 at 12:07 PM, Bruce Leban <bruce at leapyear.org> wrote:
>
> On Mon, Oct 31, 2011 at 4:19 PM, Mike Meyer <mwm at mired.org> wrote:
>>
>> On Mon, Oct 31, 2011 at 3:58 PM, Bruce Leban <bruce at leapyear.org> wrote:
>>>
>>> On Sun, Oct 30, 2011 at 8:11 PM, Mike Meyer <mwm at mired.org> wrote:
>>>>
>>>> Any attempt to mutate an object that isn't currently locked will raise
>>>> an exception. Possibly ValueError, possibly a new exception class just
>>>> for this purpose. This includes rebinding attributes of objects that
>>>> aren't locked.
>>>
>>> Do you mean that at any time attempting to mutate an unlocked object
>>> throws an exception?
>>
>> Yes, that's the idea. ?There are some exceptions, but you have to
>> explicitly work around that restriction.
>>
>>>
>>> That would mean that all of my current code is broken.
>>
>> Pretty much, yes. It's like adding garbage collection and removing
>> alloc*/free. It's going to break a *lot* of code. That's why I said "not in
>> 3. and possibly never in cPython."
>
> In order to make concurrent code slightly safer, you're going to break all
> existing programs that don't use concurrency. That seems to me like a new
> language, not Python. You've been on this list long enough to see the
> attention that's paid to backward compatibility.
>
>>>
>>> Furthermore, merely *reading* an object that isn't locked can cause
>>> problems. This code is not thread-safe:
>>> ? ? if element in dictionary: return dictionary[element]
>>> so you have to decide how much safety you want and what cost we're
>>> willing to pay for this.
>>
>> You're right - it's not thread safe. However, it also doesn't suffer from
>> the problem I'm trying to deal with, where you mutate an object in a way
>> that leaves things broken, but won't be detected at that point. If it breaks
>> because someone mutates the object underneath it, it'll throw an exception
>> at that point. I know you can construct cases where that isn't so.
>
> I think the cases where non-thread-safe code won't throw an exception are
> numerous, for example, the equally trivial:
> ? ? if element not in dictionary: dictionary[element] = 0
> heck even this is not safe:
> ? ? dictionary[element] +=1
> If you're going to tackle thread safety, it should address more of the
> problem. These bugs are in many ways worse than mutating "an object in a way
> that leaves things broken, but won't be detected at that point." The above
> bugs may *never* be detected. I've come across bugs like that that were in
> code for many years before I found them (and I'm sure that's happened to
> others on this list as well).
> The first thing to do is identify the problems you want to solve and make
> sure that the problems are well understood. Then design some solutions.
> Starting with a bad solution to a fraction of the problem isn't a good
> start.
> --- Bruce
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


From ron3200 at gmail.com  Tue Nov  1 05:58:26 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 31 Oct 2011 23:58:26 -0500
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<4EA94C53.2060209@pearwood.info>
	<20111027183208.GH20970@pantoffel-wg.de>
	<4EA9AB03.8070302@stoneleaf.us>
	<CAJ6cK1aUy76tY5Q6HPTPoBaJ2h4T2hv-NVSJw31BkvUt_FGOUA@mail.gmail.com>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
Message-ID: <1320123506.9456.57.camel@Gutsy>

On Tue, 2011-11-01 at 11:14 +1000, Nick Coghlan wrote:
> On Tue, Nov 1, 2011 at 10:22 AM, Ron Adam <ron3200 at gmail.com> wrote:
> > On Tue, 2011-11-01 at 10:06 +1300, Greg Ewing wrote:
> >> Ron Adam wrote:
> >> > If we put some strict requirements on the idea.
> >> >
> >> >     1. Only have a *SINGLE* exception type as being resumable.
> >> >
> >> >     2. That exception should *NEVER* occur naturally.
> >> >
> >> >     3. Only allow continuing after it's *EXPLICITLY RAISED* by a
> >> >        raised statement.
> >> >
> >> > All of the problem issues go away with those requirements in place, and
> >> > you only have the issue of how to actually write the patch.  Earlier
> >> > discussions indicated, it might not be that hard to do.
> >>
> >> I'm not familiar with these earlier discussions. Did they go
> >> as far as sketching a feasible implementation? It's all very
> >> well to propose things like this, but the devil is very much
> >> in the details.
> >
> > Yeah, there isn't very much about the details, but I think it is worth
> > looking into as it would pretty much does exactly what is needed. (IMHO)
> 
> It gave me another thought on an existing utility worth exploring in
> this context: pdb's post-mortem capabilities.
> 
> Now, those *don't* implement coroutines (when you do a postmortem, you
> end up in an emulation of the eval loop, not the eval loop itself).
> However, that exception instance *does* contain the full frame stack,
> all the way down to where the exception was thrown. Figuring out what
> hooks you would need in the core eval loop in order to reinstate an
> exception's frame stack as the "real" frame stack might be an
> interesting exercise.

Poking around a bit, it looks like 'raise' does most of the work and the
exception is just an envelope for what ever 'raise' puts in it.  Is that
right?



I'd like to be able to make this work.

class Suspend:
    def __init__(self, source):
        self.source = source
        self.suspend = True

    def __next__(self):
	nonlocal self.suspend
        if self.suspend:
            self.suspend = False
            raise SuspendException
        self.suspend = True
        return next(self.source)

There are two issues with it...

The "self.suspend = False" doesn't seem to work. The __next__ seems to
get it's own copies of the attributes at the time the generator is
created.

And after the SuspendException is raised, a StopIteratoion is issued on
the next next() call.  The StopIteration is from the whole chain.  The
only reason the scheduler doesn't stop is it catches the
Suspendexception.

I want to be able to stick something like this in the generator chained
pipe example below.

Cheers,
   Ron



*This is broken down into finer steps than you would normally do in
order to test how it behaves.

"""
   Basic scheduler test -- co-pipes version
"""
from co_pipes import *

def Person(args):
    name, count = args
    p = Producer(lambda:name)           # call function each time
    p = Limit(p, count)                 # exit after count yields.
    p = Enumerate(p)                    # -> (n, data)
    #p = Suspend(p)                     # suspend doesn't work.
    p = Apply(p, "{0[0]}: {0[1]} ".format)
    p = Apply(p, print)                 # consumer
    for _ in p:
        yield                           # pull data from here.

def main(data):
    p = Source(data)               # take an iterable
    p = Apply(p, Person)           # apply a callable to data
    p = Collect(p)                 # collect items in a list
    p = Scheduler(p)               # take a list of generators.
    next(p)                        # start the scheduler.
    
if __name__ == "__main__":
    data = [("John", 2), ("Micheal", 3), ("Terry", 4)]
    main(data)


Prints...

1: John 
1: Micheal 
1: Terry 
2: John 
2: Micheal 
2: Terry 
3: Micheal 
3: Terry 
4: Terry 






From ron3200 at gmail.com  Tue Nov  1 06:10:46 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Tue, 01 Nov 2011 00:10:46 -0500
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <1320123506.9456.57.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<4EA94C53.2060209@pearwood.info>
	<20111027183208.GH20970@pantoffel-wg.de>
	<4EA9AB03.8070302@stoneleaf.us>
	<CAJ6cK1aUy76tY5Q6HPTPoBaJ2h4T2hv-NVSJw31BkvUt_FGOUA@mail.gmail.com>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy>
Message-ID: <1320124246.9456.62.camel@Gutsy>

On Mon, 2011-10-31 at 23:58 -0500, Ron Adam wrote:
> On Tue, 2011-11-01 at 11:14 +1000, Nick Coghlan wrote:
> > On Tue, Nov 1, 2011 at 10:22 AM, Ron Adam <ron3200 at gmail.com> wrote:
> > > On Tue, 2011-11-01 at 10:06 +1300, Greg Ewing wrote:
> > >> Ron Adam wrote:
> > >> > If we put some strict requirements on the idea.
> > >> >
> > >> >     1. Only have a *SINGLE* exception type as being resumable.
> > >> >
> > >> >     2. That exception should *NEVER* occur naturally.
> > >> >
> > >> >     3. Only allow continuing after it's *EXPLICITLY RAISED* by a
> > >> >        raised statement.
> > >> >
> > >> > All of the problem issues go away with those requirements in place, and
> > >> > you only have the issue of how to actually write the patch.  Earlier
> > >> > discussions indicated, it might not be that hard to do.
> > >>
> > >> I'm not familiar with these earlier discussions. Did they go
> > >> as far as sketching a feasible implementation? It's all very
> > >> well to propose things like this, but the devil is very much
> > >> in the details.
> > >
> > > Yeah, there isn't very much about the details, but I think it is worth
> > > looking into as it would pretty much does exactly what is needed. (IMHO)
> > 
> > It gave me another thought on an existing utility worth exploring in
> > this context: pdb's post-mortem capabilities.
> > 
> > Now, those *don't* implement coroutines (when you do a postmortem, you
> > end up in an emulation of the eval loop, not the eval loop itself).
> > However, that exception instance *does* contain the full frame stack,
> > all the way down to where the exception was thrown. Figuring out what
> > hooks you would need in the core eval loop in order to reinstate an
> > exception's frame stack as the "real" frame stack might be an
> > interesting exercise.
> 
> Poking around a bit, it looks like 'raise' does most of the work and the
> exception is just an envelope for what ever 'raise' puts in it.  Is that
> right?
> 
> 
> 
> I'd like to be able to make this work.
> 
> class Suspend:
>     def __init__(self, source):
>         self.source = source
>         self.suspend = True
> 
>     def __next__(self):
> 	nonlocal self.suspend
>         if self.suspend:
>             self.suspend = False
>             raise SuspendException
>         self.suspend = True
>         return next(self.source)

LOL... Need to recheck my cut and copy between edits.

Remove the nonlocal self.suspend.  It was just to see what if anything
acted different and I forgot to remove it.

> There are two issues with it...
> 
> The "self.suspend = False" doesn't seem to work. The __next__ seems to
> get it's own copies of the attributes at the time the generator is
> created.

Ok, the self.suspend reference does work as it should.  I was just not
putting my print statement in the right place.  Time to call it a night.

The rest is ok.

Cheers,
   Ron


> And after the SuspendException is raised, a StopIteratoion is issued on
> the next next() call.  The StopIteration is from the whole chain.  The
> only reason the scheduler doesn't stop is it catches the
> Suspendexception.
> 
> I want to be able to stick something like this in the generator chained
> pipe example below.
> 
> Cheers,
>    Ron
> 
> 
> 
> *This is broken down into finer steps than you would normally do in
> order to test how it behaves.
> 
> """
>    Basic scheduler test -- co-pipes version
> """
> from co_pipes import *
> 
> def Person(args):
>     name, count = args
>     p = Producer(lambda:name)           # call function each time
>     p = Limit(p, count)                 # stop after count yields.
>     p = Enumerate(p)                    # -> (n, data)
>     #p = Suspend(p)                     # suspend doesn't work.
>     p = Apply(p, "{0[0]}: {0[1]} ".format)
>     p = Apply(p, print)                 # consumer
>     for _ in p:
>         yield                           # pull data from here.
> 
> def main(data):
>     p = Source(data)               # take an iterable
>     p = Apply(p, Person)           # apply a callable to data
>     p = Collect(p)                 # collect items in a list
>     p = Scheduler(p)               # take a list of generators.
>     next(p)                        # start the scheduler.
>     
> if __name__ == "__main__":
>     data = [("John", 2), ("Micheal", 3), ("Terry", 4)]
>     main(data)
> 
> 
> Prints...
> 
> 1: John 
> 1: Micheal 
> 1: Terry 
> 2: John 
> 2: Micheal 
> 2: Terry 
> 3: Micheal 
> 3: Terry 
> 4: Terry 
> 
> 
> 
> 




From mwm at mired.org  Tue Nov  1 06:32:59 2011
From: mwm at mired.org (Mike Meyer)
Date: Mon, 31 Oct 2011 22:32:59 -0700
Subject: [Python-ideas] Fwd:  Concurrent safety?
In-Reply-To: <20111031223048.6e5d2798@bhuda.mired.org>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
Message-ID: <CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>

Sigh. Sent this only to bruce by accident. Sorry about the duplicates Bruce.

---------- Forwarded message ----------
From: Mike Meyer <mwm at mired.org>
Date: Mon, Oct 31, 2011 at 10:30 PM
Subject: Re: [Python-ideas] Concurrent safety?
To: Bruce Leban <bruce at leapyear.org>


On Mon, 31 Oct 2011 18:07:19 -0700
Bruce Leban <bruce at leapyear.org> wrote:
> On Mon, Oct 31, 2011 at 4:19 PM, Mike Meyer <mwm at mired.org> wrote:
> >> That would mean that all of my current code is broken.
> > Pretty much, yes. It's like adding garbage collection and removing
> > alloc*/free. It's going to break a *lot* of code. That's why I said
"not in
> > 3. and possibly never in cPython."
> In order to make concurrent code slightly safer, you're going to break all
> existing programs that don't use concurrency. That seems to me like a new
> language, not Python. You've been on this list long enough to see the
> attention that's paid to backward compatibility.

The way adding automatic memory management just made pointer
arithmetic slightly safer.

But yeah, the first thing I said was "never in 3.x, possibly never in
cPython." I've been on this list long enough to know that, while the
community pays a lot of attention to backwards compatibility, it is
willing to throw it out when there's enough benefit.

As you point out, this is a hard problem. I know I haven't covered all
the issues. That's why the second thing I said was that I'm hoping to
get people smarter than me to look at things.

The cpu manufacturers have switched to improving performance by adding
more cores instead of cranking clock speeds. As time goes by, more and
more programmers are going to want to leverage that in serious ways. I
already do, and find that Python makes some design choices unavailable
or very fragile(*). I'd like to make those choices available, and help
Python get ready for the time when that desire is the norm instead of
the exception.

> > Furthermore, merely *reading* an object that isn't locked can cause
> >> problems. This code is not thread-safe:
> >>     if element in dictionary: return dictionary[element]
> >> so you have to decide how much safety you want and what cost we're
> >> willing to pay for this.
> > You're right - it's not thread safe. However, it also doesn't suffer
from
> > the problem I'm trying to deal with, where you mutate an object in a way
> > that leaves things broken, but won't be detected at that point. If it
> > breaks because someone mutates the object underneath it, it'll throw an
> > exception at that point. I know you can construct cases where that isn't
> > so.
> I think the cases where non-thread-safe code won't throw an exception are
> numerous, for example, the equally trivial:

Again, I said such cases can be built. I *didn't* say they were
exceptions, I proposed a change to deal with them.

> If you're going to tackle thread safety, it should address more of the
> problem. These bugs are in many ways worse than mutating "an object in a
> way that leaves things broken, but won't be detected at that point." The
> above bugs may *never* be detected. I've come across bugs like that that
> were in code for many years before I found them (and I'm sure that's
> happened to others on this list as well).

Like me. That's part of why I want to get the interpreter to help find
them.

> The first thing to do is identify the problems you want to solve and make
> sure that the problems are well understood. Then design some solutions.
> Starting with a bad solution to a fraction of the problem isn't a good
> start.

I've identified the problem I want to solve: I want to make concurrent
use of python objects "safe by default", so that doing unsafe things
causes the programmer to have to do something explicit about making
things safe.  I believe this can be done at the mutation points
(again, clojure shows that it can be done). I also want to preserve as
much of Python's existing code as possible. It may be that Python's
existing data structures mean my believe about mutation points is
wrong. This may be the wrong solution. It may be that such a change is
to large to be acceptable.  But the only way to find out is to
investigate it.

This discussion has already generated significant changes in the
original proposal, plus some implementation ideas.

  <mike

*) I'll note that some of the alternatives make the choices available
in Python unavailable in a similar way.
--
Mike Meyer <mwm at mired.org>              http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111031/30ed1676/attachment.html>

From greg.ewing at canterbury.ac.nz  Tue Nov  1 07:00:15 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 01 Nov 2011 19:00:15 +1300
Subject: [Python-ideas] Concurrent safety?
In-Reply-To: <CAD=7U2BvbDWt9YWkqBwmuOt7++RKJTqe0OYF+=-zLRR=u4_rUg@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CA+OGgf5VSEuwHP39cM-wnyhJ6hoKVLOSwYEhTb+Q=d10ATshsg@mail.gmail.com>
	<CAD=7U2BvbDWt9YWkqBwmuOt7++RKJTqe0OYF+=-zLRR=u4_rUg@mail.gmail.com>
Message-ID: <4EAF8AEF.5050804@canterbury.ac.nz>

Mike Meyer wrote:
> The goal here is to move from where we are to a place similar to where 
> handling files is, so that failing to properly deal with the possibility 
> of concurrent access causes an error when it happens, not at a point 
> distant in both time and space.

I don't think what you're suggesting would achieve this,
though. The locking required for correctness often involves
more than one object or more than one operation on an
object. Consider

    new_balance = balance + deposit
    lock(balance)
    balance = new_balance
    unlock(balance)

This wouldn't trigger any of your alarms, but it would
still be wrong.

-- 
Greg


From tjreedy at udel.edu  Tue Nov  1 07:27:55 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 01 Nov 2011 02:27:55 -0400
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
	protocol
In-Reply-To: <1320123506.9456.57.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<20111027183208.GH20970@pantoffel-wg.de>
	<4EA9AB03.8070302@stoneleaf.us>
	<CAJ6cK1aUy76tY5Q6HPTPoBaJ2h4T2hv-NVSJw31BkvUt_FGOUA@mail.gmail.com>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy>
Message-ID: <j8o3hk$afm$1@dough.gmane.org>

On 11/1/2011 12:58 AM, Ron Adam wrote:

> Poking around a bit, it looks like 'raise' does most of the work and the
> exception is just an envelope for what ever 'raise' puts in it.  Is that
> right?

I believe raise just instantiates the indicated exception. I expect that 
Exception.__new__ or .__init__ captures the traceback info. Subclasses 
can add more. A SuspendExecution exception should be able to grab as 
much as is needed for a resume. A CAPI call could be added if needed.

I hope you keep looking at this idea. Function calls stop execution and 
pass control 'down', to be resumed by return. yield stops execution and 
passes control 'up', to be resumed by next (or .send). Exceptions pass 
control 'up' (or 'out') without the possibility of resuming. All that is 
lacking is something to suspend and pass control 'sideways', to a 
specific target. A special exception makes some sense in that exceptions 
already get the call stack needed to resume after suspension.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Tue Nov  1 07:55:23 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 01 Nov 2011 02:55:23 -0400
Subject: [Python-ideas] Fwd:  Concurrent safety?
In-Reply-To: <CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
Message-ID: <j8o555$i65$1@dough.gmane.org>

On 11/1/2011 1:32 AM, Mike Meyer wrote:

> As you point out, this is a hard problem. I know I haven't covered all
> the issues. That's why the second thing I said was that I'm hoping to
> get people smarter than me to look at things.

This is one of the hard problems that keep getting swept under the rug 
while we do easier things. Well, we have overhauled unicode and 
packaging for 3.3, so maybe concurrency can get some attention.

I keep thinking that CPython's design of allowing C coded modules either 
outside or inside the stdlib should allow some progress.

Would it be helpful, for instance, to have truly immutable restricted 
tuples and frozensets, whose __new__ methods only allowed true 
immutables (None, booleans, numbers, strings, other restricted tuples 
and frozensets) as members?

How about a metaclass, say 'immutable', that made the instances of a 
user class truly immutable? (I don't know how to do this, but lets 
assume it can be done -- perhaps with a new frozendict.) If such were 
possible, instances of instances of such a metaclass could be added to 
the list above.

Could a metaclass automatically add fine-grained locks around around 
attribute mutations?

-- 
Terry Jan Reedy



From stephen at xemacs.org  Tue Nov  1 09:01:19 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 01 Nov 2011 17:01:19 +0900
Subject: [Python-ideas]  Fwd:  Concurrent safety?
In-Reply-To: <CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
Message-ID: <87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>

Mike Meyer writes:

 > The cpu manufacturers have switched to improving performance by adding
 > more cores instead of cranking clock speeds. As time goes by, more and
 > more programmers are going to want to leverage that in serious
 > ways.

Please, everybody is aware of that.  Anybody against improved support
for concurrency is probably in favor of criminalizing motherhood and
apple pie, too.  What you need to justify is the apparently expensive
approach you propose, and you need to resolve the apparent
contradiction between that expense and the basic argument for threads
which is precisely how expensive they *aren't*.

 > I've identified the problem I want to solve: I want to make
 > concurrent use of python objects "safe by default",

But that's not what you've proposed, AIUI.  You've proposed making
concurrent use *safer*, but not yet *safe*.  That's quite different
from the analogy with automatic memory management, where the
programmer can't do anything dangerous with pointers (because they
can't do anything at all).  The analogous model for concurrency is
processes, it seems to me.  (I don't have a desperate need for high-
performance concurrency, so I take no position on processes + message
passing vs. threads + shared resources.)

 > so that doing unsafe things causes the programmer to have to do
 > something explicit about making things safe.

This is un-Pythonic, IMO[1].  Python generally permits dangerous (and
even ugly) things when done by "consenting adults", on the theory that
the programmer knows more about her problem than Python does.  It
seems to me that a more Pythonic approach to this would be to provide
something like STM as a metaclass, mixin class, or decorator.  (Don't
ask me how.)

 > I believe this can be done at the mutation points (again, clojure
 > shows that it can be done).

But clojure is a Lisp-derived language.  Lisp was designed as a pure
functional language (although AFAIK it pretty much immediately
acquired "set"), and a very large number of Lisp algorithms are
designed around conses which are (like Python tuples) basically
immutable (yes, I know about setcar and setcdr, but use of those
functions is generally considered a bug).  Whether that orientation
toward immutable objects continues in Clojure I don't know, but if it
does, the problem of designing a "write barrier" for mutations may be
(a) simpler and (b) have less performance impact than the analogous
task applied to Python.

While Python-the-language does have some immutable objects like tuples
and strings, it's really kind of hard to avoid use of containers like
lists and dictionaries and classes with mutable objects.


Footnotes: 
[1]  But I've been clobbered for expressing my opinion on Pythonicity
in the past, so don't put too much weight on that.<wink/>



From ncoghlan at gmail.com  Tue Nov  1 09:15:47 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 1 Nov 2011 18:15:47 +1000
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
	protocol
In-Reply-To: <j8o3hk$afm$1@dough.gmane.org>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<20111027183208.GH20970@pantoffel-wg.de>
	<4EA9AB03.8070302@stoneleaf.us>
	<CAJ6cK1aUy76tY5Q6HPTPoBaJ2h4T2hv-NVSJw31BkvUt_FGOUA@mail.gmail.com>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
Message-ID: <CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>

On Tue, Nov 1, 2011 at 4:27 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> I believe raise just instantiates the indicated exception. I expect that
> Exception.__new__ or .__init__ captures the traceback info. Subclasses can
> add more. A SuspendExecution exception should be able to grab as much as is
> needed for a resume. A CAPI call could be added if needed.

No, the traceback info is added by the eval loop itself. Remember that
when you raise an exception *type* (rather than an instance), the
exception doesn't get instantiated until it gets caught somewhere -
the eval loop maintains the unwinding stack for the traceback as part
of the thread state until it is time to attach it to the exception
object.

This is all at the tail end of the eval loop in CPython, but be warned
it's fairly brain bending stuff that depends on various internal
details of the eval loop:
http://hg.python.org/cpython/file/default/Python/ceval.c#l2879

> I hope you keep looking at this idea. Function calls stop execution and pass
> control 'down', to be resumed by return. yield stops execution and passes
> control 'up', to be resumed by next (or .send). Exceptions pass control 'up'
> (or 'out') without the possibility of resuming. All that is lacking is
> something to suspend and pass control 'sideways', to a specific target. A
> special exception makes some sense in that exceptions already get the call
> stack needed to resume after suspension.

That's not actually true - due to the need to process exception
handling clauses and finally blocks (including the implicit ones
inside with statements), the internal state of those frames is
potentially no longer valid for resumption (they've moved on beyond
the point where the internal function was called).

I'll also note that it isn't necessary to pass control sideways, since
there are two different flavours of coroutine design (the PDF article
in the other thread describes this well). The Lua version is
"asymmetric coroutines", and they only allow you to return to the
point that first invoked the coroutine (this model is a fairly close
fit with Python's generators and exception handling). The greenlet
version is "symmetric" coroutines, and those let you switch directly
to any other coroutine.

Both models have their pros and cons, but the main advantage of
asymmetric coroutines is that you can just say "suspend this thread"
without having to say *where* you want to switch to. Of course, you
can implement much the same API with symmetric coroutines as well, so
long as you can look up your parent coroutine easily. Ultimately, I
expect the symmetric vs asymmetric decision will be driven more by
implementation details than by philosophical preferences one way or
the other.

I will note that Ron's suggestion to leverage the existing eval loop
stack collection provided by the exception handling machinery does
heavily favour the asymmetric approach. Having a quick look to refresh
my memory of some of the details of CPython's exception handling, I've
come to the following tentative conclusions:

- an ordinary exception won't do, since you don't want to trigger
except and finally blocks in outer frames (ceval.c#2903)
- in CPython, a new "why = WHY_SUSPEND" at the eval loop layer is
likely a better approach, since it would allow the frame stack to be
collected without triggering exception handling
- the stack unwinding would then end when a "SETUP_COCALL" block was
encountered on the block stack (just as SETUP_EXCEPT and SETUP_FINALLY
can stop the stack unwinding following an exception
- with the block stacks within the individual frames preserved, the
collected stack should be in a fit state for later restoration
- the "fast_yield" code and the generator resumption code should also
provide useful insight

There's nothing too magical there - once we disclaim the ability to
suspend coroutines while inside a C function (even one that has called
back in via the C/Python API), it should boil down to a combination of
the existing mechanics for generators and exception handling. So, even
though the above description is (highly) CPython specific, it should
be feasible for other implementations to come up with something
similar (although perhaps not easy:
http://lua-users.org/lists/lua-l/2007-07/msg00002.html).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Tue Nov  1 09:31:24 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 1 Nov 2011 18:31:24 +1000
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7cu-BPOuP=JbRGStWH7M40yNW3o1WokammOqSr+Q5kS3w@mail.gmail.com>

On Tue, Nov 1, 2011 at 6:01 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> ?> I've identified the problem I want to solve: I want to make
> ?> concurrent use of python objects "safe by default",
>
> But that's not what you've proposed, AIUI. ?You've proposed making
> concurrent use *safer*, but not yet *safe*. ?That's quite different
> from the analogy with automatic memory management, where the
> programmer can't do anything dangerous with pointers (because they
> can't do anything at all). ?The analogous model for concurrency is
> processes, it seems to me. ?(I don't have a desperate need for high-
> performance concurrency, so I take no position on processes + message
> passing vs. threads + shared resources.)

Guido and python-dev in general *have* effectively taken a position on
that, though (mainly due to Global Interpreter Lock discussions).

1. Even for threads, the recommended approach is to use queue.Queue to
avoid the common concurrency issues (such as race conditions and
deadlock) associated with explicit locking
2. In Python 3, concurrent.futures offers an even *safer* interface
and higher level interface for many concurrent workloads
3. If you use multiple processes and serialised messages, or higher
level APIs like concurrent.futures, you can not only scale to multiple
cores, but also to multiple *machines*.

This has led to a quite deserved reputation for being intolerant of
changes that claim to make multithreaded development "better", but
only at the expense of making single-threaded development worse.
That's actually one of the more interesting aspects of PyPy's
experiments with software transactional memory - the sophistication of
their toolchain means that they can make the STM feature optional at
the translation stage, without resorting to ugly #ifdef hackery the
way we would need to in order to make such a feature similarly
optional in CPython.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From stephen at xemacs.org  Tue Nov  1 10:13:01 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 01 Nov 2011 18:13:01 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CADiSq7cu-BPOuP=JbRGStWH7M40yNW3o1WokammOqSr+Q5kS3w@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CADiSq7cu-BPOuP=JbRGStWH7M40yNW3o1WokammOqSr+Q5kS3w@mail.gmail.com>
Message-ID: <87hb2op5nm.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:
 > sjt wrote:

 > > (I don't have a desperate need for high-performance concurrency,
 > > so I take no position on processes + message passing vs. threads
 > > + shared resources.)
 > 
 > Guido and python-dev in general *have* effectively taken a position on
 > that, though (mainly due to Global Interpreter Lock discussions).

Sure, as a matter of "development politics" that's pretty clear.  I'm
sure Mike understands that, too.  (And is frustrated by it!)

My point is that Mike's approach of trying to make *everything* safe
for concurrency seems to point in the direction of process + message
passing, but I don't claim that this proves that processes are in any
sense technically superior, just that his approach needs justification
beyond "hey, we need to do something about concurrency in Python!"



From greg.ewing at canterbury.ac.nz  Tue Nov  1 11:24:44 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 01 Nov 2011 23:24:44 +1300
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
Message-ID: <4EAFC8EC.4080601@canterbury.ac.nz>

A Coroutine Protocol
====================

Here are some thoughts on the design of a new protocol to support lightweight
threads using a mechanism similar to, but distinct from, generators and
yield-from. Separating the two protocols will make it much easier to support
suspendable generators, something that is not possible using the cofunction
mechanism as currently specified in PEP 3152.

The protocol to be described is similar in many ways to the generator
protocol, and in what follows, analogies will be drawn between the two protocols
where it may aid understanding.


API
---

This section describes the outward appearance of the coroutine mechanism to
the programmer.

A coroutine is created using the following constructor:

::

     coroutine(f, *args, **kwds)

where ``f`` is an object obeying the "coroutine protocol" to be described
below. Syntactic support will be provided for creating such an object using
a special form of Python function definition, analogous to a generator.

The result is a "coroutine object" having the following methods:

``resume(value = None)``

     Resumes execution of the coroutine at the point where it was last
     suspended. The value, if any, is passed into the coroutine and
     becomes the return value of the operation that caused the suspension.
     The coroutine executes until its next suspension point, at which
     time the ``resume`` call returns with the value passed into the
     suspension operation.

     (Note: This is analogous to calling next() or send() on a generator-iterator.
     Suspension of a coroutine is analogous to a generator executing a
     ``yield`` operation.)

     If the coroutine has been freshly created, the passed-in value is
     ignored and the coroutine executes up to its first suspension point.

     If the top level of the coroutine finishes execution without
     encountering any further suspension points, a ``CoReturn`` exception
     is raised. This exception has a ``value`` attribute containing the
     return value from the coroutine.

     (Note: ``CoReturn`` is analogous to the ``StopIteration`` exception
     raised by an exhausted iterator or generator.)

``throw(exception)``

     Causes the given exception to be raised in the coroutine at its
     current suspension point.

``close()``

     Requests that the coroutine shut down and clean itself up. This is
     achieved by throwing in a ``CoExit`` exception (analogous to 
``GeneratorExit``).

It is expected that programmers will not write code that deals directly with
coroutine objects very often; rather, some kind of driver or scheduler will be
used that takes care of making ``resume()`` calls and handling ``CoReturn``
exceptions.


Cofunctions
-----------

There will be a special form of Python function called a "cofunction", defined
using the new keyword ``codef`` in place of ``def``. A cofunction provides a
convenient way of creating an object obeying the coroutine protocol. (This is
similar to how a generator provides a convenient way of creating an object
obeying the iterator protocol).

Suspension of a cofunction is achieved using the expression

::

     ``coyield`` [value]

This is analogous to a ``yield`` expression in a generator, and like ``yield``,
it can both provide and receive a value. However, unlike ``yield``, it is *not*
restricted to communicating with the immediate caller. It communicates directly
with the ``resume`` method of the coroutine, however deep the nesting of calls
is between the ``resume`` call and the ``coyield``.

There are some restrictions, however:

* A ``coyield`` is only allowed in the body of a cofunction (a function defined
with ``codef``), not in any other context.

* A cofunction can only be called from the body of another cofunction, not in
any other context.

Exceptions are raised if any of these restrictions are violated.

As a consequence, there must be an unbroken chain of cofunctions (or other objects
obeying the cofunction protocol, see below) making up the call stack from the
``resume`` method down to the suspension point. A cofunction may call an ordinary
function, but that function or anything called by it will not be able to suspend
the coroutine.

Note that the class of "ordinary functions" includes most functions and methods
written in C. However, it is possible for an object implemented in C to participate
in a coroutine stack by implementing the coroutine protocol below explicitly.


Coroutine Protocol
------------------

As well as the coroutine object, the coroutine protocol involves three other kinds
of objects, "cocallable objects", "coframe objects" and "coiterator objects".

A cocallable object has the following method:

``__cocall__(*args, **kwds)``

     Initiates a suspendable computation. Returns a coframe object.

     (This is analogous to the __iter__ method of an iterable object.)

     May return NotImplemented to signal that the object does not support the
     coroutine protocol. This enables wrapper objects such as bound methods to
     reflect whether or not the wrapped object supports the coroutine protocol.

A coframe object has the following methods:

``__resume__(costack, value)``

     There are two purposes for which this method is called: to continue
     execution from a suspension point, and to pass in the return value resulting
     from a nested call to another cocallable object.

     In both cases, the ``resume`` method is expected to continue execution until
     the next suspension point, and return the value produced by it. If the
     computation finishes before reaching another suspension point,
     ``CoReturn(retval)`` must be raised, where ``retval`` is the return value of
     the computation.

     (This method is analogous to the __send__ method of a generator-iterator.
     With a value of None, it is analogous to the __next__ method of an iterator.)

     The currently-executing coroutine object is passed in as the ``costack``
     parameter. The ``__resume__`` method can make a nested call to another 
cocallable
     object ``sub`` by performing:

         ``return costack.call(sub, *args, **kwds)``

     No further calls to this coframe will be made until ``obj`` finishes. When
     it does, the ``__resume__`` method of this coframe  is called with the
     return value from ``sub``.

     It is the responsibility of the coframe object to keep track of whether the
     previous call to its ``__resume__`` method resulted in a suspension or a nested
     call, and make use of the ``value`` parameter accordingly.

``__throw__(costack, exception)``

     Called to throw an exception into the computation. The coframe may choose to
     absorb the exception and continue executing, in which case ``__throw__`` should
     return the value produced by the next exception point or raise ``CoReturn`` as
     for ``__resume__``. Alternatively it may allow the same or a different 
exception
     to propagate out.

     Implementation of this method is optional. If it is not present, the behaviour
     is as if a trivial ``__throw__`` method were present that simply re-raises the
     exception.

A coiterator is an iterator that permits iteration to be carried out in a 
suspendable
manner. A coiterator object has the following method:

``__conext__()``

     Returns a coframe for computing the next item from the iteration. This is the
     coroutine equivalent of an iterator's ``__next__`` method, and behaves 
accordingly:
     its ``__resume__`` method must return an item by raising ``CoReturn(item)``. To
     finish the iteration, it raises ``StopIteration`` as usual.

To support coiteration, whenever a "next" operation is invoked by a cofunction
(whether implicitly by means of a for-loop or explicitly by calling ``next()``)
a ``__conext__`` method is first looked for, and if found, the operation is
carried out suspendably. Otherwise a normal call is made to the ``__next__``
method.


Formal Semantics
----------------

The semantics of the coroutine object are defined by the following Python 
implementation.

::

     class coroutine(object):

         #  Public methods

         def __init__(self, main, *args, **kwds):
             self._stack = []
             self._push(_cocall(main, *args, **kwds))

         def resume(self, value = None):
             return self._run(value, None)

         def throw(self, exc):
             return self._run(None, exc)

         def close(self):
             try:
                 self.throw(CoExit)
             except (CoExit, CoReturn):
                 pass

         def call(self, subroutine, *args, **kwds):
             meth = getattr(subroutine, '__cocall__', None)
             if meth is not None:
                 frame = meth(*args, **kwds)
                 if frame is not NotImplemented:
                     self._push(frame)
                     return self._run(None, None)
             return CoReturn(subroutine(*args, **kwds))

         #  Private methods

         def _run(self, value, exc):
             while True:
                 try:
                     frame = self._top()
                     if exc is None:
                         return frame.__resume__(self, value)
                     else:
                         meth = getattr(frame, '__throw__', None)
                         if meth is not None:
                             return meth(self, exc)
                         else:
                             raise exc
                 except BaseException as exc:
                     if self._pop():
                         if isinstance(exc, CoReturn):
                             value = exc.value
                             exc = None
                     else:
                         raise

         def _push(self, frame):
             self._stack.append(frame)

         def _pop(self):
             if len(self._stack) > 0:
                 del self._stack[-1]
                 return True
             else:
                 return False

         def _top(self):
             return self._stack[-1]

-- 
Greg



From mwm at mired.org  Tue Nov  1 16:25:08 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 1 Nov 2011 08:25:08 -0700
Subject: [Python-ideas] Concurrent safety?
In-Reply-To: <4EAF8AEF.5050804@canterbury.ac.nz>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CA+OGgf5VSEuwHP39cM-wnyhJ6hoKVLOSwYEhTb+Q=d10ATshsg@mail.gmail.com>
	<CAD=7U2BvbDWt9YWkqBwmuOt7++RKJTqe0OYF+=-zLRR=u4_rUg@mail.gmail.com>
	<4EAF8AEF.5050804@canterbury.ac.nz>
Message-ID: <CAD=7U2BfaLCZXkmmHvNXe5YdBj_c5eLmrCiSYTWJvs+1OZshkA@mail.gmail.com>

On Mon, Oct 31, 2011 at 11:00 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>wrote:

> Mike Meyer wrote:
>
>> The goal here is to move from where we are to a place similar to where
>> handling files is, so that failing to properly deal with the possibility of
>> concurrent access causes an error when it happens, not at a point distant
>> in both time and space.
>>
>
> I don't think what you're suggesting would achieve this,
> though. The locking required for correctness often involves
> more than one object or more than one operation on an
> object. Consider
>
>   new_balance = balance + deposit
>   lock(balance)
>   balance = new_balance
>   unlock(balance)
>
> This wouldn't trigger any of your alarms, but it would
> still be wrong.


You're right - I chose my words poorly. As stated, solving it would involve
solving the halting problem. Replace the word "properly" with "at all".
I.e. - if you don't think about a concurrent access and should have, it'll
cause an error. If you think about it and get it wrong - well, nothing will
prevent all bugs. Partially automated resource allocation doesn't prevent
the programmer from writing bad code, and this is in that category.

    <mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111101/937e6e11/attachment.html>

From mwm at mired.org  Tue Nov  1 16:38:20 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 1 Nov 2011 08:38:20 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>

On Tue, Nov 1, 2011 at 1:01 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:

> Mike Meyer writes:
>  > I've identified the problem I want to solve: I want to make
>  > concurrent use of python objects "safe by default",
> But that's not what you've proposed, AIUI.  You've proposed making
> concurrent use *safer*, but not yet *safe*.  That's quite different
> from the analogy with automatic memory management, where the
> programmer can't do anything dangerous with pointers (because they
> can't do anything at all).  The analogous model for concurrency is
> processes, it seems to me.  (I don't have a desperate need for high-
> performance concurrency, so I take no position on processes + message
> passing vs. threads + shared resources.)
>

No, the proposal does make things "safe by default". The default behavior
disallows all mutation. You have to do something explicit to allow it -
because "explicit is better than implicit."


>  > so that doing unsafe things causes the programmer to have to do
>  > something explicit about making things safe.
> This is un-Pythonic, IMO[1].  Python generally permits dangerous (and
> even ugly) things when done by "consenting adults", on the theory that
> the programmer knows more about her problem than Python does.  It
> seems to me that a more Pythonic approach to this would be to provide
> something like STM as a metaclass, mixin class, or decorator.  (Don't
> ask me how.)
>

Adding STM would make concurrency easier to deal with, but wouldn't address
the fundamental problem.

The proposed change doesn't prevent users from doing dangerous (and even
ugly things). It just forces them to *think* about what they're doing
beforehand. I can even see allowing immutable objects to change their
attributes, with the caveat that this shouldn't change the externally
visible behavior of the object.


>  > I believe this can be done at the mutation points (again, clojure
>  > shows that it can be done).
> But clojure is a Lisp-derived language.  Lisp was designed as a pure
> functional language (although AFAIK it pretty much immediately
> acquired "set"), and a very large number of Lisp algorithms are
> designed around conses which are (like Python tuples) basically
> immutable (yes, I know about setcar and setcdr, but use of those
> functions is generally considered a bug).  Whether that orientation
> toward immutable objects continues in Clojure I don't know, but if it
> does, the problem of designing a "write barrier" for mutations may be
> (a) simpler and (b) have less performance impact than the analogous
> task applied to Python.


Um, yeah, I did point out  later in the paragraph that preserving pythons
data types may make this assumption false.


> While Python-the-language does have some immutable objects like tuples
> and strings, it's really kind of hard to avoid use of containers like
> lists and dictionaries and classes with mutable objects.
>

And I also pointed out that this may be to much of a change to be palatable
to Python users. For that matter, if it requires losing pythons primitive
container types, it's probably to much of a change to be palatable to me.

    <mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111101/a97c161e/attachment.html>

From mwm at mired.org  Tue Nov  1 16:54:58 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 1 Nov 2011 08:54:58 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CADiSq7cu-BPOuP=JbRGStWH7M40yNW3o1WokammOqSr+Q5kS3w@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CADiSq7cu-BPOuP=JbRGStWH7M40yNW3o1WokammOqSr+Q5kS3w@mail.gmail.com>
Message-ID: <CAD=7U2D5A3i5qDMOoedXGrDZFeB=9XHZCaM0Lb_6SceWg0Tb4g@mail.gmail.com>

On Tue, Nov 1, 2011 at 1:31 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Tue, Nov 1, 2011 at 6:01 PM, Stephen J. Turnbull <stephen at xemacs.org>
> wrote:
> >  > I've identified the problem I want to solve: I want to make
> >  > concurrent use of python objects "safe by default",
> >
> > But that's not what you've proposed, AIUI.  You've proposed making
> > concurrent use *safer*, but not yet *safe*.  That's quite different
> > from the analogy with automatic memory management, where the
> > programmer can't do anything dangerous with pointers (because they
> > can't do anything at all).  The analogous model for concurrency is
> > processes, it seems to me.  (I don't have a desperate need for high-
> > performance concurrency, so I take no position on processes + message
> > passing vs. threads + shared resources.)
>
> Guido and python-dev in general *have* effectively taken a position on
> that, though (mainly due to Global Interpreter Lock discussions).
>
> 1. Even for threads, the recommended approach is to use queue.Queue to
> avoid the common concurrency issues (such as race conditions and
> deadlock) associated with explicit locking
> 2. In Python 3, concurrent.futures offers an even *safer* interface
> and higher level interface for many concurrent workloads
> 3. If you use multiple processes and serialised messages, or higher
> level APIs like concurrent.futures, you can not only scale to multiple
> cores, but also to multiple *machines*.
>

I am aware of all this. I've written large systems using Queue.queue and
the multiple process/serialized messages model. I've dealt with code that
tried to mix the two (*not* a good idea). The process model works really
well - if you can use it. The problem is, if you can't, you lose all the
protection it provides. That's the area I'm trying to address.

Also, the process model doesn't prevent these concurrency issues, it just
moves them to external objects. I figure that's an even harder problem,
since it can involve multiple machines. An improvement in the shared
storage case might shed some light on it.


> This has led to a quite deserved reputation for being intolerant of
> changes that claim to make multithreaded development "better", but
> only at the expense of making single-threaded development worse.
>

I think I've found a way to implement the proposal without having a serious
impact on single-threaded code - at least in terms of performance and
having to change the code.

     <mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111101/db421415/attachment.html>

From p.f.moore at gmail.com  Tue Nov  1 17:36:36 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 1 Nov 2011 16:36:36 +0000
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
Message-ID: <CACac1F8QycK1+=7TvoaUiFNnVrB6TGru0d20SuC84pEzza5-Mw@mail.gmail.com>

On 1 November 2011 15:38, Mike Meyer <mwm at mired.org> wrote:
> No, the proposal does make things "safe by default". The default behavior
> disallows all mutation. You have to do something explicit to allow it -
> because "explicit is better than implicit."
[...]
> Um, yeah, I did point out ?later in the paragraph that preserving pythons
> data types may make this assumption false.
[...]
> And I also pointed out that this may be to much of a change to be palatable
> to Python users. For that matter, if it requires losing pythons primitive
> container types, it's probably to much of a change to be palatable to me.

I don't know if you've considered this already, but a for-loop in
Python creates an iterator and then mutates it (by calling next()) on
each run through the loop. I can't see any way this could be a
concurrency problem in itself, but you'd likely need to either
reimplement the for loop to avoid relying on mutable iterators, or
you'd need to add some sort of exclusion for iterators in for loops.

It'll be details like this that will be the hardest to thrash out, I suspect...

Paul.


From van.lindberg at gmail.com  Tue Nov  1 18:08:22 2011
From: van.lindberg at gmail.com (VanL)
Date: Tue, 01 Nov 2011 12:08:22 -0500
Subject: [Python-ideas] Draft PEP for the regularization of Python
	install layouts
In-Reply-To: <CA+OGgf5wp5-JrmhjBg8No2ZN-hv8AH1UYigoxt+FcgaKY-h26A@mail.gmail.com>
References: <j8f887$lhb$1@dough.gmane.org>
	<CA+OGgf5wp5-JrmhjBg8No2ZN-hv8AH1UYigoxt+FcgaKY-h26A@mail.gmail.com>
Message-ID: <j8p926$hno$1@dough.gmane.org>

Hi Jim,

On Oct 30, 2011 5:58 PM, "Jim Jewett" <jimjjewett at gmail.com> wrote:

 > Is this something that Python even *can* reasonably control,
 > particularly on the various Linux distributions?

In my experience, the location for the Python environment changes a bit, 
but the internal layout is general consistent with what is set out in 
sysconfig and distutils.command.install. The most unique layout I have 
seen is for Ubuntu, which adds a vendor-packages directory.

I would love to be corrected in this regard.

 > What might be helpful would be a few more symbolics (if any are
 > actually missing) and a few test environments that use something
 > unexpected for each value, so that you *will* notice if you have
 > hardcoded assumptions specific to your own setup.

The suggested values are taken from the stdlib, not from my own setup, 
so if someone is generating or using different values, they are not 
coming from the stdlib.

Thanks,

Van



From phd at phdru.name  Tue Nov  1 18:29:37 2011
From: phd at phdru.name (Oleg Broytman)
Date: Tue, 1 Nov 2011 21:29:37 +0400
Subject: [Python-ideas] Draft PEP for the regularization of Python
 install layouts
In-Reply-To: <j8p926$hno$1@dough.gmane.org>
References: <j8f887$lhb$1@dough.gmane.org>
	<CA+OGgf5wp5-JrmhjBg8No2ZN-hv8AH1UYigoxt+FcgaKY-h26A@mail.gmail.com>
	<j8p926$hno$1@dough.gmane.org>
Message-ID: <20111101172937.GA30350@iskra.aviel.ru>

On Tue, Nov 01, 2011 at 12:08:22PM -0500, VanL wrote:
> On Oct 30, 2011 5:58 PM, "Jim Jewett" <jimjjewett at gmail.com> wrote:
> > Is this something that Python even *can* reasonably control,
> > particularly on the various Linux distributions?
> 
> In my experience, the location for the Python environment changes a
> bit, but the internal layout is general consistent with what is set
> out in sysconfig and distutils.command.install. The most unique
> layout I have seen is for Ubuntu, which adds a vendor-packages
> directory.

   Debian 6 "squeeze" added dist-packages for 3rd-party modules
installed via apt/dpkg.

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.


From stephen at xemacs.org  Tue Nov  1 19:05:06 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 02 Nov 2011 03:05:06 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
Message-ID: <87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>

Mike Meyer writes:

 > The proposed change doesn't prevent users from doing dangerous (and even
 > ugly things).

I didn't say it did, it "merely" imposes substantial inconvenience in
hope that:

 > It just forces them to *think* about what they're doing beforehand.

which I believe to be un-Pythonic.

But you say that you have an approach in mind which is reasonably
performant and doesn't change things too much for single-threaded
apps, which would make the discussion moot.  So let's see how that
works out.




From mwm at mired.org  Tue Nov  1 22:09:33 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 1 Nov 2011 14:09:33 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CACac1F8QycK1+=7TvoaUiFNnVrB6TGru0d20SuC84pEzza5-Mw@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<CACac1F8QycK1+=7TvoaUiFNnVrB6TGru0d20SuC84pEzza5-Mw@mail.gmail.com>
Message-ID: <CAD=7U2Aa=aFc_-MbazGsS-SS+YQ0w_UZtJ5Rr4MDni9p9F=zEQ@mail.gmail.com>

On Tue, Nov 1, 2011 at 9:36 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 1 November 2011 15:38, Mike Meyer <mwm at mired.org> wrote:
> > No, the proposal does make things "safe by default". The default behavior
> > disallows all mutation. You have to do something explicit to allow it -
> > because "explicit is better than implicit."
> [...]
> > Um, yeah, I did point out  later in the paragraph that preserving pythons
> > data types may make this assumption false.
> [...]
> > And I also pointed out that this may be to much of a change to be
> palatable
> > to Python users. For that matter, if it requires losing pythons primitive
> > container types, it's probably to much of a change to be palatable to me.
>
> I don't know if you've considered this already, but a for-loop in
> Python creates an iterator and then mutates it (by calling next()) on
> each run through the loop. I can't see any way this could be a
> concurrency problem in itself, but you'd likely need to either
> reimplement the for loop to avoid relying on mutable iterators, or
> you'd need to add some sort of exclusion for iterators in for loops.


How about a third option? Iterators have to be locked to do a next in
general, as they can be bound and thus shared between execution threads. On
the other hand, locking & unlocking should be the major performance hit, so
you don't want to do that on something that's going to be happening a lot,
so the caller should be allowed to do something to indicate that it's not
required. Locking the iterator should do that. So the next method needs to
add a test to see if self is locked, and if not lock and then unlock self.

It'll be details like this that will be the hardest to thrash out, I
> suspect...


Yup.

    Thanks,
    <mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111101/a0b5d9b9/attachment.html>

From mwm at mired.org  Tue Nov  1 22:13:07 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 1 Nov 2011 14:13:07 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>

On Tue, Nov 1, 2011 at 11:05 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:

> Mike Meyer writes:
>  > The proposed change doesn't prevent users from doing dangerous (and even
>  > ugly things).
> I didn't say it did, it "merely" imposes substantial inconvenience in
> hope that:
>  > It just forces them to *think* about what they're doing beforehand.
> which I believe to be un-Pythonic.
>

Really? Thinking is unpythonic?


> But you say that you have an approach in mind which is reasonably
> performant and doesn't change things too much for single-threaded
> apps, which would make the discussion moot.  So let's see how that
> works out.
>

If all you want to do is get the old semantics back in a single-threaded
application, you could do something like turning:

if __name__ == '__main__':
     main()

into:

if __name__ == '__main__':
     locking:
           main()

Actually, that achieves my goal - you hopefully thought about this long
enough to realize that this was safe before doing it.

     <mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111101/b7c71802/attachment.html>

From mwm at mired.org  Wed Nov  2 00:49:50 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 1 Nov 2011 16:49:50 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <j8o555$i65$1@dough.gmane.org>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<j8o555$i65$1@dough.gmane.org>
Message-ID: <CAD=7U2A6cjso-owd-4iERySansVe4nM_dTgxjgdwejx31LWnvw@mail.gmail.com>

On Mon, Oct 31, 2011 at 11:55 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 11/1/2011 1:32 AM, Mike Meyer wrote:
>> As you point out, this is a hard problem. I know I haven't covered all
>> the issues. That's why the second thing I said was that I'm hoping to
>> get people smarter than me to look at things.
>
> This is one of the hard problems that keep getting swept under the rug while we do easier things. Well, we have overhauled unicode and packaging for 3.3, so maybe concurrency can get some attention.

Hey, it worked!

>
> I keep thinking that CPython's design of allowing C coded modules either outside or inside the stdlib should allow some progress.
>
> Would it be helpful, for instance, to have truly immutable restricted tuples and frozensets, whose __new__ methods only allowed true immutables (None, booleans, numbers, strings, other restricted tuples and frozensets) as members?

Possibly. However, so long as the mutations they make don't change the
externally visible behavior, then for the purposes of this discussion,
they already are immutable. Or is it possible that concurrent updates
of that not-externally-visible state could cause things to break?

> How about a metaclass, say 'immutable', that made the instances of a user class truly immutable? (I don't know how to do this, but lets assume it can be done -- perhaps with a new frozendict.) If such were possible, instances of instances of such a metaclass could be added to the list above.

Well, on the basis that we're all adults, I'm willing to accept that a
programmer saying "I want instances of this class to be immutable"
means they'll only subvert whatever mechanism is used to do this when
it's safe to do so (i.e. - "not externally visible"), so catching
casual attempts - assignments to attributes - to do so will do, then
we can do this by providing a __setattr__ method that always throws an
exception.

Actually, I think that's the key to implementing this efficiently.
__setattr__ on objects that aren't locked throws an exception (or
triggers locking inside an STM). Locking them changes __setattr__ to
something that works appropriately. Builtin types will need more
extensive tweaking along those lines. An immutable type doesn't need
the working variant of __setattr__.

> Could a metaclass automatically add fine-grained locks around around attribute mutations?

Wouldn't that be another variation on the __setattr__ method, that did:

    locking self.__dict__:
       self.__dict__[name] = value

I can see that that would be useful, but would expect most objects
would want to change more than one attribute in a consistent method,
so they'd have a method that locked self and made all those changes.

? ? ?<mike


From tjreedy at udel.edu  Wed Nov  2 01:05:56 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 01 Nov 2011 20:05:56 -0400
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <4EAFC8EC.4080601@canterbury.ac.nz>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz>
Message-ID: <j8q1hf$4kq$1@dough.gmane.org>

On 11/1/2011 6:24 AM, Greg Ewing wrote:
...
> Cofunctions
> -----------
>
> There will be a special form of Python function called a "cofunction",
> defined
> using the new keyword ``codef`` in place of ``def``.

Is this really needed? The presence of 'coyield' signals 'cofunction', 
just as 'yield' signals 'generator'. Does a cofunction without a suspend 
point make sense? (And if it did, 'if False: coyield' or 'coyield' after 
'return' could serve as a signal.)

 > A cofunction
> provides a
> convenient way of creating an object obeying the coroutine protocol.
> (This is
> similar to how a generator provides a convenient way of creating an object
> obeying the iterator protocol).
>
> Suspension of a cofunction is achieved using the expression
>
> ::
>
> ``coyield`` [value]
>
> This is analogous to a ``yield`` expression in a generator, and like
> ``yield``,
> it can both provide and receive a value. However, unlike ``yield``, it
> is *not*
> restricted to communicating with the immediate caller. It communicates
> directly
> with the ``resume`` method of the coroutine, however deep the nesting of
> calls
> is between the ``resume`` call and the ``coyield``.
>
> There are some restrictions, however:
>
> * A ``coyield`` is only allowed in the body of a cofunction (a function
> defined
> with ``codef``), not in any other context.
>
> * A cofunction can only be called from the body of another cofunction,
> not in
> any other context.
>
> Exceptions are raised if any of these restrictions are violated.

Except that an initial 'call' from a coroutine.resume is needed to get 
the first cofunction started ;-).

-- 
Terry Jan Reedy



From ncoghlan at gmail.com  Wed Nov  2 01:45:05 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 2 Nov 2011 10:45:05 +1000
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <j8q1hf$4kq$1@dough.gmane.org>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz> <j8q1hf$4kq$1@dough.gmane.org>
Message-ID: <CADiSq7e1LAweBTa0Z8j0gyxFyP1RXH=Rwz9-byGB39z+zVeDOA@mail.gmail.com>

On Wed, Nov 2, 2011 at 10:05 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> Cofunctions
>> -----------
>>
>> There will be a special form of Python function called a "cofunction",
>> defined
>> using the new keyword ``codef`` in place of ``def``.
>
> Is this really needed? The presence of 'coyield' signals 'cofunction', just
> as 'yield' signals 'generator'. Does a cofunction without a suspend point
> make sense? (And if it did, 'if False: coyield' or 'coyield' after 'return'
> could serve as a signal.)

Something is needed, since there probably won't *be* an explicit
coyield in the top level function (instead, it would call async I/O
operations that used coyield internally).

However, as per the previous thread, I don't believe this needs to be
embedded in the bytecode by the compiler - it could instead be a
runtime switch in the eval loop, changing the way function calls and
iteration are handled.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From stephen at xemacs.org  Wed Nov  2 02:45:42 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 02 Nov 2011 10:45:42 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
Message-ID: <87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>

Mike Meyer writes:
 > On Tue, Nov 1, 2011 at 11:05 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:

 > > I didn't say it did, it "merely" imposes substantial inconvenience in
 > > hope that:
 > >  > It just forces them to *think* about what they're doing beforehand.
 > > which I believe to be un-Pythonic.
 > >
 > 
 > Really? Thinking is unpythonic?

No, "forcing" is.  Consenting adults and all that.

 > If all you want to do is get the old semantics back in a single-threaded
 > application, you could do something like turning:
 > 
 > if __name__ == '__main__':
 >      main()
 > 
 > into:
 > 
 > if __name__ == '__main__':
 >      locking:
 >            main()
 > 
 > Actually, that achieves my goal - you hopefully thought about this long
 > enough to realize that this was safe before doing it.

Anybody who does that is simply shutting off the warning/errors, and
clearly is not thinking about their app at all.

But this is revealing: you say *your* goal is making *me* think.
That's what I consider un-Pythonic.

A Pythonic approach would allow me to worry about it when *I* think it
necessary.  Maybe we don't have that choice, maybe concurrency is too
hard to solve without some annoying constraints.  But that's not at
all clear to me, and I'd rather make gradual progress toward safety in
a language that's fun and profitable to use, rather than have safety
in a language that is a pain in the neck to use.



From mwm at mired.org  Wed Nov  2 06:53:41 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 1 Nov 2011 22:53:41 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20111101225341.51801e1b@bhuda.mired.org>

On Wed, 02 Nov 2011 10:45:42 +0900
"Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> Mike Meyer writes:
>  > On Tue, Nov 1, 2011 at 11:05 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:
>  > > I didn't say it did, it "merely" imposes substantial inconvenience in
>  > > hope that:
>  > >  > It just forces them to *think* about what they're doing beforehand.
>  > > which I believe to be un-Pythonic.
>  > Really? Thinking is unpythonic?
> No, "forcing" is.  Consenting adults and all that.

But you yourself admit that this isn't forcing you to think:

>  > If all you want to do is get the old semantics back in a single-threaded
>  > application, you could do something like turning:
>  > 
>  > if __name__ == '__main__':
>  >      main()
>  > 
>  > into:
>  > 
>  > if __name__ == '__main__':
>  >      locking:
>  >            main()
>  > 
>  > Actually, that achieves my goal - you hopefully thought about this long
>  > enough to realize that this was safe before doing it.
> 
> Anybody who does that is simply shutting off the warning/errors, and
> clearly is not thinking about their app at all.

So you admit this doesn't force you to think. It just makes you add a
statement to shut up the warnings. Pretty much the same thing as using
a bare except clause.

Me, I'd think about it long enough to convince myself that the app
really was single-threaded.

> But this is revealing: you say *your* goal is making *me* think.

Only if I may wind up maintaining the code you wrote. But that's a
driving factor in a *lot* of the design decisions when it comes to
extending python.

> That's what I consider un-Pythonic.

I feel just the opposite. Python doesn't allow errors to silently
pass, or guess what the programmer wanted to do, or make inferences
about things - it raises exceptions. That forces the programmer to
think about the exception and handle it properly. Or they can not
think about it, and just use a bare except clause. I think that's very
pythonic. In fact, getting tired of chasing down such bugs in Perl
code was why I switched from Perl to Python, and then cut my rates in
order to convince my clients to let me write in what was then a
strange new language.

This proposal builds on that base: it catches errors of a type that
are currently ignored and raises an exception. It also adds a new
statement for *dealing* with those errors, because handling them with
exceptions won't really work. There's even an analog for the bare
except if you want to use it. And it comes about for much the same
reason: I'm getting tired of chasing down bugs in concurrent
code. There are languages that offer that. Some even run in
environments I like, and are fun to write when they're applicable. But
I find myself wishing for Python's features when I write in them.

> A Pythonic approach would allow me to worry about it when *I* think it
> necessary.  Maybe we don't have that choice, maybe concurrency is too
> hard to solve without some annoying constraints.  But that's not at
> all clear to me, and I'd rather make gradual progress toward safety in
> a language that's fun and profitable to use, rather than have safety
> in a language that is a pain in the neck to use.

Based on my experience, your second sentence is true. If that were all
it were, the Queue module would be most of a solution, and there are
STM modules available if that's not good enough. But they only solve
half the problem - they make it easier to get things right once you
decide the data is shared. People are as likely to miss that data is
shared as they are to screw up the locking. In other words, if we do
it your way, it'll deal with less than half of whats bugging me.

It may be that Python's data structures will make this unworkable. It
may be that a workable solution will suck the performance out of
non-concurrent applications. It may be that anything that fixes both
of those will be unpalatable for other reasons.  There's no way to
find out except by trying. And I'd rather try that than start trying
to convince people to let me write in some strange new language again.

   <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org


From ncoghlan at gmail.com  Wed Nov  2 07:12:06 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 2 Nov 2011 16:12:06 +1000
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <20111101225341.51801e1b@bhuda.mired.org>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
Message-ID: <CADiSq7cPU+ZWBVJkXqcoHnfVfbgGaNV8AAa8vAg_BNVnL0AaRA@mail.gmail.com>

On Wed, Nov 2, 2011 at 3:53 PM, Mike Meyer <mwm at mired.org> wrote:
> On Wed, 02 Nov 2011 10:45:42 +0900
> "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
>> Mike Meyer writes:
>> ?> On Tue, Nov 1, 2011 at 11:05 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote:
>> ?> > I didn't say it did, it "merely" imposes substantial inconvenience in
>> ?> > hope that:
>> ?> > ?> It just forces them to *think* about what they're doing beforehand.
>> ?> > which I believe to be un-Pythonic.
>> ?> Really? Thinking is unpythonic?
>> No, "forcing" is. ?Consenting adults and all that.
>
> But you yourself admit that this isn't forcing you to think:

It's forcing you to think the way Java's checked exceptions force you
to think - they make you think "Gee, it's tedious having to write all
this boilerplate to get the compiler/interpreter to STFU and let me
get on with doing my job".

"safe by default" is an excellent design principle, but so is "stay
out of the way". The two are often in tension, and this is one of
those times.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From stephen at xemacs.org  Wed Nov  2 09:49:48 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 02 Nov 2011 17:49:48 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <20111101225341.51801e1b@bhuda.mired.org>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
Message-ID: <87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>

Mike Meyer writes:

 > > No, "forcing" is.  Consenting adults and all that.
 > 
 > But you yourself admit that this isn't forcing you to think:

Nice try, but I didn't say it forces me to think.  It forces me to do
something to shut up the language.  That's ugly.

 > It just makes you add a statement to shut up the warnings. Pretty
 > much the same thing as using a bare except clause.

The bare except clause is optional; I can (and often do) simply let
the exception terminate the process *if* it ever happens.  My
understanding is that that isn't good enough for you (because
concurrency errors usually lead to silent data corruption rather than
a spectacular and immediate crash).

 > And it comes about for much the same reason: I'm getting tired of
 > chasing down bugs in concurrent code. There are languages that
 > offer that.

Well, if you want help chasing down bugs in concurrent code, I would
think that you would want to focus on concurrent code.  First, AFAICS
ordinary function calls don't expose additional objects to concurrency
(they may access exposed objects, of course, but they were passed in
from above by a task, or are globals).  So basically every object
exposed to concurrency is in either args or kwargs in a call to
threading.Thread (or thread.start_new_thread), no?

Wouldn't it be possible to wrap those objects (and only those objects)
such that the wrapper intercepts attempts to access the wrapped
objects, and "does something" (warn, raise, dance on the head of a
pin) if the access is unlocked or whatever?  Then only concurrent code
and the objects exposed to it pay the cost.  If it's really feasible
to do it via wrapper, you could write a decorator or something that
could easily be turned into a no-op for tested code ready to go into
production.

 > People are as likely to miss that data is shared as they are to
 > screw up the locking. In other words, if we do it your way, it'll
 > deal with less than half of whats bugging me.
[...]
 > There's no way to find out except by trying.

Well, no, it's not about doing it my way; I'm perfectly happy with
processes and message-passing in my applications, and aside from wacky
ideas like the above, that I don't really know how to implement
myself, I don't have a lot of suggestions for concurrency by
threading.  Rather, it's that my guess is that if you don't make the
costs of safe(r) concurrency look more reasonable you won't be getting
much help here.


From p.f.moore at gmail.com  Wed Nov  2 10:27:51 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 2 Nov 2011 09:27:51 +0000
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2Aa=aFc_-MbazGsS-SS+YQ0w_UZtJ5Rr4MDni9p9F=zEQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<CACac1F8QycK1+=7TvoaUiFNnVrB6TGru0d20SuC84pEzza5-Mw@mail.gmail.com>
	<CAD=7U2Aa=aFc_-MbazGsS-SS+YQ0w_UZtJ5Rr4MDni9p9F=zEQ@mail.gmail.com>
Message-ID: <CACac1F-xxoGasb4KD_m4YvMYCs=+ABkhD_jbsnnTNkWy3p=nVg@mail.gmail.com>

On 1 November 2011 21:09, Mike Meyer <mwm at mired.org> wrote:
> On Tue, Nov 1, 2011 at 9:36 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> I don't know if you've considered this already, but a for-loop in
>> Python creates an iterator and then mutates it (by calling next()) on
>> each run through the loop. I can't see any way this could be a
>> concurrency problem in itself, but you'd likely need to either
>> reimplement the for loop to avoid relying on mutable iterators, or
>> you'd need to add some sort of exclusion for iterators in for loops.
>
> How about a third option? Iterators have to be locked to do a next in
> general, as they can be bound and thus shared between execution threads. On
> the other hand, locking & unlocking should be the major performance hit, so
> you don't want to do that on something that's going to be happening a lot,
> so the caller should be allowed to do something to indicate that it's not
> required. Locking the iterator should do that. So the next method needs to
> add a test to see if self is locked, and if not lock and then unlock self.

I'm not sure what you mean here. Suppose I have

l = [1,2,3]
for i in l:
  print(i)

Here, the thing you need to lock is not l, as it's not being mutated,
but the temporary iterator generated by the for loop. That's not
exposed to the user, so you can't lock it manually. Should it be
locked? It can never be seen from another thread. But how do you code
that exception to the rule?

What about

l = iter([1,2,3])
for i in l:
  print(i)

Here the for loop gnerates iter(l) - which, simply because of the
implementation of __iter__ for iterators, returns l. So should I lock
l here? It *is* exposed to other threads, potentially. How does the
compiler detect the difference between this and the previous example?

This seems to me to be a recipe for having users scatter arbitrary
locks around their code "just to shut the  interpreter up". It's not
at all clear that it helps people think, in that there's no easy
mental model people can acquire to help them reason about what is
going on. Just a load of exceptions that need to be silenced somehow.

Paul.


From greg.ewing at canterbury.ac.nz  Wed Nov  2 10:44:28 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 02 Nov 2011 22:44:28 +1300
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <j8q1hf$4kq$1@dough.gmane.org>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz> <j8q1hf$4kq$1@dough.gmane.org>
Message-ID: <4EB110FC.1040108@canterbury.ac.nz>

Terry Reedy wrote:

> Is this really needed? The presence of 'coyield' signals 'cofunction', 
> just as 'yield' signals 'generator'.

The 'coyield' doesn't have to be directly in that function,
it could be in something called by that function, any number
of levels deep.

However, it's since occurred to me that 'coyield' doesn't have
to be a keyword, it could be a built-in cofunction.

>> * A cofunction can only be called from the body of another cofunction,
>> not in
>> any other context.
> 
> Except that an initial 'call' from a coroutine.resume is needed to get 
> the first cofunction started ;-).

Yes, but that's not done using the normal call syntax, which
is what I'm talking about there (that could perhaps be made
clearer).

-- 
Greg



From greg.ewing at canterbury.ac.nz  Wed Nov  2 10:54:56 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 02 Nov 2011 22:54:56 +1300
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <CADiSq7e1LAweBTa0Z8j0gyxFyP1RXH=Rwz9-byGB39z+zVeDOA@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz> <j8q1hf$4kq$1@dough.gmane.org>
	<CADiSq7e1LAweBTa0Z8j0gyxFyP1RXH=Rwz9-byGB39z+zVeDOA@mail.gmail.com>
Message-ID: <4EB11370.8060205@canterbury.ac.nz>

Nick Coghlan wrote:

> However, as per the previous thread, I don't believe this needs to be
> embedded in the bytecode by the compiler - it could instead be a
> runtime switch in the eval loop, changing the way function calls and
> iteration are handled.

Yes, but I'm no longer sure whether it's such a good idea
to have no special syntax at all to mark a cofunction,
seeing as cofunctionness won't be able to propagate through
C calls, special methods, etc.

By having cofunctions declared in a distinctive way, you
can look at the source and see exactly where the boundary is
between cofunction and non-cofunction code. Without such
markers, when you get an exception because you tried to
suspend in a non-coroutine zone, it may not be obvious
at which point along the call chain you made a mistake.

-- 
Greg


From anacrolix at gmail.com  Wed Nov  2 11:07:08 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Wed, 2 Nov 2011 21:07:08 +1100
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <4EAFC8EC.4080601@canterbury.ac.nz>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz>
Message-ID: <CAB4yi1OkBfmZdPamhQ6+j7BqVX_r1UY42q7HeqAiru+Bux+QMA@mail.gmail.com>

I don't think new keywords should be necessary. A module should be sufficient.
Also why CoExit when you have GeneratorExit? Might as well make it
CoroutineExit.

On Tue, Nov 1, 2011 at 9:24 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> A Coroutine Protocol
> ====================
>
> Here are some thoughts on the design of a new protocol to support
> lightweight
> threads using a mechanism similar to, but distinct from, generators and
> yield-from. Separating the two protocols will make it much easier to support
> suspendable generators, something that is not possible using the cofunction
> mechanism as currently specified in PEP 3152.
>
> The protocol to be described is similar in many ways to the generator
> protocol, and in what follows, analogies will be drawn between the two
> protocols
> where it may aid understanding.
>
>
> API
> ---
>
> This section describes the outward appearance of the coroutine mechanism to
> the programmer.
>
> A coroutine is created using the following constructor:
>
> ::
>
> ? ?coroutine(f, *args, **kwds)
>
> where ``f`` is an object obeying the "coroutine protocol" to be described
> below. Syntactic support will be provided for creating such an object using
> a special form of Python function definition, analogous to a generator.
>
> The result is a "coroutine object" having the following methods:
>
> ``resume(value = None)``
>
> ? ?Resumes execution of the coroutine at the point where it was last
> ? ?suspended. The value, if any, is passed into the coroutine and
> ? ?becomes the return value of the operation that caused the suspension.
> ? ?The coroutine executes until its next suspension point, at which
> ? ?time the ``resume`` call returns with the value passed into the
> ? ?suspension operation.
>
> ? ?(Note: This is analogous to calling next() or send() on a
> generator-iterator.
> ? ?Suspension of a coroutine is analogous to a generator executing a
> ? ?``yield`` operation.)
>
> ? ?If the coroutine has been freshly created, the passed-in value is
> ? ?ignored and the coroutine executes up to its first suspension point.
>
> ? ?If the top level of the coroutine finishes execution without
> ? ?encountering any further suspension points, a ``CoReturn`` exception
> ? ?is raised. This exception has a ``value`` attribute containing the
> ? ?return value from the coroutine.
>
> ? ?(Note: ``CoReturn`` is analogous to the ``StopIteration`` exception
> ? ?raised by an exhausted iterator or generator.)
>
> ``throw(exception)``
>
> ? ?Causes the given exception to be raised in the coroutine at its
> ? ?current suspension point.
>
> ``close()``
>
> ? ?Requests that the coroutine shut down and clean itself up. This is
> ? ?achieved by throwing in a ``CoExit`` exception (analogous to
> ``GeneratorExit``).
>
> It is expected that programmers will not write code that deals directly with
> coroutine objects very often; rather, some kind of driver or scheduler will
> be
> used that takes care of making ``resume()`` calls and handling ``CoReturn``
> exceptions.
>
>
> Cofunctions
> -----------
>
> There will be a special form of Python function called a "cofunction",
> defined
> using the new keyword ``codef`` in place of ``def``. A cofunction provides a
> convenient way of creating an object obeying the coroutine protocol. (This
> is
> similar to how a generator provides a convenient way of creating an object
> obeying the iterator protocol).
>
> Suspension of a cofunction is achieved using the expression
>
> ::
>
> ? ?``coyield`` [value]
>
> This is analogous to a ``yield`` expression in a generator, and like
> ``yield``,
> it can both provide and receive a value. However, unlike ``yield``, it is
> *not*
> restricted to communicating with the immediate caller. It communicates
> directly
> with the ``resume`` method of the coroutine, however deep the nesting of
> calls
> is between the ``resume`` call and the ``coyield``.
>
> There are some restrictions, however:
>
> * A ``coyield`` is only allowed in the body of a cofunction (a function
> defined
> with ``codef``), not in any other context.
>
> * A cofunction can only be called from the body of another cofunction, not
> in
> any other context.
>
> Exceptions are raised if any of these restrictions are violated.
>
> As a consequence, there must be an unbroken chain of cofunctions (or other
> objects
> obeying the cofunction protocol, see below) making up the call stack from
> the
> ``resume`` method down to the suspension point. A cofunction may call an
> ordinary
> function, but that function or anything called by it will not be able to
> suspend
> the coroutine.
>
> Note that the class of "ordinary functions" includes most functions and
> methods
> written in C. However, it is possible for an object implemented in C to
> participate
> in a coroutine stack by implementing the coroutine protocol below
> explicitly.
>
>
> Coroutine Protocol
> ------------------
>
> As well as the coroutine object, the coroutine protocol involves three other
> kinds
> of objects, "cocallable objects", "coframe objects" and "coiterator
> objects".
>
> A cocallable object has the following method:
>
> ``__cocall__(*args, **kwds)``
>
> ? ?Initiates a suspendable computation. Returns a coframe object.
>
> ? ?(This is analogous to the __iter__ method of an iterable object.)
>
> ? ?May return NotImplemented to signal that the object does not support the
> ? ?coroutine protocol. This enables wrapper objects such as bound methods to
> ? ?reflect whether or not the wrapped object supports the coroutine
> protocol.
>
> A coframe object has the following methods:
>
> ``__resume__(costack, value)``
>
> ? ?There are two purposes for which this method is called: to continue
> ? ?execution from a suspension point, and to pass in the return value
> resulting
> ? ?from a nested call to another cocallable object.
>
> ? ?In both cases, the ``resume`` method is expected to continue execution
> until
> ? ?the next suspension point, and return the value produced by it. If the
> ? ?computation finishes before reaching another suspension point,
> ? ?``CoReturn(retval)`` must be raised, where ``retval`` is the return value
> of
> ? ?the computation.
>
> ? ?(This method is analogous to the __send__ method of a generator-iterator.
> ? ?With a value of None, it is analogous to the __next__ method of an
> iterator.)
>
> ? ?The currently-executing coroutine object is passed in as the ``costack``
> ? ?parameter. The ``__resume__`` method can make a nested call to another
> cocallable
> ? ?object ``sub`` by performing:
>
> ? ? ? ?``return costack.call(sub, *args, **kwds)``
>
> ? ?No further calls to this coframe will be made until ``obj`` finishes.
> When
> ? ?it does, the ``__resume__`` method of this coframe ?is called with the
> ? ?return value from ``sub``.
>
> ? ?It is the responsibility of the coframe object to keep track of whether
> the
> ? ?previous call to its ``__resume__`` method resulted in a suspension or a
> nested
> ? ?call, and make use of the ``value`` parameter accordingly.
>
> ``__throw__(costack, exception)``
>
> ? ?Called to throw an exception into the computation. The coframe may choose
> to
> ? ?absorb the exception and continue executing, in which case ``__throw__``
> should
> ? ?return the value produced by the next exception point or raise
> ``CoReturn`` as
> ? ?for ``__resume__``. Alternatively it may allow the same or a different
> exception
> ? ?to propagate out.
>
> ? ?Implementation of this method is optional. If it is not present, the
> behaviour
> ? ?is as if a trivial ``__throw__`` method were present that simply
> re-raises the
> ? ?exception.
>
> A coiterator is an iterator that permits iteration to be carried out in a
> suspendable
> manner. A coiterator object has the following method:
>
> ``__conext__()``
>
> ? ?Returns a coframe for computing the next item from the iteration. This is
> the
> ? ?coroutine equivalent of an iterator's ``__next__`` method, and behaves
> accordingly:
> ? ?its ``__resume__`` method must return an item by raising
> ``CoReturn(item)``. To
> ? ?finish the iteration, it raises ``StopIteration`` as usual.
>
> To support coiteration, whenever a "next" operation is invoked by a
> cofunction
> (whether implicitly by means of a for-loop or explicitly by calling
> ``next()``)
> a ``__conext__`` method is first looked for, and if found, the operation is
> carried out suspendably. Otherwise a normal call is made to the ``__next__``
> method.
>
>
> Formal Semantics
> ----------------
>
> The semantics of the coroutine object are defined by the following Python
> implementation.
>
> ::
>
> ? ?class coroutine(object):
>
> ? ? ? ?# ?Public methods
>
> ? ? ? ?def __init__(self, main, *args, **kwds):
> ? ? ? ? ? ?self._stack = []
> ? ? ? ? ? ?self._push(_cocall(main, *args, **kwds))
>
> ? ? ? ?def resume(self, value = None):
> ? ? ? ? ? ?return self._run(value, None)
>
> ? ? ? ?def throw(self, exc):
> ? ? ? ? ? ?return self._run(None, exc)
>
> ? ? ? ?def close(self):
> ? ? ? ? ? ?try:
> ? ? ? ? ? ? ? ?self.throw(CoExit)
> ? ? ? ? ? ?except (CoExit, CoReturn):
> ? ? ? ? ? ? ? ?pass
>
> ? ? ? ?def call(self, subroutine, *args, **kwds):
> ? ? ? ? ? ?meth = getattr(subroutine, '__cocall__', None)
> ? ? ? ? ? ?if meth is not None:
> ? ? ? ? ? ? ? ?frame = meth(*args, **kwds)
> ? ? ? ? ? ? ? ?if frame is not NotImplemented:
> ? ? ? ? ? ? ? ? ? ?self._push(frame)
> ? ? ? ? ? ? ? ? ? ?return self._run(None, None)
> ? ? ? ? ? ?return CoReturn(subroutine(*args, **kwds))
>
> ? ? ? ?# ?Private methods
>
> ? ? ? ?def _run(self, value, exc):
> ? ? ? ? ? ?while True:
> ? ? ? ? ? ? ? ?try:
> ? ? ? ? ? ? ? ? ? ?frame = self._top()
> ? ? ? ? ? ? ? ? ? ?if exc is None:
> ? ? ? ? ? ? ? ? ? ? ? ?return frame.__resume__(self, value)
> ? ? ? ? ? ? ? ? ? ?else:
> ? ? ? ? ? ? ? ? ? ? ? ?meth = getattr(frame, '__throw__', None)
> ? ? ? ? ? ? ? ? ? ? ? ?if meth is not None:
> ? ? ? ? ? ? ? ? ? ? ? ? ? ?return meth(self, exc)
> ? ? ? ? ? ? ? ? ? ? ? ?else:
> ? ? ? ? ? ? ? ? ? ? ? ? ? ?raise exc
> ? ? ? ? ? ? ? ?except BaseException as exc:
> ? ? ? ? ? ? ? ? ? ?if self._pop():
> ? ? ? ? ? ? ? ? ? ? ? ?if isinstance(exc, CoReturn):
> ? ? ? ? ? ? ? ? ? ? ? ? ? ?value = exc.value
> ? ? ? ? ? ? ? ? ? ? ? ? ? ?exc = None
> ? ? ? ? ? ? ? ? ? ?else:
> ? ? ? ? ? ? ? ? ? ? ? ?raise
>
> ? ? ? ?def _push(self, frame):
> ? ? ? ? ? ?self._stack.append(frame)
>
> ? ? ? ?def _pop(self):
> ? ? ? ? ? ?if len(self._stack) > 0:
> ? ? ? ? ? ? ? ?del self._stack[-1]
> ? ? ? ? ? ? ? ?return True
> ? ? ? ? ? ?else:
> ? ? ? ? ? ? ? ?return False
>
> ? ? ? ?def _top(self):
> ? ? ? ? ? ?return self._stack[-1]
>
> --
> Greg
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From jh at improva.dk  Wed Nov  2 12:54:07 2011
From: jh at improva.dk (Jacob Holm)
Date: Wed, 02 Nov 2011 12:54:07 +0100
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <4EB11370.8060205@canterbury.ac.nz>
References: <4EA8BD66.6010807@canterbury.ac.nz>	<4EA9FED3.6050505@pearwood.info>	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>	<4EADBEA7.9000608@canterbury.ac.nz>	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>	<1320123506.9456.57.camel@Gutsy>
	<j8o3hk$afm$1@dough.gmane.org>	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>	<4EAFC8EC.4080601@canterbury.ac.nz>
	<j8q1hf$4kq$1@dough.gmane.org>	<CADiSq7e1LAweBTa0Z8j0gyxFyP1RXH=Rwz9-byGB39z+zVeDOA@mail.gmail.com>
	<4EB11370.8060205@canterbury.ac.nz>
Message-ID: <4EB12F5F.4080109@improva.dk>

On 2011-11-02 10:54, Greg Ewing wrote:
> Nick Coghlan wrote:
> 
>> However, as per the previous thread, I don't believe this needs to be
>> embedded in the bytecode by the compiler - it could instead be a
>> runtime switch in the eval loop, changing the way function calls and
>> iteration are handled.
> 
> Yes, but I'm no longer sure whether it's such a good idea
> to have no special syntax at all to mark a cofunction,
> seeing as cofunctionness won't be able to propagate through
> C calls, special methods, etc.
> 
> By having cofunctions declared in a distinctive way, you
> can look at the source and see exactly where the boundary is
> between cofunction and non-cofunction code. Without such
> markers, when you get an exception because you tried to
> suspend in a non-coroutine zone, it may not be obvious
> at which point along the call chain you made a mistake.
> 

If the switch Nick describes is available as a flag on the frame
objects, it would be easy to extend the traceback to show *exactly*
where you entered the no-coroutine zone that you are now failing to
suspend.  I don't think the additional syntax is helpful, and it would
be quite annoying to need to have two versions of every wrapper
function/decorator to make it useable in both contexts.

- Jacob


From ron3200 at gmail.com  Wed Nov  2 17:21:12 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Wed, 02 Nov 2011 11:21:12 -0500
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<20111027183208.GH20970@pantoffel-wg.de>
	<4EA9AB03.8070302@stoneleaf.us>
	<CAJ6cK1aUy76tY5Q6HPTPoBaJ2h4T2hv-NVSJw31BkvUt_FGOUA@mail.gmail.com>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
Message-ID: <1320250872.14285.65.camel@Gutsy>

On Tue, 2011-11-01 at 18:15 +1000, Nick Coghlan wrote:
> On Tue, Nov 1, 2011 at 4:27 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> > I believe raise just instantiates the indicated exception. I expect that
> > Exception.__new__ or .__init__ captures the traceback info. Subclasses can
> > add more. A SuspendExecution exception should be able to grab as much as is
> > needed for a resume. A CAPI call could be added if needed.
> 
> No, the traceback info is added by the eval loop itself. Remember that
> when you raise an exception *type* (rather than an instance), the
> exception doesn't get instantiated until it gets caught somewhere -
> the eval loop maintains the unwinding stack for the traceback as part
> of the thread state until it is time to attach it to the exception
> object.
> 
> This is all at the tail end of the eval loop in CPython, but be warned
> it's fairly brain bending stuff that depends on various internal
> details of the eval loop:
> http://hg.python.org/cpython/file/default/Python/ceval.c#l2879

Thanks for the link, I've been trying to get my brain bent around it,
but, yes it is hard to understand how it all ties together.

This morning I had a thought and maybe it may lead somewhere... 


Would it be possible to rewrite the 'yield' internals so they work in
the following way...  

     # a = yield b
     try:
          raise SuspendException(b, _self=_self)
     Except ContinueException as exc:
          a = exc.args 


     # b = gen.send(a)
     def send(gen, a=None):
         try:
             gen.throw(ContinueException(a))
         except SuspendException as exc:
             (gen, *b) = exc.args
         return b


The two requirements for this to work are...

    *A SuspendException needs to be able to pass out of the
    generator without causing it to stop.

    *A throw needs to be able to work where the
    SuspendException was raised.


The next issue after that is how to allow a subclass of SuspendException
to get pass the next() or .send() caller.  A subclassed SuspendException
would still be caught by 'except SuspendException as exc'.  This is
needed as a scheduler or other outer framework sits outside the scope
the generator is *called in.  

*Exceptions work in the callers frame rather than the defining scope.
That's an important feature as it will allow coroutines much more
freedom to be used in different contexts.


What this does is give the non_local symantics you mentioned earlier. 

Cheers,
   Ron



> > I hope you keep looking at this idea. Function calls stop execution and pass
> > control 'down', to be resumed by return. yield stops execution and passes
> > control 'up', to be resumed by next (or .send). Exceptions pass control 'up'
> > (or 'out') without the possibility of resuming. All that is lacking is
> > something to suspend and pass control 'sideways', to a specific target. A
> > special exception makes some sense in that exceptions already get the call
> > stack needed to resume after suspension.
> 
> That's not actually true - due to the need to process exception
> handling clauses and finally blocks (including the implicit ones
> inside with statements), the internal state of those frames is
> potentially no longer valid for resumption (they've moved on beyond
> the point where the internal function was called).
> 
> I'll also note that it isn't necessary to pass control sideways, since
> there are two different flavours of coroutine design (the PDF article
> in the other thread describes this well). The Lua version is
> "asymmetric coroutines", and they only allow you to return to the
> point that first invoked the coroutine (this model is a fairly close
> fit with Python's generators and exception handling). The greenlet
> version is "symmetric" coroutines, and those let you switch directly
> to any other coroutine.
> 
> Both models have their pros and cons, but the main advantage of
> asymmetric coroutines is that you can just say "suspend this thread"
> without having to say *where* you want to switch to. Of course, you
> can implement much the same API with symmetric coroutines as well, so
> long as you can look up your parent coroutine easily. Ultimately, I
> expect the symmetric vs asymmetric decision will be driven more by
> implementation details than by philosophical preferences one way or
> the other.
> 
> I will note that Ron's suggestion to leverage the existing eval loop
> stack collection provided by the exception handling machinery does
> heavily favour the asymmetric approach. Having a quick look to refresh
> my memory of some of the details of CPython's exception handling, I've
> come to the following tentative conclusions:
> 
> - an ordinary exception won't do, since you don't want to trigger
> except and finally blocks in outer frames (ceval.c#2903)
> - in CPython, a new "why = WHY_SUSPEND" at the eval loop layer is
> likely a better approach, since it would allow the frame stack to be
> collected without triggering exception handling
> - the stack unwinding would then end when a "SETUP_COCALL" block was
> encountered on the block stack (just as SETUP_EXCEPT and SETUP_FINALLY
> can stop the stack unwinding following an exception
> - with the block stacks within the individual frames preserved, the
> collected stack should be in a fit state for later restoration
> - the "fast_yield" code and the generator resumption code should also
> provide useful insight
> 
> There's nothing too magical there - once we disclaim the ability to
> suspend coroutines while inside a C function (even one that has called
> back in via the C/Python API), it should boil down to a combination of
> the existing mechanics for generators and exception handling. So, even
> though the above description is (highly) CPython specific, it should
> be feasible for other implementations to come up with something
> similar (although perhaps not easy:
> http://lua-users.org/lists/lua-l/2007-07/msg00002.html).
> 
> Cheers,
> Nick.
> 




From guido at python.org  Wed Nov  2 18:26:20 2011
From: guido at python.org (Guido van Rossum)
Date: Wed, 2 Nov 2011 10:26:20 -0700
Subject: [Python-ideas] Changing str(someclass) to return only the class
	name
In-Reply-To: <alpine.DEB.1.00.1110281451480.31423@holly.lfw.org>
References: <4EA18598.9060602@netwok.org> <4EA1AFB0.4080000@pearwood.info>
	<CADiSq7fJwz6bTx9ryEhQt-GBpt=uK_jL6mAnCkPgHtCUcdiq7Q@mail.gmail.com>
	<4EA27189.8010002@pearwood.info>
	<CADiSq7dsockWVZEUHTvWDpeKaSEV1tQtiPbgf+3QSUYTUryUYA@mail.gmail.com>
	<4EA32507.7010900@pearwood.info> <4EAAD51A.9030608@netwok.org>
	<CAP7+vJJAK37ceCQ4bUJtaUi-VZBiN0mt4YQRdxTsJ_5FQ0ggDw@mail.gmail.com>
	<alpine.DEB.1.00.1110281451480.31423@holly.lfw.org>
Message-ID: <CAP7+vJ+mkRg6M8LKiT-K-zc=-cWNxGUdDRWA2ZxyD8K13aYQsQ@mail.gmail.com>

-1. I'd like it so that (given a suitable set of imports) if you typed
back to the interpreter what it printed at you, you get the same thing
back again.

>>> print(x)
42
>>> print(42)
42
>>> print(y)
None
>>> print(None)
None
>>> print(z)
foo
>>> print(foo)
foo

Ping's proposal would goes against this:

>>> print(x)
foo()
>>> print(foo())
42
>>> print(Splat)
class Splat
>>> print(class Splat)
SyntaxError: invalid syntax

I'm expecting that in most cases there is enough redundancy in the
name that you'll know what kind of thing it is. And if you want to
know for sure, continue to use repr() -- or, at the interactive
prompt, just omit the print() call, since the interactive interpreter
automatically calls repr() on your expression.

--Guido

On Fri, Oct 28, 2011 at 3:00 PM, Ka-Ping Yee <python at zesty.ca> wrote:
> Hi there,
>
> I get that repr() is supposed to be the precise representation
> and str() is intended more to be friendly than precise. ?My
> concern with the proposal is just that this:
>
> ? ?>>> print x
> ? ?foo
>
> ...doesn't actually feel that friendly to me. ?I want to know
> that it's *probably* a function or *probably* a class, the same
> way that today, when I see:
>
> ? ?>>> print x
> ? ?biscuit
>
> ? ?>>> print y
> ? ?[1, 2, 3]
>
> I can guess that x is *probably* a string and y is *probably*
> a list (e.g. because I know I'm not working with any custom
> objects whose __str__ returns those things).
>
> It would create a slightly higher mental burden (or slightly
> higher probability of human error) if, when I see:
>
> ? ?>>> print x
> ? ?Splat
>
> ...I have to remember that x might be a string or a function or
> a class.
>
> I'd just like some kind of visual hint as to what it is. ?Like:
>
> ? ?>>> print x
> ? ?foo()
>
> or:
>
> ? ?>>> print x
> ? ?function foo
>
> or:
>
> ? ?>>> print x
> ? ?function foo(a, b)
>
> or:
>
> ? ?>>> print x
> ? ?class Bar
>
> In fact "function foo(a, b)" would actually be rather useful
> in a lot of situations, and I would argue, friendlier than "foo".
>
>
> --Ping
>



-- 
--Guido van Rossum (python.org/~guido)


From mwm at mired.org  Wed Nov  2 19:10:48 2011
From: mwm at mired.org (Mike Meyer)
Date: Wed, 2 Nov 2011 11:10:48 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CACac1F-xxoGasb4KD_m4YvMYCs=+ABkhD_jbsnnTNkWy3p=nVg@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<CACac1F8QycK1+=7TvoaUiFNnVrB6TGru0d20SuC84pEzza5-Mw@mail.gmail.com>
	<CAD=7U2Aa=aFc_-MbazGsS-SS+YQ0w_UZtJ5Rr4MDni9p9F=zEQ@mail.gmail.com>
	<CACac1F-xxoGasb4KD_m4YvMYCs=+ABkhD_jbsnnTNkWy3p=nVg@mail.gmail.com>
Message-ID: <CAD=7U2ASqkFrkjxcgS+HkxZCRpT8X_1v2QPZeoj531kncN=wyA@mail.gmail.com>

On Wed, Nov 2, 2011 at 2:27 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 1 November 2011 21:09, Mike Meyer <mwm at mired.org> wrote:
>> On Tue, Nov 1, 2011 at 9:36 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>> I don't know if you've considered this already, but a for-loop in
>>> Python creates an iterator and then mutates it (by calling next()) on
>>> each run through the loop. I can't see any way this could be a
>>> concurrency problem in itself, but you'd likely need to either
>>> reimplement the for loop to avoid relying on mutable iterators, or
>>> you'd need to add some sort of exclusion for iterators in for loops.
>>
>> How about a third option? Iterators have to be locked to do a next in
>> general, as they can be bound and thus shared between execution threads. On
>> the other hand, locking & unlocking should be the major performance hit, so
>> you don't want to do that on something that's going to be happening a lot,
>> so the caller should be allowed to do something to indicate that it's not
>> required. Locking the iterator should do that. So the next method needs to
>> add a test to see if self is locked, and if not lock and then unlock self.
>
> I'm not sure what you mean here. Suppose I have
>
> l = [1,2,3]
> for i in l:
> ?print(i)
>
> Here, the thing you need to lock is not l, as it's not being mutated,
> but the temporary iterator generated by the for loop. That's not
> exposed to the user, so you can't lock it manually. Should it be
> locked? It can never be seen from another thread. But how do you code
> that exception to the rule?

You don't have to do anything. Iterators need to lock themselves to be
safe in concurrent use, so this will work fine, with the temporary
iterator doing whatever locking is needed.

> What about
>
> l = iter([1,2,3])
> for i in l:
> ?print(i)

This will work the same as the above. However, since you have the
iterator in hand, you have the option to lock it before entering the
for loop, which will cause it to *not* do it's own internal locking.

This brings up an interesting point, though - look for mail with the
subject "Concurrency survey..."

    <mike


From mwm at mired.org  Wed Nov  2 19:40:54 2011
From: mwm at mired.org (Mike Meyer)
Date: Wed, 2 Nov 2011 11:40:54 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
	<87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>

On Wed, Nov 2, 2011 at 1:49 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Mike Meyer writes:
> Well, if you want help chasing down bugs in concurrent code, I would
> think that you would want to focus on concurrent code. ?First, AFAICS
> ordinary function calls don't expose additional objects to concurrency
> (they may access exposed objects, of course, but they were passed in
> from above by a task, or are globals). ?So basically every object
> exposed to concurrency is in either args or kwargs in a call to
> threading.Thread (or thread.start_new_thread), no?

No. You missed two things. First, all the objects that can be accessed
- however indirectly - through those objects are exposed to
concurrency. Also, any globals and anything that can be accessed
through them are exposed to concurrency.  No, make that three things:
a wrapped C library that has call backs to Python code and uses
threads internally can expose anything it's passed (and anything
accessible from those objects) to concurrency, without ever using the
Python threading code.

I mentioned see a bug yesterday. My clients response means I can't in
good faith try and fix it (it's in a testing framework, so doesn't
affect the product, so they don't care). So this is a guess, but
here's what I think is going on:

1) We're using a test framework that creates a mock time module for
some reason. At some point, the mock object has the value None.
2) The application being tested  uses a database module that uses
threads as part of managing a connection pool.

The concurrency unsafe codde in the test framework (which is clearly
and obviously single-threaded, right?) managed to get the None-valued
mock inserted in sys.modules due to a concurrency bug. So when I then
use the time module in testing, I get an exception trying to access
it's attributes.

This does show a bigger problem. Look for my next mail...

   <mike


From p.f.moore at gmail.com  Wed Nov  2 20:31:12 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 2 Nov 2011 19:31:12 +0000
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2ASqkFrkjxcgS+HkxZCRpT8X_1v2QPZeoj531kncN=wyA@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<CACac1F8QycK1+=7TvoaUiFNnVrB6TGru0d20SuC84pEzza5-Mw@mail.gmail.com>
	<CAD=7U2Aa=aFc_-MbazGsS-SS+YQ0w_UZtJ5Rr4MDni9p9F=zEQ@mail.gmail.com>
	<CACac1F-xxoGasb4KD_m4YvMYCs=+ABkhD_jbsnnTNkWy3p=nVg@mail.gmail.com>
	<CAD=7U2ASqkFrkjxcgS+HkxZCRpT8X_1v2QPZeoj531kncN=wyA@mail.gmail.com>
Message-ID: <CACac1F-ZnmoGnNkKY2QmbsvOcpiQzkwBt_hfVWiO3Dd=z+xOcw@mail.gmail.com>

On 2 November 2011 18:10, Mike Meyer <mwm at mired.org> wrote:
> You don't have to do anything. Iterators need to lock themselves to be
> safe in concurrent use, so this will work fine, with the temporary
> iterator doing whatever locking is needed.

So all iterators automatically lock themselves? Sounds like
potentially quite an overhead.

But you've obviously thought about it (even if I don't follow all the
details) so I'll leave it at that.
Paul.


From mwm at mired.org  Wed Nov  2 20:36:26 2011
From: mwm at mired.org (Mike Meyer)
Date: Wed, 2 Nov 2011 12:36:26 -0700
Subject: [Python-ideas] A concurrency survey of sorts
Message-ID: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>

In order to get a better idea of where things stand, I'd like to get
answers to a few questions. This isn't a traditional broadbased
survey, but an attempt to get answers from a few people who might know
or have good ideas. This is probably where I should have started, but
better late than never.

1) How much of the Python standard library is known to be thread safe?

2) How many packages in PyPI are known to be thread safe?

3) Can you suggest another approach to getting safe high-performance
shared data in concurrent operation? I've already considered:

 a) I proposed making actions that mutate data require locked objects,
because I've seen that work in other languages. I recognize that
doesn't mean it will work in Python, but it's more than I can say
about the alternatives I knew about then.,

  b) Bertrand Meyer's SCOOPS system, designed for Eiffel. It has two
major strikes against it: 1) it is based on type attributes on
*variables*, andI could figure out how to translate that to a language
where variables aren't typed. 2) I don't know that there's a working
implementation.

4) Can you suggest a minor change that would move things toward safer
concurrent code with high-performance shared data? I can see two
possibilities:

 a) Audit any parts of the standard library that aren't already known
to be thread safe, and flag those that aren't.  Fixing them may want
to wait on a better mechanism than posix locks.

 b) Add a high-level, high-performance shared object facility to the
multiprocess package.

    Thanks,
    <mike


From mwm at mired.org  Wed Nov  2 20:47:01 2011
From: mwm at mired.org (Mike Meyer)
Date: Wed, 2 Nov 2011 12:47:01 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CACac1F-ZnmoGnNkKY2QmbsvOcpiQzkwBt_hfVWiO3Dd=z+xOcw@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<CACac1F8QycK1+=7TvoaUiFNnVrB6TGru0d20SuC84pEzza5-Mw@mail.gmail.com>
	<CAD=7U2Aa=aFc_-MbazGsS-SS+YQ0w_UZtJ5Rr4MDni9p9F=zEQ@mail.gmail.com>
	<CACac1F-xxoGasb4KD_m4YvMYCs=+ABkhD_jbsnnTNkWy3p=nVg@mail.gmail.com>
	<CAD=7U2ASqkFrkjxcgS+HkxZCRpT8X_1v2QPZeoj531kncN=wyA@mail.gmail.com>
	<CACac1F-ZnmoGnNkKY2QmbsvOcpiQzkwBt_hfVWiO3Dd=z+xOcw@mail.gmail.com>
Message-ID: <CAD=7U2BN2-4Kwn0nFi-tXND0kynAizgNNNpg0Xe+K-Vd1KUK+A@mail.gmail.com>

On Wed, Nov 2, 2011 at 12:31 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 2 November 2011 18:10, Mike Meyer <mwm at mired.org> wrote:
>> You don't have to do anything. Iterators need to lock themselves to be
>> safe in concurrent use, so this will work fine, with the temporary
>> iterator doing whatever locking is needed.
> So all iterators automatically lock themselves? Sounds like
> potentially quite an overhead.

No, all iterators are written to be thread safe. This is pretty much a
requirement if you want to use them in a threaded environment. Some
iterators may be able to do this without locking. I suspect most
won't.

This makes me wonder about something. Is there a high-performance
threading world that 1) doesn't assume that threading is the norm and
thus doesn't worry about single-threaded performance (this is the
impression I get about Java, but I may well be wrong) and 2) doesn't
require knowing at either build (C/C++) or launch (haskell) time that
it's going to be threaded? I haven't found such.

    <mike


From tjreedy at udel.edu  Wed Nov  2 23:28:15 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 02 Nov 2011 18:28:15 -0400
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CACac1F-xxoGasb4KD_m4YvMYCs=+ABkhD_jbsnnTNkWy3p=nVg@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<CACac1F8QycK1+=7TvoaUiFNnVrB6TGru0d20SuC84pEzza5-Mw@mail.gmail.com>
	<CAD=7U2Aa=aFc_-MbazGsS-SS+YQ0w_UZtJ5Rr4MDni9p9F=zEQ@mail.gmail.com>
	<CACac1F-xxoGasb4KD_m4YvMYCs=+ABkhD_jbsnnTNkWy3p=nVg@mail.gmail.com>
Message-ID: <j8sg6b$ntp$1@dough.gmane.org>

On 11/2/2011 5:27 AM, Paul Moore wrote:

> I'm not sure what you mean here. Suppose I have
>
> l = [1,2,3]
> for i in l:
>    print(i)
>
> Here, the thing you need to lock is not l, as it's not being mutated,

If l is exposed to another thread, it can be mutated, and the hidden 
iterator in the for loop will work, but with indeterminant, or at least 
unexpected results. Were you implicitly excluding such exposure?

(A list can also be mutated within its iteration loop. There is even a 
use case for deleting an item while iterating in reverse.)

Dicts *are* locked for iteration because mutating a hash array during 
iteration could have more drastic effects and there is no good use case. 
A built-in subclass of list could use the same mechanism as dict for 
locking during iteration.

> but the temporary iterator generated by the for loop. That's not
> exposed to the user, so you can't lock it manually. Should it be
> locked? It can never be seen from another thread.

So no need to lock *it*.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Wed Nov  2 23:34:55 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 02 Nov 2011 18:34:55 -0400
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <4EB110FC.1040108@canterbury.ac.nz>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz> <j8q1hf$4kq$1@dough.gmane.org>
	<4EB110FC.1040108@canterbury.ac.nz>
Message-ID: <j8sgiq$qb0$1@dough.gmane.org>

On 11/2/2011 5:44 AM, Greg Ewing wrote:
> Terry Reedy wrote:
>
>> Is this really needed? The presence of 'coyield' signals 'cofunction',
>> just as 'yield' signals 'generator'.
>
> The 'coyield' doesn't have to be directly in that function,
> it could be in something called by that function, any number
> of levels deep.
>
> However, it's since occurred to me that 'coyield' doesn't have
> to be a keyword, it could be a built-in cofunction.

My *feeling* is that 'codef' and 'coyield' are a bit ugly, so that
a) they may hardly be used in extand code, and would be 'safe' to use as 
keywords, but
b) I would not like to see them as keywords ;-).

This is aside from thinking that the relative smallness of Python's 
keyword list is a *feature*.

>>> * A cofunction can only be called from the body of another cofunction,
>>> not in
>>> any other context.
>>
>> Except that an initial 'call' from a coroutine.resume is needed to get
>> the first cofunction started ;-).
>
> Yes, but that's not done using the normal call syntax, which
> is what I'm talking about there

I understood that,

> (that could perhaps be made clearer).

but the sentence, taken in isolation, does raise the bootstrap issue.
So from an editorial viewpoint, I suggested the short addition.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Thu Nov  3 00:01:38 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 02 Nov 2011 19:01:38 -0400
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2A6cjso-owd-4iERySansVe4nM_dTgxjgdwejx31LWnvw@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<j8o555$i65$1@dough.gmane.org>
	<CAD=7U2A6cjso-owd-4iERySansVe4nM_dTgxjgdwejx31LWnvw@mail.gmail.com>
Message-ID: <j8si4t$4ht$1@dough.gmane.org>

On 11/1/2011 7:49 PM, Mike Meyer wrote:
> On Mon, Oct 31, 2011 at 11:55 PM, Terry
> Reedy<tjreedy at udel.edu>  wrote:

>> This is one of the hard problems that keep getting swept under the
>> rug while we do easier things. Well, we have overhauled unicode and
>> packaging for 3.3, so maybe concurrency can get some attention.
>
> Hey, it worked!

Yes. Coroutines, which are a form of in-thread concurrency, are another 
'under the rug' subject, which Greg has re-exposed. We need some 
possibly initially off-the-wall ideas for both.

>> Would it be helpful, for instance, to have truly immutable
>> restricted tuples and frozensets, whose __new__ methods only
>> allowed true immutables (None, booleans, numbers, strings, other
>> restricted tuples and frozensets) as members?
>
> Possibly. However, so long as the mutations they make don't change
> the externally visible behavior, then for the purposes of this
> discussion, they already are immutable. Or is it possible that
> concurrent updates of that not-externally-visible state could cause
> things to break?

It would seem to me that a list within a tuple or frozenset is just as 
'dangerous' as a list that is exposed directly. Correspondingly, a 
safe_list that locked all appropriate operations should be equally safe 
within or without.

Tuples containing mutables cannot be a dict key because they cannot be 
hashed because (truthful) mutables do not have a hash method returning 
an int. ({}.__hash__() and [].__hash__() are both None.) This suggests 
that one implementation for a safe(r) tuple class would be to try to 
hash a tuple upon creation (and might as well stash it away).

>> How about a metaclass, say 'immutable', that made the instances of
>> a user class truly immutable? (I don't know how to do this, but
>> lets assume it can be done -- perhaps with a new frozendict.) If
>> such were possible, instances of instances of such a metaclass
>> could be added to the list above.
>
> Well, on the basis that we're all adults, I'm willing to accept that
> a programmer saying "I want instances of this class to be immutable"
> means they'll only subvert whatever mechanism is used to do this
> when it's safe to do so (i.e. - "not externally visible"), so
> catching casual attempts - assignments to attributes - to do so will
> do, then we can do this by providing a __setattr__ method that always
> throws an exception.

I know about that, but there is no way for a safe_tuple class to know 
what a __setattr__ method does. But my new idea here is just to call 
hash(). That is not completely safe, but perhaps within 'consenting 
adults' territory.

(In other words, if someone adds a __hash__ method to a class with 
mutable instances, puts such instances into a safe_tuple shared with 
multiple threads, mutates from multiple threads, and has a problem, ... 
too bad. And I hope you would never have to debug such ;-).

-- 
Terry Jan Reedy



From mwm at mired.org  Thu Nov  3 00:13:49 2011
From: mwm at mired.org (Mike Meyer)
Date: Wed, 2 Nov 2011 16:13:49 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <j8si4t$4ht$1@dough.gmane.org>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<j8o555$i65$1@dough.gmane.org>
	<CAD=7U2A6cjso-owd-4iERySansVe4nM_dTgxjgdwejx31LWnvw@mail.gmail.com>
	<j8si4t$4ht$1@dough.gmane.org>
Message-ID: <CAD=7U2AtJz437_V7Uv-GkET1FadtkSAZSbrY0qZ49oPGSccAnw@mail.gmail.com>

On Wed, Nov 2, 2011 at 4:01 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 11/1/2011 7:49 PM, Mike Meyer wrote:
>> On Mon, Oct 31, 2011 at 11:55 PM, Terry
>> Reedy<tjreedy at udel.edu> ?wrote:
>>> This is one of the hard problems that keep getting swept under the
>>> rug while we do easier things. Well, we have overhauled unicode and
>>> packaging for 3.3, so maybe concurrency can get some attention.
>> Hey, it worked!
> Yes. Coroutines, which are a form of in-thread concurrency, are another
> 'under the rug' subject, which Greg has re-exposed. We need some possibly
> initially off-the-wall ideas for both.

I've been avoiding pointing out that connection, but that's one of the
reasons I kept saying "thread of execution" instead of just "thread" -
at least until recently. There are more ways to get concurrency than
threading and processes.

>>> Would it be helpful, for instance, to have truly immutable
>>> restricted tuples and frozensets, whose __new__ methods only
>>> allowed true immutables (None, booleans, numbers, strings, other
>>> restricted tuples and frozensets) as members?
>> Possibly. However, so long as the mutations they make don't change
>> the externally visible behavior, then for the purposes of this
>> discussion, they already are immutable. Or is it possible that
>> concurrent updates of that not-externally-visible state could cause
>> things to break?
> It would seem to me that a list within a tuple or frozenset is just as
> 'dangerous' as a list that is exposed directly. Correspondingly, a safe_list
> that locked all appropriate operations should be equally safe within or
> without.

Right. However, What needs to made safe for concurrency here is the
list, not the tuple. As you point out, that doesn't help for things
like dict keys.

This discussion was more about objects that compute and then cache a
value based on their contents - hash values being one of them.

>>> How about a metaclass, say 'immutable', that made the instances of
>>> a user class truly immutable? (I don't know how to do this, but
>>> lets assume it can be done -- perhaps with a new frozendict.) If
>>> such were possible, instances of instances of such a metaclass
>>> could be added to the list above.
>>
>> Well, on the basis that we're all adults, I'm willing to accept that
>> a programmer saying "I want instances of this class to be immutable"
>> means they'll only subvert whatever mechanism is used to do this
>> when it's safe to do so (i.e. - "not externally visible"), so
>> catching casual attempts - assignments to attributes - to do so will
>> do, then we can do this by providing a __setattr__ method that always
>> throws an exception.
> I know about that, but there is no way for a safe_tuple class to know what a
> __setattr__ method does. But my new idea here is just to call hash(). That
> is not completely safe, but perhaps within 'consenting adults' territory.

Presumably, there'd be a better way to ask that question. Something
like "isinstance(object, Immutable)" where Immutable adds the
appropriate __setattr__ method (though that other problems).

> (In other words, if someone adds a __hash__ method to a class with mutable
> instances, puts such instances into a safe_tuple shared with multiple
> threads, mutates from multiple threads, and has a problem, ... too bad. And
> I hope you would never have to debug such ;-).

Doesn't sound much worse than most concurrency bugs to me. They tend
to crop up in places where it's "obvious" things aren't concurrent.
Those have replaced stray pointer bugs as the nastiest bugs to deal
with, at least for me.

    <mike


From tjreedy at udel.edu  Thu Nov  3 00:25:05 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 02 Nov 2011 19:25:05 -0400
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <20111101225341.51801e1b@bhuda.mired.org>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
Message-ID: <j8sjgt$dvg$1@dough.gmane.org>

On 11/2/2011 1:53 AM, Mike Meyer wrote:
> On Wed, 02 Nov 2011 10:45:42 +0900
> "Stephen J. Turnbull"<stephen at xemacs.org>  wrote:

>> But this is revealing: you say *your* goal is making *me* think.
>
> Only if I may wind up maintaining the code you wrote. But that's a
> driving factor in a *lot* of the design decisions when it comes to
> extending python.
>
>> That's what I consider un-Pythonic.
>
> I feel just the opposite. Python doesn't allow errors to silently
> pass, or guess what the programmer wanted to do, or make inferences
> about things - it raises exceptions.

It is true that Python actually does a lot to protects programmers (from 
their own ignorance and foolishness -- but with ctypes wiping out all 
protections ;-). One easily overloop example is the impossibility by 
design of buffer over-runs, a common security problem. But it does so 
without feeling like a straightjacket.

Breaking a large fraction of existing code is too much of a sledgehammer 
approach. If Python had been designed from the beginning for 
multi-threading as the default, with single-threading as the exception, 
the case would be different. But even now, many are happy with single 
thread in multiple processes or multiple task objects within a single 
thread for concurrency. So burdening single thread code will not be popular.

-- 
Terry Jan Reedy



From greg.ewing at canterbury.ac.nz  Thu Nov  3 01:31:04 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 03 Nov 2011 13:31:04 +1300
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <CAB4yi1OkBfmZdPamhQ6+j7BqVX_r1UY42q7HeqAiru+Bux+QMA@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz>
	<CAB4yi1OkBfmZdPamhQ6+j7BqVX_r1UY42q7HeqAiru+Bux+QMA@mail.gmail.com>
Message-ID: <4EB1E0C8.5040707@canterbury.ac.nz>

On 02/11/11 23:07, Matt Joiner wrote:
> Also why CoExit when you have GeneratorExit? Might as well make it
> CoroutineExit.

The exact names of things are up for negotiation.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Thu Nov  3 01:40:23 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 03 Nov 2011 13:40:23 +1300
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <4EB12F5F.4080109@improva.dk>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz> <j8q1hf$4kq$1@dough.gmane.org>
	<CADiSq7e1LAweBTa0Z8j0gyxFyP1RXH=Rwz9-byGB39z+zVeDOA@mail.gmail.com>
	<4EB11370.8060205@canterbury.ac.nz> <4EB12F5F.4080109@improva.dk>
Message-ID: <4EB1E2F7.2030404@canterbury.ac.nz>

On 03/11/11 00:54, Jacob Holm wrote:
> If the switch Nick describes is available as a flag on the frame
> objects, it would be easy to extend the traceback to show *exactly*
> where you entered the no-coroutine zone

Yes, the same thing occurred to me shortly after posting that.

> I don't think the additional syntax is helpful,

It would help with auditing by making it possible to see whether
the rules are being followed by statically examining the text,
instead of having to wait for a run-time failure. However, that
needs to be weighed against the two-versions problem, which
I acknowledge is fairly serious.

-- 
Greg


From victor.stinner at haypocalc.com  Thu Nov  3 01:44:33 2011
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Thu, 3 Nov 2011 01:44:33 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111030001801.2f52ceb2@pitrou.net>
References: <20111030001801.2f52ceb2@pitrou.net>
Message-ID: <201111030144.33941.victor.stinner@haypocalc.com>

> >>> class C:
> ...   def f(): pass
> ...   class D:
> ...     def g(): pass
> ...
> 
> >>> C.D.__qname__
> 'C.D'
> 
> >>> C.D.g.__qname__
> 'C.D.g'
> 
> >>> def f():
> ...   def g(): pass
> ...   return g
> ...
> 
> >>> f().__qname__
> 'f.g'

Oh, +1 just for these examples.

It would help debugging if the __repr__ method of common objects are patched 
to use it.

Victor


From greg.ewing at canterbury.ac.nz  Thu Nov  3 02:47:41 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 03 Nov 2011 14:47:41 +1300
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <1320250872.14285.65.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy>
Message-ID: <4EB1F2BD.7070202@canterbury.ac.nz>

On 03/11/11 05:21, Ron Adam wrote:

> Would it be possible to rewrite the 'yield' internals so they work in
> the following way...
>
>       # a = yield b
>       try:
>            raise SuspendException(b, _self=_self)
>       Except ContinueException as exc:
>            a = exc.args

I've just been thinking about something like that, while pondering
whether there is a middle ground somewhere between the yield-from
mechanism and a completely new coroutine protocol.

The problem I had with building all of it on yield-from was that
there was no way to distinguish a 'yield' being used for the purpose
of suspending a coroutine from one being used to return a value
to a next() call.

However, if something other than 'yield' is used for coroutine
suspension -- such as a 'coyield' keyword or coyield() function --
then I think this problem becomes solvable. In a cogenerator
(i.e. a generator running in coroutine mode), 'coyield' would
do what 'yield' does in normal mode (simply suspend the frame),
and 'yield(value)' would raise StopIteration(value).

(If the latter seems unintuitive, I sympathise. It arises because
we're effectively making a cocall to the __next__ method of the
generator, and in the yield-from universe, the way a cocall
returns a value is by raising StopIteration.)

But now we need a different way for the cogenerator to signal
that it has finished iterating! I think what will need to happen
is that a cogenerator raises CoStopIteration instead of
StopIteration when it falls off the end, and the cocaller of
the cogenerator catches that and turns it into a normal
StopIteration.

Confused enough yet? I had better finish my cosandwitch and get
some more cocoffee before trying to think about this any more...

-- 
Greg


From ncoghlan at gmail.com  Thu Nov  3 02:51:10 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 3 Nov 2011 11:51:10 +1000
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <4EB12F5F.4080109@improva.dk>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz> <j8q1hf$4kq$1@dough.gmane.org>
	<CADiSq7e1LAweBTa0Z8j0gyxFyP1RXH=Rwz9-byGB39z+zVeDOA@mail.gmail.com>
	<4EB11370.8060205@canterbury.ac.nz> <4EB12F5F.4080109@improva.dk>
Message-ID: <CADiSq7c9-jrrTRaDm=gAh+B=oKiVQwVYBC3uJMFAWbCYyvtiLQ@mail.gmail.com>

On Wed, Nov 2, 2011 at 9:54 PM, Jacob Holm <jh at improva.dk> wrote:
> If the switch Nick describes is available as a flag on the frame
> objects, it would be easy to extend the traceback to show *exactly*
> where you entered the no-coroutine zone that you are now failing to
> suspend. ?I don't think the additional syntax is helpful, and it would
> be quite annoying to need to have two versions of every wrapper
> function/decorator to make it useable in both contexts.

Indeed, this is an error reporting problem, not something to be solved
by maintaining the bifurcation of the language into
"coroutine-friendly" and "normal". Again, I go back to the (lack of)
contrast between code being run in the main process thread and code
being run as part of a separate OS level thread. There, the "boundary"
between the two is covered in the way the exception is reported:

>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
Exception
>>> threading.Thread(target=f).start()
>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.2/threading.py", line 736, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.2/threading.py", line 689, in run
    self._target(*self._args, **self._kwargs)
  File "<stdin>", line 2, in f
Exception

Heck, Thread + input Queue + output Queue may be an easier to
understand conceptual model for coroutine programming than generators.
While the internals would be completely different (since the whole
point of coroutines is to avoid the OS level thread overhead), it may
help to give people the correct mental model of what is going on.

>>> def f():
...     print("Started")
...     data = cothread.suspend(1)
...     print("Resumed:", data)
...     return 42
...
>>> cf, data = cothread.cocall(f)
Started
>>> data
1
>>> cf.resume()
Resumed: None
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cothread.py", line 80, in resume
    return self._wait_for_output()
  File "cothread.py", line 66, in _wait_for_output
    raise CoroutineReturn(data.args[0])
cothread.CoroutineReturn: 42

>>> cf, data = cothread.cocall(f)
Started
>>> cf.throw(Exception)
Traceback (most recent call last):
  File "cothread.py", line 34, in run
    result = self._target(*self._args, **self._kwargs)
  File "<stdin>", line 3, in f
  File "cothread.py", line 6, in suspend
    return current.suspend(*args, **kwds)
  File "cothread.py", line 98, in suspend
    raise exc
Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cothread.py", line 89, in throw
    return self._wait_for_output()
  File "cothread.py", line 70, in _wait_for_output
    raise type(exc)(*exc.args) from exc
Exception

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cothread.py
Type: text/x-python
Size: 2982 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111103/4c6179b8/attachment.py>

From ncoghlan at gmail.com  Thu Nov  3 03:31:34 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 3 Nov 2011 12:31:34 +1000
Subject: [Python-ideas] Cofunctions - A New Protocol
In-Reply-To: <CADiSq7c9-jrrTRaDm=gAh+B=oKiVQwVYBC3uJMFAWbCYyvtiLQ@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<4EAFC8EC.4080601@canterbury.ac.nz> <j8q1hf$4kq$1@dough.gmane.org>
	<CADiSq7e1LAweBTa0Z8j0gyxFyP1RXH=Rwz9-byGB39z+zVeDOA@mail.gmail.com>
	<4EB11370.8060205@canterbury.ac.nz> <4EB12F5F.4080109@improva.dk>
	<CADiSq7c9-jrrTRaDm=gAh+B=oKiVQwVYBC3uJMFAWbCYyvtiLQ@mail.gmail.com>
Message-ID: <CADiSq7e4zFYAN72xvkzc-T2YHsbysHPixUYggmyAgRC4mZnzxA@mail.gmail.com>

On Thu, Nov 3, 2011 at 11:51 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>> def f():
> ... ? ? print("Started")
> ... ? ? data = cothread.suspend(1)
> ... ? ? print("Resumed:", data)
> ... ? ? return 42
> ...
>>>> cf, data = cothread.cocall(f)
> Started


An easier-to-read copy of the cothread module that was attached to my
previous email:
https://bitbucket.org/ncoghlan/misc/src/default/cothread.py

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From stephen at xemacs.org  Thu Nov  3 05:35:15 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 03 Nov 2011 13:35:15 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
	<87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>
Message-ID: <87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>

Mike Meyer writes:

 > No. You missed two things.

Actually, I didn't.  Containers contain; I considered that
sufficiently obvious that I neglected to mention it.  And I did
mention globals as "exposed to concurrency".  Both just expand the
list of objects needing protection, but don't change the basic
principle AFAICS.  I point this out because I hope to convince you to
concentrate on arguing *for* your proposal, instead of wasting time
scoring points *against* those who question it.  Please understand:
*nobody* is against better support for developing concurrent programs,
including those using threads to implement concurrency.  But nobody is
willing to pay arbitrarily high cost for arbitrarily small
improvements.

As for "wrapped C libraries", I'm having trouble imagining what you're
talking about.  Wrapped libraries won't be accessing data via Python
protocols anyway, so any accesses that your framework could intercept
will happen in the marshaling code.  If that creates threads, I don't
see why it wouldn't use threading.Thread.  Ditto for Python modules
implemented in C.

 > So this is a guess, but here's what I think is going on:
 > 
 > 1) We're using a test framework that creates a mock time module for
 > some reason. At some point, the mock object has the value None.
 > 2) The application being tested  uses a database module that uses
 > threads as part of managing a connection pool.
 > 
 > The concurrency unsafe codde in the test framework (which is clearly
 > and obviously single-threaded, right?)

Again, I don't understand.  In the von Neumann model of computation,
all *code* is single-threaded pretty much by definition.  With
preemptively scheduled threads, the process executing that code is
single-threaded if and only if it blocks until all other threads exit.

The test framework is "obviously" multiple-threaded in the scenario
you describe.

 > managed to get the None-valued mock inserted in sys.modules due to
 > a concurrency bug.

I don't see your point.  You claim your proposal is supposed to help
identify which objects are shared.  Taking that literally, this
example is useless as an argument for your framework -- you already
know which object is the problem.  Of course, I suppose you actually
meant to get more precise information in cases like this, specifically
about in what execution contexts the object is mutated.  But in this
case, you probably already know precisely where in the code the
offending mutation happens.  The bug is that execution of the
framework code resumes "unexpectedly" before the "subordinate" thread
restores the problem object's state to status quo ante.  Well, these
are *threads*, so you can't say anything about when to expect them to
resume, you have to expect resumption at any point.

*Now* what does your framework buy you?  If the answer is "nothing,"
you need to find better use cases to motivate your proposal.




From ncoghlan at gmail.com  Thu Nov  3 05:48:56 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 3 Nov 2011 14:48:56 +1000
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
	<87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>
	<87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7cA1UyS9Qs5N=+-QDOF+abQXYxRMygE=9CYEXMqPaqjwA@mail.gmail.com>

On Thu, Nov 3, 2011 at 2:35 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Mike Meyer writes:
> ?> managed to get the None-valued mock inserted in sys.modules due to
> ?> a concurrency bug.
>
> I don't see your point. ?You claim your proposal is supposed to help
> identify which objects are shared. ?Taking that literally, this
> example is useless as an argument for your framework -- you already
> know which object is the problem. ?Of course, I suppose you actually
> meant to get more precise information in cases like this, specifically
> about in what execution contexts the object is mutated. ?But in this
> case, you probably already know precisely where in the code the
> offending mutation happens. ?The bug is that execution of the
> framework code resumes "unexpectedly" before the "subordinate" thread
> restores the problem object's state to status quo ante. ?Well, these
> are *threads*, so you can't say anything about when to expect them to
> resume, you have to expect resumption at any point.

This discussion is actually starting to sound a bit like the Python
security model discussions to me - ensuring that only certain kinds of
objects are reachable from a given point in the code, and controlling
the operations that can be performed on them.

Victor Stinner eventually managed to create his pysandbox module based
on those discussions, which does a fairly good job of locking things
down, even though I'd personally still be inclined to back it up with
additional OS level defences.

Still, like the GIL vs free threading discussions, there are unlimited
numbers of things that *could* be done, the question is what is
*worth* doing. And that's not going to be figured out in a mailing
list discussion - it's going to take real code and plenty of
experimentation, and it's going to have be done by people that don't
already believe the "right" answer is to use message queues and a "no
concurrent access to mutable objects" architecture at the application
level (i.e. where what threads buy you is the fact that you can hand
over an object just by putting the reference in a queue and dropping
yours, without any serialisation overhead).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ron3200 at gmail.com  Thu Nov  3 07:15:30 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Thu, 03 Nov 2011 01:15:30 -0500
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <4EB1F2BD.7070202@canterbury.ac.nz>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CACac1F_YoJmm3HSQtiMfy1iHNgBipO1iHSjcbJ8tQru0TvL9Ow@mail.gmail.com>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy> <4EB1F2BD.7070202@canterbury.ac.nz>
Message-ID: <1320300930.10353.78.camel@Gutsy>

On Thu, 2011-11-03 at 14:47 +1300, Greg Ewing wrote:
> On 03/11/11 05:21, Ron Adam wrote:
> 
> > Would it be possible to rewrite the 'yield' internals so they work in
> > the following way...
> >
> >       # a = yield b
> >       try:
> >            raise SuspendException(b, _self=_self)
> >       Except ContinueException as exc:
> >            a = exc.args
> 
> I've just been thinking about something like that, while pondering
> whether there is a middle ground somewhere between the yield-from
> mechanism and a completely new coroutine protocol.

I was thinking if we could substitute an alternative spelling like that,
then it gives us a bit more control on how to interact with other outer
frameworks.  If the internals can be done that way, then it may open up
more options in python generators.  So instead of a completely new
coroutine proticol, we have better tools for others to create their own
frameworks and proticols.

> The problem I had with building all of it on yield-from was that
> there was no way to distinguish a 'yield' being used for the purpose
> of suspending a coroutine from one being used to return a value
> to a next() call.

Right.  The obvious way is just to add a second of everything.
   next2()
   .send2()
   .throw2()
   yield2

I don't think that is the best way.


> However, if something other than 'yield' is used for coroutine
> suspension -- such as a 'coyield' keyword or coyield() function --
> then I think this problem becomes solvable. In a cogenerator
> (i.e. a generator running in coroutine mode), 'coyield' would
> do what 'yield' does in normal mode (simply suspend the frame),
> and 'yield(value)' would raise StopIteration(value).

Well it sounds reasonable,  but how would that actually work?  What if
the coroutine is paused at coyield, and you need to do a next rather
than a conext?  And also in the case of it being the othe way round.

> (If the latter seems unintuitive, I sympathise. It arises because
> we're effectively making a cocall to the __next__ method of the
> generator, and in the yield-from universe, the way a cocall
> returns a value is by raising StopIteration.)

> But now we need a different way for the cogenerator to signal
> that it has finished iterating! I think what will need to happen
> is that a cogenerator raises CoStopIteration instead of
> StopIteration when it falls off the end, and the cocaller of
> the cogenerator catches that and turns it into a normal
> StopIteration.
> 
> Confused enough yet? I had better finish my cosandwitch and get
> some more cocoffee before trying to think about this any more...

And that is the whole problem... trying to make this all un_coconfusing
to the average python programmer.  If it's coconfusing to us, they don't
have a chance.  ;-)

Hmm... I have a craving for some hot co_co now.

I've been poking around in genobject.c, frameobject.c, and ceval.c, to
try to get a handle on just how it all fits together.

One of the odd things is a throw() done before a generator is started
rasies an excption at he first line, where it has no chance to act on
it.  And it also doesn't propagate back out.  So the first thing I'm
trying (to help me learn the C code better/again) is to see if I can get
it to ignore an exception thrown in at that state.  And then see if I
can make that specific to just g.throw(ContinueException).

That way I don't have to pre_start the generators if I'm using
throw(ContinueException) as my scheduler interface.

It's a start. ;-)

Cheers,
   Ron




From stephen at xemacs.org  Thu Nov  3 08:39:45 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 03 Nov 2011 16:39:45 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CADiSq7cA1UyS9Qs5N=+-QDOF+abQXYxRMygE=9CYEXMqPaqjwA@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
	<87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>
	<87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CADiSq7cA1UyS9Qs5N=+-QDOF+abQXYxRMygE=9CYEXMqPaqjwA@mail.gmail.com>
Message-ID: <87vcr1odry.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > This discussion is actually starting to sound a bit like the Python
 > security model discussions to me - ensuring that only certain kinds of
 > objects are reachable from a given point in the code, and controlling
 > the operations that can be performed on them.

That's where I'm going, yes.  I don't think that's what Mike has in
mind, though, and I'd like to understand why he thinks it's insufficient.



From bruce at leapyear.org  Thu Nov  3 08:39:17 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Thu, 3 Nov 2011 00:39:17 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
Message-ID: <CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>

Here are issues to think about. You wrote:

'locking' value [',' value]*':' suite

The list of values are the objects that can be mutated in this lock. An
immutable object showing up in the list of values is a TypeError.


However:

(1) Greg Ewing gave an example sort of like this:

new_balance = balance + deposit
lock(balance)
balance = new_balance
unlock(balance)


and pointed out that the locks don't help. He was talking about the race
condition on reading balance before writing. But it's worse than that. If
these are numbers, then they're immutable and locking them does nothing and
this isn't any better:

 lock(balance)
new_balance = balance + deposit
balance = new_balance
unlock(balance)


Consider this operating on lists:

locking stuff:  #1
    stuff += added_stuff

locking stuff:  #2
    stuff = stuff + added_stuff


In #2, locking is completely useless. Sure the values are mutable but I'm
not mutating them. Is it obvious that #1 works and #2 doesn't? You don't
want to lock the *value* of stuff, you want to lock the *variable*, i.e.,
locking globals() or wherever the value of stuff is found. Locking
globals() seems like a bad idea.  Which brings me to...

(2) How does locking something like a dictionary work? Does it lock the
dictionary and all the values in it, walking the entire data structure to
lock it? Ypu suggest that's the case and that seems like a performance
nightmare. Given

x = [1, 2]
d = {3: x, 4: 5}


How do I lock d[3]? That is, I want to lock the dictionary slot where the
value of d[3] is stored so another thread can't do d[3] = 0. If I write

locking d[3]: # do stuff


that's going to lock the value of x. Another thread won't be able to do
x.append(0) but they would be able to do x = 0 or d[3] = 0. If I have to
write

locking d: # do stuff


that hampers concurrency. If I can lock the dictionary slot d[3], can I
lock list slots? After all, the compiler doesn't know they type of d. How
do I lock just an attribute without locking the whole object?

(3) What happens if one thread does:

locking d, x: # stuff

and another does


locking x, d: # stuff

I think I know the answer and it's "deadlock". Avoiding deadlock is an
important problem and by forcing me to lock every single object before I
mutate it the important locks (the ones for objects that are actually
shared) will be lost among the many that are unnecessary, making it very
difficult to find bad lock ordering.

(4) What if the lock can't be acquired right away? Maybe there's other
stuff my thread can do while it's waiting for the lock. You don't consider
that. Maybe I have a whole bunch of things all of which can be done in any
order.

(5) You haven't thought about read vs. write locks. If I'm passed an
iterable or a sequence in a concurrent program, I want to read lock it so
no one else changes it while I'm working with it. But you prohibit locking
immutable objects, so I first have to find out if it's immutable which is
something else you'll have to add to the language.


On Mon, Oct 31, 2011 at 10:32 PM, Mike Meyer <mwm at mired.org> wrote:

> I've identified the problem I want to solve: I want to make concurrent
> use of python objects "safe by default", ...
>

To me it looks like this proposal deals with a small part of the problem
with the equivalent of a sledgehammer. When I said identify the problem,
the above issues are more on the lines of what I was talking about, not a
general statement "make concurrency better."

But as an overall goal, I think this is better: "make it as easy as
possible to write error-free concurrent code." I would think that "without
making it hard to write non-concurrent code" is implicit but I'll spell
that out since I've heard that explicit is better than implicit.

And here are some things people writing concurrent programs want to do (in
no particular order):

(1) lock an object (even when the object doesn't have any code to support
locking)
(2) lock part of an object (a slot in list or dictionary, or an attribute)
- consider that databases support row and rang elocking because if the only
thing you can do is lock entire tables, you're going to get poor performance
(3) lock multiple things at the same time
(4) queue operations to be performed on an object when it can be locked,
which requires ...
(5) wait until a queued operation completes, which requires ...
(6) avoid starvation
(7) avoid deadlock
(8) avoid other concurrency bugs
(9) avoid debugging (not avoid it when it's necessary but do things to
avoid it being necessary)


... so that doing unsafe things
> causes the programmer to have to do something explicit about making
> things safe.  I believe this can be done at the mutation points
> (again, clojure shows that it can be done). I also want to preserve as
> much of Python's existing code as possible. It may be that Python's
> existing data structures mean my believe about mutation points is
> wrong. This may be the wrong solution. It may be that such a change is
> to large to be acceptable.  But the only way to find out is to
> investigate it.
>

Clojure is very different from Python. Core data structures are immutable
in Clojure so adding to a list creates a new also immutable list. Changing
Python's core data structures to be immutable would be a
bigger compatibility break than 2 to 3.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111103/c565a359/attachment.html>

From ubershmekel at gmail.com  Thu Nov  3 11:02:20 2011
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Thu, 3 Nov 2011 12:02:20 +0200
Subject: [Python-ideas] Testing with numbers
Message-ID: <CANSw7KyK+pME=K2NWmFf0siz6=73FeQRaJfivg34hfsF07HJxg@mail.gmail.com>

Working on algorithms that don't have just the boolean states of
"success" or "failure", I find myself wanting to have regular unit
tests adjacent to where I put my scalar result of how well the
algorithm worked. E.g. the percentage of test samples failed, an x
minus y squared sum, etc.

As I understand it unittest doesn't have the facilities for such test
cases or results.

One could use?assertGreater to?test for passing a threshold but
usually the acceptable numbers are rapidly changing. When running a
test, you'd want to see the resulting number instead of an F/E/.
result.

Another use case is pypy speed benchmarks.

Are there others here who would/could use this?

--Yuval Greenfield


From ubershmekel at gmail.com  Thu Nov  3 14:10:23 2011
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Thu, 3 Nov 2011 15:10:23 +0200
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
References: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
Message-ID: <CANSw7KwW2h6HvjsrwEpWpZOOkYMujtUa6hy3GOV1kY_yuK=JRA@mail.gmail.com>

turtle isn't thread safe.

http://bugs.python.org/issue1702036

Also, here's just a random exception:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import turtle
>>> turtle.forward(10)
>>> import turtle
>>> from threading import Thread
>>> class walker(Thread):
...     def run(self):
...             for i in range(100):
...                     turtle.forward(10)
...                     turtle.left(10)
...
>>> [walker().start() for i in range(5)]
[None, None, None, None, None]
>>> Exception in thread Thread-2:
Traceback (most recent call last):
  File "c:\python32\lib\threading.py", line 736, in _bootstrap_inner
    self.run()
  File "<stdin>", line 4, in run
  File "<string>", line 1, in forward
  File "c:\python32\lib\turtle.py", line 1637, in forward
    self._go(distance)
  File "c:\python32\lib\turtle.py", line 1605, in _go
    self._goto(ende)
  File "c:\python32\lib\turtle.py", line 3159, in _goto
    screen._pointlist(self.currentLineItem),
  File "c:\python32\lib\turtle.py", line 755, in _pointlist
    cl = self.cv.coords(item)
  File "<string>", line 1, in coords
  File "c:\python32\lib\tkinter\__init__.py", line 2162, in coords
    self.tk.call((self._w, 'coords') + args))]
  File "c:\python32\lib\tkinter\__init__.py", line 2160, in <listcomp>
    return [getdouble(x) for x in
ValueError: could not convert string to float: 'itemconfigure'



On Wed, Nov 2, 2011 at 9:36 PM, Mike Meyer <mwm at mired.org> wrote:
> In order to get a better idea of where things stand, I'd like to get
> answers to a few questions. This isn't a traditional broadbased
> survey, but an attempt to get answers from a few people who might know
> or have good ideas. This is probably where I should have started, but
> better late than never.
>
> 1) How much of the Python standard library is known to be thread safe?
>
> 2) How many packages in PyPI are known to be thread safe?
>
> 3) Can you suggest another approach to getting safe high-performance
> shared data in concurrent operation? I've already considered:
>
> ?a) I proposed making actions that mutate data require locked objects,
> because I've seen that work in other languages. I recognize that
> doesn't mean it will work in Python, but it's more than I can say
> about the alternatives I knew about then.,
>
> ?b) Bertrand Meyer's SCOOPS system, designed for Eiffel. It has two
> major strikes against it: 1) it is based on type attributes on
> *variables*, andI could figure out how to translate that to a language
> where variables aren't typed. 2) I don't know that there's a working
> implementation.
>
> 4) Can you suggest a minor change that would move things toward safer
> concurrent code with high-performance shared data? I can see two
> possibilities:
>
> ?a) Audit any parts of the standard library that aren't already known
> to be thread safe, and flag those that aren't. ?Fixing them may want
> to wait on a better mechanism than posix locks.
>
> ?b) Add a high-level, high-performance shared object facility to the
> multiprocess package.
>
> ? ?Thanks,
> ? ?<mike
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From solipsis at pitrou.net  Thu Nov  3 14:39:27 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 3 Nov 2011 14:39:27 +0100
Subject: [Python-ideas] A concurrency survey of sorts
References: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
Message-ID: <20111103143927.3a9bfe0a@pitrou.net>

On Wed, 2 Nov 2011 12:36:26 -0700
Mike Meyer <mwm at mired.org> wrote:
> In order to get a better idea of where things stand, I'd like to get
> answers to a few questions. This isn't a traditional broadbased
> survey, but an attempt to get answers from a few people who might know
> or have good ideas. This is probably where I should have started, but
> better late than never.
> 
> 1) How much of the Python standard library is known to be thread safe?

It depends what the thread safety assumptions are.
I'd say not much of it, but not much of it *needs* to either.
For example, if you mutate the same XML tree concurrently,
my opinion is that the problem is with your code, not the stdlib :-)
(on the other hand, if mutating *different* XML trees concurrently
produces errors, then it's a stdlib bug)

Buffered binary file objects are known to be thread-safe.
Text file objects are not, except perhaps for writing (I think we did
the latter, because of print() and logging; I'm not sure it's well
tested, though).
Raw file objects are not (they are "raw" after all: they simply expose
the OS' behaviour).

As a separate issue, binary file objects forbid reentrant accesses from
signal handlers. Therefore I would advocate against using print() in a
signal handler.
(see http://bugs.python.org/issue10478)

>  b) Add a high-level, high-performance shared object facility to the
> multiprocess package.

It will be difficult (IMHO: very difficult) to devise such a thing.
multiprocessing already has shared memory facilities, though - but they
are very low-level.

Regards

Antoine.




From mwm at mired.org  Thu Nov  3 17:38:26 2011
From: mwm at mired.org (Mike Meyer)
Date: Thu, 3 Nov 2011 09:38:26 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
Message-ID: <CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>

On Thu, Nov 3, 2011 at 12:39 AM, Bruce Leban <bruce at leapyear.org> wrote:
> Here are issues to think about. You wrote:
>
> 'locking' value [',' value]*':' suite
> The list of values are the objects that can be mutated in this?lock. An
> immutable object showing up in the list of values is a?TypeError.
>
> However:
> (1) Greg Ewing gave an example sort of like this:
>
> new_balance = balance + deposit
> lock(balance)
> balance = new_balance
> unlock(balance)
>
> and pointed out that the locks don't help


Yup. You can't keep  people from writing code that's just wrong. But
they can't get it right if they don't add any locking at all.

> lock(balance)
> new_balance = balance + deposit
> balance = new_balance
> unlock(balance)

This case would still throw an exception, because what needs to be
locked isn't balance, but whatever balance is an attribute of. Unless
it's a local variable, in which case it doesn't need to be locked.
Given the code as is, balance needing to be locked would make it
global, so you'd lock the module. A more realistic implementation
would be if balance were self.balance, in which case you'd lock self.

> Consider this operating on lists:
>
> locking stuff: ?#1
> ? ? stuff += added_stuff
> locking stuff: ?#2
> ? ? stuff = stuff + added_stuff
>
> In #2, locking is completely useless. Sure the values are mutable but I'm
> not mutating them. Is it obvious that #1 works and #2 doesn't? You don't
> want to lock the *value* of stuff, you want to lock the *variable*, i.e.,
> locking globals() or wherever the value of stuff is found.

Right. Possibly "value" was the wrong choice for a placeholder in the
original description, because what you're actually locking is a
container of some sort.

> Locking globals() seems like a bad idea.??Which brings me to...
> (2) How does locking something like a dictionary work? Does it lock the
> dictionary and all the values in it, walking the entire data structure to
> lock it? Ypu suggest that's the case and that seems?like a performance
> nightmare.

If I suggested that, then it was unintentional. Yes, it's a
performance nightmare. Locking a container just locks for changes to
set of contained objects. It's analogous to tuples being immutable:
you can't change the set of objects in the tuple, but if those objects
are mutable, you can change them. If you want to change an object in a
list/dictionary/etc - then you have to lock that object, but not the
dictionary.

Given
>
> x = [1, 2]
> d = {3: x, 4: 5}
>
> How do I lock d[3]? That is, I want to lock the dictionary slot where the
> value of d[3] is stored so another thread can't do d[3] = 0. If I write
>
> locking d[3]: # do stuff
>
> that's going to lock the value of x. Another thread won't be able to do
> x.append(0) but they would be able to do x = 0 or d[3] = 0. If I have to
> write
>
> locking d: # do stuff
>
> that hampers concurrency.

Yes, it does. But so does any form of locking.

This proposal locks objects. Changes that don't change the object -
even if they might change it's value by changing a contained object -
aren't locked, because of the expense you pointed out. If there's some
different level of granularity for locking that makes sense, I don't
know what it is.

> If I can lock the dictionary slot d[3], can I lock
> list slots? After all, the compiler doesn't know they type of d.?How do I
> lock just an attribute without locking the whole object?

Attributes are names. You don't lock names, you lock objects. To
prevent binding of an attribute, you lock the object it's an attribute
of. If you want to prevent mutating the value of the object bound to
the attribute, you lock that object.

> (3) What happens if one thread does:
>
> locking d, x: # stuff
>
> and another does
>
> locking x, d: # stuff

That one I dealt with in the specification. There's an implementation
requirement that if two locking statements share objects, the shared
objects have to be locked in the same order. So one of the two will
lock things in the opposite of the order they appear in the statement.

> I think I know the answer and it's "deadlock".

No, avoiding deadlock is one of the goals. That's why that requirement
exists, and there's a further requirement that you can only nest
locking statements if the inner one locks a subset of the outer one.

I have as yet to work out how an automatic STM version (this wasn't in
the original proposal) would interact here.

    <mike


From mikegraham at gmail.com  Thu Nov  3 18:26:13 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Thu, 3 Nov 2011 13:26:13 -0400
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
References: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
Message-ID: <CAEBZo3OKntA6_x9CRomh77xi5TJfdUA5oN08006US0iGPP178g@mail.gmail.com>

On Wed, Nov 2, 2011 at 3:36 PM, Mike Meyer <mwm at mired.org> wrote:
> 1) How much of the Python standard library is known to be thread safe?
>
> 2) How many packages in PyPI are known to be thread safe?

"Thread safe" isn't nearly as well-defined as many people act, and
certainly doesn't mean it's safe to use something with threads. When
people try to use the very, very, very few things that are thread safe
without their own synchronization, they almost always end up with
buggy code.

It's also worth noting that many of the most important
concurrency-supporting packages in PyPI don't use multithreading at
all.

> 3) Can you suggest another approach to getting safe high-performance
> shared data in concurrent operation? I've already considered:
>
>  a) I proposed making actions that mutate data require locked objects,
> because I've seen that work in other languages. I recognize that
> doesn't mean it will work in Python, but it's more than I can say
> about the alternatives I knew about then.,

I don't see how this is feasible or makes Python a better language.
This would add complication that doesn't benefit lots of people, would
slow down normal cases, and wouldn't solve the datasharing problem for
important cases that aren't just sharing memory between threads.

>  b) Bertrand Meyer's SCOOPS system, designed for Eiffel. It has two
> major strikes against it: 1) it is based on type attributes on
> *variables*, andI could figure out how to translate that to a language
> where variables aren't typed. 2) I don't know that there's a working
> implementation.

I don't mean to be rude, but I don't understand how this is an idea at all.

We already have lot of tools for sharing data predictably among
threads, concurrent tasks, processes, and machines: Queue.Queue,
thread locks, callbacks, MPI, message queues, and databases to name a
few. Each of these has disadvantages and most of these have
advantages.

> 4) Can you suggest a minor change that would move things toward safer
> concurrent code with high-performance shared data? I can see two
> possibilities:
>
>  a) Audit any parts of the standard library that aren't already known
> to be thread safe, and flag those that aren't.  Fixing them may want
> to wait on a better mechanism than posix locks.

I am not convinced adding this at a language level is net good at all.
Flagging things as "thread unsafe" is silly, as practically everything
is thread unsafe. Flagging things as "thread safe" is seldom useful,
because you should still be handling synchronization in your code.
Creating locks on everything in the stdlib would make Python bigger,
more complex, and slower and still not solve concurrency problems for
users ? indeed, it could make them less apparent.

And none of this goes to address concurrency that isn't based on
multithreading, which is important and in many, many applications
preferable.

>  b) Add a high-level, high-performance shared object facility to the
> multiprocess package.

The multiprocessing module already provides means to pass data which
are fairly implicit. Trying to encapsulate the shared state as a
Python object would be even more troublesome.

Mike


From greg.ewing at canterbury.ac.nz  Thu Nov  3 22:35:47 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 04 Nov 2011 10:35:47 +1300
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
Message-ID: <4EB30933.7090808@canterbury.ac.nz>

Mike Meyer wrote:

> This case would still throw an exception, because what needs to be
> locked isn't balance, but whatever balance is an attribute of.

All right, suppose we have an "account" object that
holds the balance. Now we can do

    locking account:
       account.balance = account.balance + deposit

That's okay. But what if there are two accounts
involved, and we want to do

    account1.balance = account1.balance - amount
    account2.balance = account2.balance + amount

These two operations need to be done atomically,
but there is no single object to lock, so we have
to lock both of them -- and then there is an
opportunity for deadlock if two threads do this
in different orders.

> and there's a further requirement that you can only nest
> locking statements if the inner one locks a subset of the outer one.

Then it's *impossible* to solve the above problem,
because neither of the objects needing to be locked
is contained in the other.

The notion of containment is not even well defined
in general, because Python objects can form arbitrary
graphs.

-- 
Greg


From mwm at mired.org  Thu Nov  3 22:57:36 2011
From: mwm at mired.org (Mike Meyer)
Date: Thu, 3 Nov 2011 14:57:36 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <4EB30933.7090808@canterbury.ac.nz>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
	<4EB30933.7090808@canterbury.ac.nz>
Message-ID: <CAD=7U2CG19ir7mU=7W7Py4b8mpuY5igPc6rzWTcqyHLR7O83_g@mail.gmail.com>

On Thu, Nov 3, 2011 at 2:35 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Mike Meyer wrote:
>
>> This case would still throw an exception, because what needs to be
>> locked isn't balance, but whatever balance is an attribute of.
>
> All right, suppose we have an "account" object that
> holds the balance. Now we can do
>
> ? locking account:
> ? ? ?account.balance = account.balance + deposit
>
> That's okay. But what if there are two accounts
> involved, and we want to do
>
> ? account1.balance = account1.balance - amount
> ? account2.balance = account2.balance + amount
>
> These two operations need to be done atomically,
> but there is no single object to lock, so we have
> to lock both of them -- and then there is an
> opportunity for deadlock if two threads do this
> in different orders.

That's why the proposal specifies that locking statements determine
the order of the locking, and require that any two locking statements
lock all objects they have in common in the same order - at least
during one run of the application. So the above is done by:

    locking account1, account:
        # stuff

If a second locking statement does "locking account2, foobar,
account1", then account1 and account2 will be locked in the same order
by both statements.

>> and there's a further requirement that you can only nest
>> locking statements if the inner one locks a subset of the outer one.
> Then it's *impossible* to solve the above problem,
> because neither of the objects needing to be locked
> is contained in the other.

I wasn't clear enough. The sets I'm talking about are the objects
being locked, not anything they may contained, in order to prevent
deadlocks. So if you start by doing:

    locking account1, account2, foobar:

and then later on - but with those locks still held - do

    locking account1, account2:

things work fine (the second locking is a nop). However, if you do
(under the same condition):

    locking account1, aardvark:

you get an exception - the outer lock has to lock aardvark as well.
And yes, the implications still need to be worked out.

> The notion of containment is not even well defined
> in general, because Python objects can form arbitrary
> graphs

Containment has a very specific meaning for this specification, in
that an object only contains things it has a direct reference to. That
should be spelled out, or maybe I need to use a better word.

   <mike


From tjreedy at udel.edu  Thu Nov  3 23:39:42 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 03 Nov 2011 18:39:42 -0400
Subject: [Python-ideas] Testing with numbers
In-Reply-To: <CANSw7KyK+pME=K2NWmFf0siz6=73FeQRaJfivg34hfsF07HJxg@mail.gmail.com>
References: <CANSw7KyK+pME=K2NWmFf0siz6=73FeQRaJfivg34hfsF07HJxg@mail.gmail.com>
Message-ID: <j8v57q$uno$1@dough.gmane.org>

On 11/3/2011 6:02 AM, Yuval Greenfield wrote:
> Working on algorithms that don't have just the boolean states of
> "success" or "failure", I find myself wanting to have regular unit
> tests adjacent to where I put my scalar result of how well the
> algorithm worked. E.g. the percentage of test samples failed, an x
> minus y squared sum, etc.

If I were using unittest for such, where I did not want to make each 
test sample an individual test but wanted to report a collective result, 
I would report the percentage (or fraction) and test equality to 100 (or 
1). The test would always fail (until it passed ;-), but the actual 
fraction would be printed, and could be compared to the current target.

-- 
Terry Jan Reedy



From bruce at leapyear.org  Fri Nov  4 02:51:40 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Thu, 3 Nov 2011 18:51:40 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
Message-ID: <CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>

On Thu, Nov 3, 2011 at 9:38 AM, Mike Meyer <mwm at mired.org> wrote:

> Attributes are names. You don't lock names, you lock objects. To
> prevent binding of an attribute, you lock the object it's an attribute
> of. If you want to prevent mutating the value of the object bound to
> the attribute, you lock that object.


You conveniently ignore my comment about the importance of database row
locking. :-)


>  > (3) What happens if one thread does:
> >
> > locking d, x: # stuff
> >
> > and another does
> >
> > locking x, d: # stuff
>
> That one I dealt with in the specification. There's an implementation
> requirement that if two locking statements share objects, the shared
> objects have to be locked in the same order. So one of the two will
> lock things in the opposite of the order they appear in the statement.
>

You're handwaving. How would that work? There's no way to know at compile
time what the correct order is. So locking just got a whole lot more
complicated. But it's worse than complicated: it's unworkable.

No, avoiding deadlock is one of the goals. That's why that requirement
> exists, and there's a further requirement that you can only nest
> locking statements if the inner one locks a subset of the outer one.
>

In other words, this code is not allowed:

def f():
    locking x:
        for y in x:
            locking y:
                y.append(1)


And neither is this:

def f(x):
    locking x:
        x.append(1)


if it is called inside code that has something locked. Which virtually all
code will do since it's required everywhere. So your proposal is that I
must lock everything before I change it but I can't lock it unless it's
already locked. That's ridiculous. You might as well have a single
semaphore to lock everything.

We're wasting our time. You said above "You don't lock names, you lock
objects." Why not? Because that was your original plan and your mind is
closed to other ideas? You want to ignore legitimate issues that aren't
convenient to your proposal. I hate to say things like "this idea sucks,"
but sometimes that's the only way to put it. I think that this proposal is
a terrible idea. It's pointless to argue about the details of something
this bad. A healthy discussion about how to make concurrency better might
be interesting but as long as all you want to do is defend your proposal,
this thread isn't sparking useful discussion. Sorry to be so blunt but I
don't see any other way to get the message across.

--- Bruce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111103/0b87ce55/attachment.html>

From python at mrabarnett.plus.com  Fri Nov  4 03:23:02 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 04 Nov 2011 02:23:02 +0000
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
	<CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
Message-ID: <4EB34C86.10404@mrabarnett.plus.com>

On 04/11/2011 01:51, Bruce Leban wrote:
>
> On Thu, Nov 3, 2011 at 9:38 AM, Mike Meyer <mwm at mired.org
> <mailto:mwm at mired.org>> wrote:
>
>     Attributes are names. You don't lock names, you lock objects. To
>     prevent binding of an attribute, you lock the object it's an attribute
>     of. If you want to prevent mutating the value of the object bound to
>     the attribute, you lock that object.
>
>
> You conveniently ignore my comment about the importance of database row
> locking. :-)
>
>      > (3) What happens if one thread does:
>      >
>      > locking d, x: # stuff
>      >
>      > and another does
>      >
>      > locking x, d: # stuff
>
>     That one I dealt with in the specification. There's an implementation
>     requirement that if two locking statements share objects, the shared
>     objects have to be locked in the same order. So one of the two will
>     lock things in the opposite of the order they appear in the statement.
>
>
> You're handwaving. How would that work? There's no way to know at
> compile time what the correct order is. So locking just got a whole lot
> more complicated. But it's worse than complicated: it's unworkable.
>
[snip]

If they're threads running in the same address space in CPython then
they could lock always in order of increasing address. In an
implementation of Python which uses references they could lock always
in increasing numeric value of reference (if that's possible).

This would not, of course, protect against general nested locks.


From jimjjewett at gmail.com  Fri Nov  4 03:30:18 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Thu, 3 Nov 2011 22:30:18 -0400
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
References: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
Message-ID: <CA+OGgf4VXtU5m5N6X8-4en4F2krsMEL_uR3NawZR3jjwdyP4eA@mail.gmail.com>

On Wed, Nov 2, 2011 at 3:36 PM, Mike Meyer <mwm at mired.org> wrote:

> 1) How much of the Python standard library is known to be thread safe?

None.  Though our confidence in the threading library is fairly high
(except when the underlying C library is broken).

Not so long ago, there were a series of changes to the regression
tests that boiled down getting rid of spurious failures caused by
tests running serially, but in an unusual order.  If that level of
separation was still new, then finer-grained parallelism can't really
be expected to work either.

That said, test cases relied far more on global state than a typical
module itself does, so problems are far more likely to occur in user
code than in the library.


> ?a) I proposed making actions that mutate data require locked objects,
> because I've seen that work in other languages. I recognize that
> doesn't mean it will work in Python, but it's more than I can say
> about the alternatives I knew about then.,

If you really want to do this, you should probably make the changes at
the level of "object" (or "type") and inherit them everywhere.  And it
may simplify things to also change the memory allocation.

There are a few projects for remote objects that already use a
different memory model to enforce locking; you could start there.

> ?b) Bertrand Meyer's SCOOPS system, designed for Eiffel. It has two
> major strikes against it: 1) it is based on type attributes on
> *variables*, andI could figure out how to translate that to a language
> where variables aren't typed.

Actually, that isn't so bad.  Common Lisp doesn't normally type
variables at the source code level, but
    (a)  You can explicitly add typing information if you want to, and
    (b)  The compiler can often infer types

If you want this to mesh with python, the constraints are similar; not
only does the locking and safety marking have to be unobtrusive, it
probably has to be optional.  And there is existing (if largely
superseded by Pypy) work on type inference for variables.

-jJ


From stephen at xemacs.org  Fri Nov  4 04:53:34 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 04 Nov 2011 12:53:34 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <4EB34C86.10404@mrabarnett.plus.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
	<CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
	<4EB34C86.10404@mrabarnett.plus.com>
Message-ID: <87pqh8o85d.fsf@uwakimon.sk.tsukuba.ac.jp>

MRAB writes:
 > On 04/11/2011 01:51, Bruce Leban wrote:

 > >      > (3) What happens if one thread does:
 > >      >
 > >      > locking d, x: # stuff
 > >      >
 > >      > and another does
 > >      >
 > >      > locking x, d: # stuff
 > >
 > >     That one I dealt with in the specification. There's an implementation
 > >     requirement that if two locking statements share objects, the shared
 > >     objects have to be locked in the same order. So one of the two will
 > >     lock things in the opposite of the order they appear in the statement.
 > >
 > > You're handwaving. How would that work?
 > 
 > If they're threads running in the same address space in CPython then
 > they could lock always in order of increasing address.

OK, I think that works.

 > In an implementation of Python which uses references they could
 > lock always in increasing numeric value of reference (if that's
 > possible).

That doesn't make sense to me; if it's not an address, how do you know
it's unique (you know the object is unique, of course, but there may
be other references)?  If the reference is not unique, how do you know
that some other thread hasn't locked it via a different reference?

I think for this to be workable you would need to use something like a
lock tick that would increase with each lock taken, and store that in
the object.  Then you would lock the objects in order of their ticks,
with unlocked objects coming last.  (You'd have to be careful about
the wraparound, though, and that could double or triple the
performance hit for incrementing the tick and determining lock order.)


From jimjjewett at gmail.com  Fri Nov  4 06:18:52 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Fri, 4 Nov 2011 01:18:52 -0400
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
	<87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>
	<87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CA+OGgf5SSq8xD3aT_SRDDwrMZ8YVbeUw3PKT6ndt7Ud1U4CUvg@mail.gmail.com>

On Thu, Nov 3, 2011 at 12:35 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:

> As for "wrapped C libraries", I'm having trouble imagining what you're
> talking about.

C code (currently) can create or modify python objects using a C
pointer, instead of python access.  That is the biggest barrier to a
tracing (as opposed to reference-counting) garbage collector.  It is
also a problem for security sandboxes (though they can just ban C
extensions).  Most relevant here, C extensions can also bypass any
locks or protections put in place for concurrency.

-jJ


From adam.jorgensen.za at gmail.com  Fri Nov  4 08:03:11 2011
From: adam.jorgensen.za at gmail.com (Adam Jorgensen)
Date: Fri, 4 Nov 2011 09:03:11 +0200
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CA+OGgf4VXtU5m5N6X8-4en4F2krsMEL_uR3NawZR3jjwdyP4eA@mail.gmail.com>
References: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
	<CA+OGgf4VXtU5m5N6X8-4en4F2krsMEL_uR3NawZR3jjwdyP4eA@mail.gmail.com>
Message-ID: <CAMX1Ron0vUnjQ7f+5zZ=zWprLuRgCCbqd6a0zWM3tXwfK2bs7w@mail.gmail.com>

Threads are unsafe, period.

Personally, I think the threading packages should be removed from Python
entirely.

The GIL makes them pseudo-pointless in CPython anyway and the headaches
arising from
threading are very frustrating.

Personally I would rather see an Actors library...

On 4 November 2011 04:30, Jim Jewett <jimjjewett at gmail.com> wrote:

> On Wed, Nov 2, 2011 at 3:36 PM, Mike Meyer <mwm at mired.org> wrote:
>
> > 1) How much of the Python standard library is known to be thread safe?
>
> None.  Though our confidence in the threading library is fairly high
> (except when the underlying C library is broken).
>
> Not so long ago, there were a series of changes to the regression
> tests that boiled down getting rid of spurious failures caused by
> tests running serially, but in an unusual order.  If that level of
> separation was still new, then finer-grained parallelism can't really
> be expected to work either.
>
> That said, test cases relied far more on global state than a typical
> module itself does, so problems are far more likely to occur in user
> code than in the library.
>
>
> >  a) I proposed making actions that mutate data require locked objects,
> > because I've seen that work in other languages. I recognize that
> > doesn't mean it will work in Python, but it's more than I can say
> > about the alternatives I knew about then.,
>
> If you really want to do this, you should probably make the changes at
> the level of "object" (or "type") and inherit them everywhere.  And it
> may simplify things to also change the memory allocation.
>
> There are a few projects for remote objects that already use a
> different memory model to enforce locking; you could start there.
>
> >  b) Bertrand Meyer's SCOOPS system, designed for Eiffel. It has two
> > major strikes against it: 1) it is based on type attributes on
> > *variables*, andI could figure out how to translate that to a language
> > where variables aren't typed.
>
> Actually, that isn't so bad.  Common Lisp doesn't normally type
> variables at the source code level, but
>    (a)  You can explicitly add typing information if you want to, and
>    (b)  The compiler can often infer types
>
> If you want this to mesh with python, the constraints are similar; not
> only does the locking and safety marking have to be unobtrusive, it
> probably has to be optional.  And there is existing (if largely
> superseded by Pypy) work on type inference for variables.
>
> -jJ
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111104/93e1128e/attachment.html>

From ncoghlan at gmail.com  Fri Nov  4 08:41:42 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 4 Nov 2011 17:41:42 +1000
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CAMX1Ron0vUnjQ7f+5zZ=zWprLuRgCCbqd6a0zWM3tXwfK2bs7w@mail.gmail.com>
References: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
	<CA+OGgf4VXtU5m5N6X8-4en4F2krsMEL_uR3NawZR3jjwdyP4eA@mail.gmail.com>
	<CAMX1Ron0vUnjQ7f+5zZ=zWprLuRgCCbqd6a0zWM3tXwfK2bs7w@mail.gmail.com>
Message-ID: <CADiSq7fpFz6DHZMe03uvurNg6n1BUr-ergoZMHPLTKk2ZpBY-A@mail.gmail.com>

On Fri, Nov 4, 2011 at 5:03 PM, Adam Jorgensen
<adam.jorgensen.za at gmail.com> wrote:
> The GIL makes them pseudo-pointless in CPython anyway and the headaches
> arising from
> threading are very frustrating.

This is just plain false. Threads are still an excellent way to take a
synchronous operation and make it asynchronous. Take a look at
concurrent.futures in 3.2, which makes it trivial to take independent
blocking tasks and run them in parallel. The *only* time the GIL
causes problems is when you have CPU bound threads written in pure
Python. That's only a fraction of all of the Python apps out there,
many of which are either calling out to calculations in C or FORTRAN
(scientific community, financial community) or else performing IO
bound tasks (most everyone else with a network connection).

People need to remember that *concurrency is a hard problem*. That's
why we layer abstractions on top of it. The threading and
multiprocessing modules are both fairly low level, so they offer lots
of ways to shoot yourself in the foot, but also a lot of power and
flexibility.

The concurrent.futures model is a higher level abstraction that's much
easier to get right.

> Personally I would rather see an Actors library...

And what is an actors library going to use as its concurrency
mechanism if the threading and multiprocessing modules aren't there
under the hood?

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From adam.jorgensen.za at gmail.com  Fri Nov  4 08:53:17 2011
From: adam.jorgensen.za at gmail.com (Adam Jorgensen)
Date: Fri, 4 Nov 2011 09:53:17 +0200
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CADiSq7fpFz6DHZMe03uvurNg6n1BUr-ergoZMHPLTKk2ZpBY-A@mail.gmail.com>
References: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>
	<CA+OGgf4VXtU5m5N6X8-4en4F2krsMEL_uR3NawZR3jjwdyP4eA@mail.gmail.com>
	<CAMX1Ron0vUnjQ7f+5zZ=zWprLuRgCCbqd6a0zWM3tXwfK2bs7w@mail.gmail.com>
	<CADiSq7fpFz6DHZMe03uvurNg6n1BUr-ergoZMHPLTKk2ZpBY-A@mail.gmail.com>
Message-ID: <CAMX1Ro=FtMxi9S-RiXCf5gDp22mLOe1xYzMDS48E=TA09Bi7Hw@mail.gmail.com>

On 4 November 2011 09:41, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Fri, Nov 4, 2011 at 5:03 PM, Adam Jorgensen
> <adam.jorgensen.za at gmail.com> wrote:
> > The GIL makes them pseudo-pointless in CPython anyway and the headaches
> > arising from
> > threading are very frustrating.
>
> This is just plain false. Threads are still an excellent way to take a
> synchronous operation and make it asynchronous. Take a look at
> concurrent.futures in 3.2, which makes it trivial to take independent
> blocking tasks and run them in parallel. The *only* time the GIL
> causes problems is when you have CPU bound threads written in pure
> Python. That's only a fraction of all of the Python apps out there,
> many of which are either calling out to calculations in C or FORTRAN
> (scientific community, financial community) or else performing IO
> bound tasks (most everyone else with a network connection).
>

I would love to seem some actual stats on this? How many multi-threaded
apps are hitting the GIL barrier, etc...

Anyway, I consider myself refuted...


>
> People need to remember that *concurrency is a hard problem*. That's
> why we layer abstractions on top of it. The threading and
> multiprocessing modules are both fairly low level, so they offer lots
> of ways to shoot yourself in the foot, but also a lot of power and
> flexibility.
>
> The concurrent.futures model is a higher level abstraction that's much
> easier to get right.
>
> > Personally I would rather see an Actors library...
>
> And what is an actors library going to use as its concurrency
> mechanism if the threading and multiprocessing modules aren't there
> under the hood?
>

I said nothing about removing the multi-processing module, although it
would be nice if it spawned child processes didn't randomly zombify
for no good reason.

Regardless, I still think the GIL should be fixed or the threading module
removed. It's disingenuous to have a threading module when it doesn't work
as advertised due to an interpreter "feature" of dubious merit anyway.


>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111104/ba1c4a20/attachment.html>

From stefan_ml at behnel.de  Fri Nov  4 10:16:26 2011
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 04 Nov 2011 10:16:26 +0100
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CAMX1Ro=FtMxi9S-RiXCf5gDp22mLOe1xYzMDS48E=TA09Bi7Hw@mail.gmail.com>
References: <CAD=7U2C9isyxbQmWd5S3N_k+PN59ouqAJWErc70tw94bkf1XuQ@mail.gmail.com>	<CA+OGgf4VXtU5m5N6X8-4en4F2krsMEL_uR3NawZR3jjwdyP4eA@mail.gmail.com>	<CAMX1Ron0vUnjQ7f+5zZ=zWprLuRgCCbqd6a0zWM3tXwfK2bs7w@mail.gmail.com>	<CADiSq7fpFz6DHZMe03uvurNg6n1BUr-ergoZMHPLTKk2ZpBY-A@mail.gmail.com>
	<CAMX1Ro=FtMxi9S-RiXCf5gDp22mLOe1xYzMDS48E=TA09Bi7Hw@mail.gmail.com>
Message-ID: <j90ahe$k4e$1@dough.gmane.org>

Adam Jorgensen, 04.11.2011 08:53:
> On 4 November 2011 09:41, Nick Coghlan wrote:
>> On Fri, Nov 4, 2011 at 5:03 PM, Adam Jorgensen wrote:
>>> The GIL makes them pseudo-pointless in CPython anyway and the headaches
>>> arising from threading are very frustrating.
>>
>> This is just plain false.

The first part, yes. The second - depends. Threading, especially when 
applied to the wrong task, is a very good way to give you headaches.


>> Threads are still an excellent way to take a
>> synchronous operation and make it asynchronous. Take a look at
>> concurrent.futures in 3.2, which makes it trivial to take independent
>> blocking tasks and run them in parallel. The *only* time the GIL
>> causes problems is when you have CPU bound threads written in pure
>> Python. That's only a fraction of all of the Python apps out there,
>> many of which are either calling out to calculations in C or FORTRAN
>> (scientific community, financial community) or else performing IO
>> bound tasks (most everyone else with a network connection).
>
> I would love to seem some actual stats on this? How many multi-threaded
> apps are hitting the GIL barrier, etc...

In the numerics corner, multi-threaded CPU bound code is surprisingly 
common. In multi-server setups, people commonly employ MPI&friends, but 
especially on multi-core machines (usually less than 64 cores), threading 
is quite widely used. Proof? Cython just gained support for OpenMP based 
parallel loops, due to popular request.

However, computational code usually doesn't hit the "GIL barrier", as you 
call it, because the really heavy computations don't run in the interpreter 
but straight on the iron. So, no, neither I/O bound tasks nor CPU bound 
numerics tasks get in conflict with the GIL, unless you do something wrong.

That's the main theme, BTW. If you're a frequent reader of c.l.py, you'll 
quickly notice that those who complain most loudly about the GIL usually 
just do so because they do something wrong. Threading isn't the silver 
bullet that you shoot at the task at hand. It's *one* way to solve *some* 
kinds of concurrency problems, and certainly not a trivial one.

Stefan



From stephen at xemacs.org  Fri Nov  4 10:36:08 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 04 Nov 2011 18:36:08 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CA+OGgf5SSq8xD3aT_SRDDwrMZ8YVbeUw3PKT6ndt7Ud1U4CUvg@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
	<87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>
	<87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CA+OGgf5SSq8xD3aT_SRDDwrMZ8YVbeUw3PKT6ndt7Ud1U4CUvg@mail.gmail.com>
Message-ID: <87ipn0nsaf.fsf@uwakimon.sk.tsukuba.ac.jp>

Jim Jewett writes:
 > On Thu, Nov 3, 2011 at 12:35 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
 > 
 > > As for "wrapped C libraries", I'm having trouble imagining what you're
 > > talking about.
 > 
 > C code (currently) can create or modify python objects using a C
 > pointer, instead of python access.

Right, and that means they complete bypass Mike's proposal, too.  So
his reference to them was a non sequitur in context, because AFAICS
his proposal couldn't do anything more about them than mine could.

Can it?


From jimjjewett at gmail.com  Fri Nov  4 15:38:29 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Fri, 4 Nov 2011 10:38:29 -0400
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87ipn0nsaf.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
	<87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>
	<87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CA+OGgf5SSq8xD3aT_SRDDwrMZ8YVbeUw3PKT6ndt7Ud1U4CUvg@mail.gmail.com>
	<87ipn0nsaf.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CA+OGgf5Byf9NXEOF7KniGJB+0fHJKENmUjAskkwpsz2CD+=A_Q@mail.gmail.com>

On Fri, Nov 4, 2011 at 5:36 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Jim Jewett writes:
> ?> On Thu, Nov 3, 2011 at 12:35 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:

> ?> > As for "wrapped C libraries", I'm having trouble imagining what you're
> ?> > talking about.

> ?> C code (currently) can create or modify python objects using a C
> ?> pointer, instead of python access.

> Right, and that means they complete bypass Mike's proposal, too. ?So
> his reference to them was a non sequitur in context, because AFAICS
> his proposal couldn't do anything more about them than mine could.

> Can it?

Not if it is just an additional compile-time restriction.

If the implementation is prepared to enforce the restriction at
run-time, that will almost certainly require some changes to memory
allocation.  If done right, that might also be able to lock out rogue
C code, which should also allow a tracing GC, stronger security
guarantees, and GIL removal.

I don't personally see it happening without a currently unacceptable
slowdown for all memory access, but if he is looking 5-10 years out,
it is plausible.  (I would still bet against it for mainstream
CPython, but it is plausible.)

-jJ


From mwm at mired.org  Fri Nov  4 17:46:28 2011
From: mwm at mired.org (Mike Meyer)
Date: Fri, 4 Nov 2011 09:46:28 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CA+OGgf5Byf9NXEOF7KniGJB+0fHJKENmUjAskkwpsz2CD+=A_Q@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<87ipn4p8z4.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2C9cUK2dp+9TsUxrRXya8Pb1QMCLqFMBgjCUJE0=sGagQ@mail.gmail.com>
	<87ehxrpvl9.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2DVq5m+mLeqK6zvTzbrAu6Ba7Z3sO7jX0qkxcAowK8LBQ@mail.gmail.com>
	<87bosvpa9l.fsf@uwakimon.sk.tsukuba.ac.jp>
	<20111101225341.51801e1b@bhuda.mired.org>
	<87aa8eq577.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CAD=7U2BM_LNnvkHt_BAV+GaqmLKRE3aR2Q+B2_ScTuCHWGz6zQ@mail.gmail.com>
	<87zkgdombg.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CA+OGgf5SSq8xD3aT_SRDDwrMZ8YVbeUw3PKT6ndt7Ud1U4CUvg@mail.gmail.com>
	<87ipn0nsaf.fsf@uwakimon.sk.tsukuba.ac.jp>
	<CA+OGgf5Byf9NXEOF7KniGJB+0fHJKENmUjAskkwpsz2CD+=A_Q@mail.gmail.com>
Message-ID: <CAD=7U2DHZRDVcr09vRmdTnZ=DE-b76a_AjH1k_e9J66vEcvGSQ@mail.gmail.com>

On Fri, Nov 4, 2011 at 7:38 AM, Jim Jewett <jimjjewett at gmail.com> wrote:
> On Fri, Nov 4, 2011 at 5:36 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>> Jim Jewett writes:
>> ?> On Thu, Nov 3, 2011 at 12:35 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>
>> ?> > As for "wrapped C libraries", I'm having trouble imagining what you're
>> ?> > talking about.
>
>> ?> C code (currently) can create or modify python objects using a C
>> ?> pointer, instead of python access.
>
>> Right, and that means they complete bypass Mike's proposal, too. ?So
>> his reference to them was a non sequitur in context, because AFAICS
>> his proposal couldn't do anything more about them than mine could.
>
>> Can it?
>
> Not if it is just an additional compile-time restriction.

Which is why it calls for raising exceptions at run-time.

> If the implementation is prepared to enforce the restriction at
> run-time, that will almost certainly require some changes to memory
> allocation. ?If done right, that might also be able to lock out rogue
> C code, which should also allow a tracing GC, stronger security
> guarantees, and GIL removal.

I'm not really interested in locking out rogue C code - or at least
not malicious code. This is very much in the "we're all adults here"
vein, in that if you want to go out of your way to defeat it, it won't
stand in your way.

> I don't personally see it happening without a currently unacceptable
> slowdown for all memory access, but if he is looking 5-10 years out,
> it is plausible. ?(I would still bet against it for mainstream
> CPython, but it is plausible.)

I'm just trying to get the ball rolling. It clearly won't be in Python
3, so I'm looking to at least Python 4. What I expect to come out of
this is an information PEP summarizing the discussion, and possibly
some code that might be useful now (i.e. - a new library, or an
automated "concurrency safe" auditor, or ...).

  <mike


From mwm at mired.org  Fri Nov  4 18:11:53 2011
From: mwm at mired.org (Mike Meyer)
Date: Fri, 4 Nov 2011 10:11:53 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
	<CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
Message-ID: <CAD=7U2B21BkgM2QDLRNcPx6x4qLKFSvbuu1GTyYs6bN7RcTtEA@mail.gmail.com>

On Thu, Nov 3, 2011 at 6:51 PM, Bruce Leban <bruce at leapyear.org> wrote:
>
> On Thu, Nov 3, 2011 at 9:38 AM, Mike Meyer <mwm at mired.org> wrote:
>>
>> Attributes are names. You don't lock names, you lock objects. To
>> prevent binding of an attribute, you lock the object it's an attribute
>> of. If you want to prevent mutating the value of the object bound to
>> the attribute, you lock that object.
>
> You conveniently ignore my comment about the importance of database row
> locking. :-)
>

I assure you it was unintentional, but there's been a lot of text
flowing here. I believe I've answered every comment, though I know
some of them were answered when first brought up, and not every time
thereafter. And some of the answers were "Yes, that's an issue" or
"Yes, but we can't keep people from writing bad code". I don't recall
anything specifically about database row locking, but if you feel it's
important enough to reiterate it, I'll treat it as equally important.

>>
>> > (3) What happens if one thread does:
>> >
>> > locking d, x: # stuff
>> >
>> > and another does
>> >
>> > locking x, d: # stuff
>>
>> That one I dealt with in the specification. There's an implementation
>> requirement that if two locking statements share objects, the shared
>> objects have to be locked in the same order. So one of the two will
>> lock things in the opposite of the order they appear in the statement.
> You're handwaving. How would that work? There's no way to know at compile
> time what the correct order is. So locking just got a whole lot more
> complicated. But it's worse than complicated: it's unworkable.

How is a requirement just handwaving? There is no "correct" order.
Given two objects, any locking statement will always lock them in the
same order. That's sufficient to prevent deadlocks. Sorting the list
passed to each value with a key of id will do the trick, but may not
be the best way to do it.

>> No, avoiding deadlock is one of the goals. That's why that requirement
>> exists, and there's a further requirement that you can only nest
>> locking statements if the inner one locks a subset of the outer one.
>
> In other words, this code is not allowed:
>
> def f():
> ? ? locking x:
> ? ? ? ? for y in x:
> ? ? ? ? ? ? locking y:
> ? ? ? ? ? ? ? ? y.append(1)
>
> And neither is this:
>
> def f(x):
> ? ? locking x:
> ? ? ? ? x.append(1)
>
> if it is called inside code that has something locked. Which virtually all
> code will do since it's required everywhere. So your proposal is that I must
> lock everything before I change it but I can't lock it unless it's already
> locked. That's ridiculous. You might as well have a single semaphore to lock
> everything.

Yup, it's pretty bad. For that reason - among others - the proposal
has mutated. The original required listing objects, and either started
an STM transaction, or did explicit locks (undecided).  Someone
pointed out that the STM variant couldn't deal with IO, causing me to
note a change to the proposal wherein listing objecs caused explicit
locking, and having an empty list started an STM transaction. This is
not only easier to use, but solves several problems - including this
one. The STM variant hasn't gotten a lot of attention.

> We're wasting our time.?You said above "You don't lock names, you lock
> objects." Why not? Because that was your original plan and your mind is
> closed to other ideas?

No, because I couldn't think of a way to make locking names work. You
presented a problem with locking names. Since the proposal doesn't
lock names, I figured I wasn't clear enough in the first place, and
tried fix that.

I've repeatedly said this wasn't meant to be a finished idea, and
asked for alternatives.  If you have a proposal that works by locking
names, by all means tell us about it!

> You want to ignore legitimate issues that aren't
> convenient to your proposal.

No, I don't. I want to collect them all. But I'm not perfect -
sometimes two comments may look similar when they in fact aren't.

> I hate to say things like "this idea sucks,"
> but sometimes that's the only way to put it. I think that this proposal is a
> terrible idea.

That may well be the case, and wouldn't particularly disappoint me.

> A healthy discussion about how to make concurrency better might be
> interesting but as long as all you want to do is defend your proposal, this
> thread isn't sparking useful discussion

I'm doing a lot more than defending the proposal. I'm tweaking this
proposal to solve problems people point out, collecting ideas for
alternative approaches, implementations, and tools that might help -
which are what make this discussion interesting and useful. I do
defend the proposal, especially when people point out issues that were
dealt with in the original post, which isn't interesting, but the
alternative (ignoring them) seems less useful.

     <mike


From jimjjewett at gmail.com  Fri Nov  4 18:49:06 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Fri, 4 Nov 2011 13:49:06 -0400
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CAD=7U2B21BkgM2QDLRNcPx6x4qLKFSvbuu1GTyYs6bN7RcTtEA@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
	<CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
	<CAD=7U2B21BkgM2QDLRNcPx6x4qLKFSvbuu1GTyYs6bN7RcTtEA@mail.gmail.com>
Message-ID: <CA+OGgf7enA74FaYO=nBGgwOOmmvk5a3UGPsL5WK10UcTENG=0w@mail.gmail.com>

On Fri, Nov 4, 2011 at 1:11 PM, Mike Meyer <mwm at mired.org> wrote:
> On Thu, Nov 3, 2011 at 6:51 PM, Bruce Leban <bruce at leapyear.org> wrote:
>>
>> On Thu, Nov 3, 2011 at 9:38 AM, Mike Meyer <mwm at mired.org> wrote:
>> We're wasting our time.?You said above "You don't lock names, you lock
>> objects." Why not? Because that was your original plan and your mind is
>> closed to other ideas?

> No, because I couldn't think of a way to make locking names work.

Replace dicts (or at least globals and attribute dicts) with something
that enforces your policy -- whether that is by a mutex, a mutex per
entry, returning only copies of the values, etc...

-jJ


From clockworksaint at gmail.com  Fri Nov  4 18:57:11 2011
From: clockworksaint at gmail.com (Weeble)
Date: Fri, 4 Nov 2011 17:57:11 +0000
Subject: [Python-ideas] A concurrency survey of sorts
Message-ID: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>

> Regardless, I still think the GIL should be fixed or the threading module
> removed. It's disingenuous to have a threading module when it doesn't work
> as advertised due to an interpreter "feature" of dubious merit anyway.

Is this a serious proposal? Without threads, is there any reasonable
cross-platform way to communicate interactively with multiple subprocesses
over pipes? Windows doesn't have select. I don't *like* using threads, but
as far as I can tell they're the only available way to do this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111104/c47f830c/attachment.html>

From python at mrabarnett.plus.com  Fri Nov  4 18:58:00 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 04 Nov 2011 17:58:00 +0000
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <87pqh8o85d.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
	<CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
	<4EB34C86.10404@mrabarnett.plus.com>
	<87pqh8o85d.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <4EB427A8.8060704@mrabarnett.plus.com>

On 04/11/2011 03:53, Stephen J. Turnbull wrote:
> MRAB writes:
>   >  On 04/11/2011 01:51, Bruce Leban wrote:
>
>   >  >       >  (3) What happens if one thread does:
>   >  >       >
>   >  >       >  locking d, x: # stuff
>   >  >       >
>   >  >       >  and another does
>   >  >       >
>   >  >       >  locking x, d: # stuff
>   >  >
>   >  >      That one I dealt with in the specification. There's an implementation
>   >  >      requirement that if two locking statements share objects, the shared
>   >  >      objects have to be locked in the same order. So one of the two will
>   >  >      lock things in the opposite of the order they appear in the statement.
>   >  >
>   >  >  You're handwaving. How would that work?
>   >
>   >  If they're threads running in the same address space in CPython then
>   >  they could lock always in order of increasing address.
>
> OK, I think that works.
>
>   >  In an implementation of Python which uses references they could
>   >  lock always in increasing numeric value of reference (if that's
>   >  possible).
>
> That doesn't make sense to me; if it's not an address, how do you know
> it's unique (you know the object is unique, of course, but there may
> be other references)?  If the reference is not unique, how do you know
> that some other thread hasn't locked it via a different reference?
>
In CPython the implementation refers to an object by its address, but
my understanding is that in an implementation which uses references the
reference is actually an index into a table which contains the
addresses of the objects, therefore you would be locking in order of
increasing index.

> I think for this to be workable you would need to use something like a
> lock tick that would increase with each lock taken, and store that in
> the object.  Then you would lock the objects in order of their ticks,
> with unlocked objects coming last.  (You'd have to be careful about
> the wraparound, though, and that could double or triple the
> performance hit for incrementing the tick and determining lock order.)
>


From amauryfa at gmail.com  Fri Nov  4 19:13:29 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 4 Nov 2011 19:13:29 +0100
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
References: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
Message-ID: <CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>

2011/11/4 Weeble <clockworksaint at gmail.com>

> > Regardless, I still think the GIL should be fixed or the threading module
> > removed. It's disingenuous to have a threading module when it doesn't
> work
> > as advertised due to an interpreter "feature" of dubious merit anyway.
>
> Is this a serious proposal? Without threads, is there any reasonable
> cross-platform way to communicate interactively with multiple subprocesses
> over pipes? Windows doesn't have select. I don't *like* using threads, but
> as far as I can tell they're the only available way to do this.
>
Windows does have a select(), even if this only works for sockets,
(you can use WaitForSingleObject for any win32 handle)
and the Twisted reactor has a spawnProcess() function that works without
any additional thread.

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111104/b5e3d0d2/attachment.html>

From mwm at mired.org  Fri Nov  4 19:20:49 2011
From: mwm at mired.org (Mike Meyer)
Date: Fri, 4 Nov 2011 11:20:49 -0700
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <CA+OGgf7enA74FaYO=nBGgwOOmmvk5a3UGPsL5WK10UcTENG=0w@mail.gmail.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
	<CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
	<CAD=7U2B21BkgM2QDLRNcPx6x4qLKFSvbuu1GTyYs6bN7RcTtEA@mail.gmail.com>
	<CA+OGgf7enA74FaYO=nBGgwOOmmvk5a3UGPsL5WK10UcTENG=0w@mail.gmail.com>
Message-ID: <CAD=7U2C0txbEbuCLxt6xNicOEP_Zhqv25A6+F=4JzFbMyH1T1g@mail.gmail.com>

On Fri, Nov 4, 2011 at 10:49 AM, Jim Jewett <jimjjewett at gmail.com> wrote:
> On Fri, Nov 4, 2011 at 1:11 PM, Mike Meyer <mwm at mired.org> wrote:
>> On Thu, Nov 3, 2011 at 6:51 PM, Bruce Leban <bruce at leapyear.org> wrote:
>>>
>>> On Thu, Nov 3, 2011 at 9:38 AM, Mike Meyer <mwm at mired.org> wrote:
>>> We're wasting our time.?You said above "You don't lock names, you lock
>>> objects." Why not? Because that was your original plan and your mind is
>>> closed to other ideas?
>
>> No, because I couldn't think of a way to make locking names work.
>
> Replace dicts (or at least globals and attribute dicts) with something
> that enforces your policy -- whether that is by a mutex, a mutex per
> entry, returning only copies of the values, etc...

Sorry, I wasn't sufficiently detailed. I never worried about how to
implement locking names, because I never could find a way to let
people use it. I'd like fine-grained locking (i.e. - specific
attributes of objects, the value at an index in  a key or dictionary,
etc.), you still need to be able to lock objects. They ways I came up
with for specifying "here's a name we can't rebind" were all very
complicated - among other things, what's a "name" is itself
complicated - and not worth the gain you got over object-level
locking. Especially if you have an STM option available.

As always, if you can think of a way to get all this working together,
please share!

    Thanks,
    <mike


From clockworksaint at gmail.com  Fri Nov  4 19:42:25 2011
From: clockworksaint at gmail.com (Weeble)
Date: Fri, 4 Nov 2011 18:42:25 +0000
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>
References: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
	<CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>
Message-ID: <CACBComEFc-AjZ02jD9jsVFwMaFcf=kQ9T+9i68U-JD3tMBmnSA@mail.gmail.com>

> 2011/11/4 Weeble <clockworksaint at gmail.com>
>> Is this a serious proposal? Without threads, is there any reasonable
>> cross-platform way to communicate interactively with multiple subprocesses
>> over pipes? Windows doesn't have select. I don't *like* using threads, but
>> as far as I can tell they're the only available way to do this.

On Fri, Nov 4, 2011 at 6:13 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> Windows does have a select(), even if this only works for sockets,
> (you can use WaitForSingleObject for any win32 handle)
> and the Twisted reactor has a spawnProcess() function that works without any
> additional thread.

select: Yes.

WaitForSingleObject: Really? The documentation seems to say only on
events, mutexes, semaphores, processes and threads. No mention of
pipes or files. http://msdn.microsoft.com/en-us/library/bb202783.aspx
Also, presumably it would be WaitForMultipleObjects? Regardless, I'm
sure there's some way to do this. I just don't think it's obvious or
easy enough to be considered a reasonable alternative to using
threads.

Twisted: That actually looks quite nice, but pulling in Twisted isn't
always going to be a reasonable option.

I just want to be sure - the idea of removing the threading module
sounds ludicrous. It's not getting serious consideration, is it?


From solipsis at pitrou.net  Fri Nov  4 19:49:50 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 4 Nov 2011 19:49:50 +0100
Subject: [Python-ideas] A concurrency survey of sorts
References: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
	<CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>
	<CACBComEFc-AjZ02jD9jsVFwMaFcf=kQ9T+9i68U-JD3tMBmnSA@mail.gmail.com>
Message-ID: <20111104194950.7127bd0d@pitrou.net>

On Fri, 4 Nov 2011 18:42:25 +0000
Weeble <clockworksaint at gmail.com> wrote:
> 
> select: Yes.
> 
> WaitForSingleObject: Really? The documentation seems to say only on
> events, mutexes, semaphores, processes and threads. No mention of
> pipes or files. http://msdn.microsoft.com/en-us/library/bb202783.aspx
> Also, presumably it would be WaitForMultipleObjects? Regardless, I'm
> sure there's some way to do this. I just don't think it's obvious or
> easy enough to be considered a reasonable alternative to using
> threads.

It works with pipes for sure, it's used in multiprocessing (in 3.3).
However, the pipes have to be set non-blocking, which implies using
overlapped I/O (or another form of non-blocking Windows I/O) which is
quite a pain.

> I just want to be sure - the idea of removing the threading module
> sounds ludicrous. It's not getting serious consideration, is it?

No :-)

Regards

Antoine.




From mwm at mired.org  Fri Nov  4 20:02:06 2011
From: mwm at mired.org (Mike Meyer)
Date: Fri, 4 Nov 2011 12:02:06 -0700
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CACBComEFc-AjZ02jD9jsVFwMaFcf=kQ9T+9i68U-JD3tMBmnSA@mail.gmail.com>
References: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
	<CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>
	<CACBComEFc-AjZ02jD9jsVFwMaFcf=kQ9T+9i68U-JD3tMBmnSA@mail.gmail.com>
Message-ID: <CAD=7U2Bzw8RwzQ2ijLS2fUFzr4K4Ve8DR+KA33xiMNdErfJvnw@mail.gmail.com>

On Fri, Nov 4, 2011 at 11:42 AM, Weeble <clockworksaint at gmail.com> wrote:
> I just want to be sure - the idea of removing the threading module
> sounds ludicrous. It's not getting serious consideration, is it?

Removing  the threading module - as part of a change that includes
providing some other concurrent processing facility - seems like a
reasonable enough idea that I'll be including it in the sequel.
Rewriting it to use those facilities might be reasonable for backwards
compatibility, but the new facilities should have enough advantages
that new code would just use them.

   <mike


From ctb at msu.edu  Fri Nov  4 20:09:36 2011
From: ctb at msu.edu (C. Titus Brown)
Date: Fri, 4 Nov 2011 12:09:36 -0700
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CAD=7U2Bzw8RwzQ2ijLS2fUFzr4K4Ve8DR+KA33xiMNdErfJvnw@mail.gmail.com>
References: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
	<CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>
	<CACBComEFc-AjZ02jD9jsVFwMaFcf=kQ9T+9i68U-JD3tMBmnSA@mail.gmail.com>
	<CAD=7U2Bzw8RwzQ2ijLS2fUFzr4K4Ve8DR+KA33xiMNdErfJvnw@mail.gmail.com>
Message-ID: <20111104190936.GA12074@idyll.org>

On Fri, Nov 04, 2011 at 12:02:06PM -0700, Mike Meyer wrote:
> On Fri, Nov 4, 2011 at 11:42 AM, Weeble <clockworksaint at gmail.com> wrote:
> > I just want to be sure - the idea of removing the threading module
> > sounds ludicrous. It's not getting serious consideration, is it?
> 
> Removing  the threading module - as part of a change that includes
> providing some other concurrent processing facility - seems like a
> reasonable enough idea that I'll be including it in the sequel.
> Rewriting it to use those facilities might be reasonable for backwards
> compatibility, but the new facilities should have enough advantages
> that new code would just use them.

I obviously haven't been paying enough attention to this conversation,
because this strikes me as a horrible idea.  In particular, I write
C extensions that release the GIL and scale just fine across multiple
processors.  Is the idea to force those kind of applications to use
a new communications model?

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu


From mwm at mired.org  Fri Nov  4 20:20:55 2011
From: mwm at mired.org (Mike Meyer)
Date: Fri, 4 Nov 2011 12:20:55 -0700
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <20111104190936.GA12074@idyll.org>
References: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
	<CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>
	<CACBComEFc-AjZ02jD9jsVFwMaFcf=kQ9T+9i68U-JD3tMBmnSA@mail.gmail.com>
	<CAD=7U2Bzw8RwzQ2ijLS2fUFzr4K4Ve8DR+KA33xiMNdErfJvnw@mail.gmail.com>
	<20111104190936.GA12074@idyll.org>
Message-ID: <CAD=7U2C5sU63YEMUrX55fNkwpKyU0=kf7LMcV4sP_zvs+GetyQ@mail.gmail.com>

On Fri, Nov 4, 2011 at 12:09 PM, C. Titus Brown <ctb at msu.edu> wrote:
> On Fri, Nov 04, 2011 at 12:02:06PM -0700, Mike Meyer wrote:
>> On Fri, Nov 4, 2011 at 11:42 AM, Weeble <clockworksaint at gmail.com> wrote:
>> > I just want to be sure - the idea of removing the threading module
>> > sounds ludicrous. It's not getting serious consideration, is it?
>>
>> Removing ?the threading module - as part of a change that includes
>> providing some other concurrent processing facility - seems like a
>> reasonable enough idea that I'll be including it in the sequel.
>> Rewriting it to use those facilities might be reasonable for backwards
>> compatibility, but the new facilities should have enough advantages
>> that new code would just use them.
>
> I obviously haven't been paying enough attention to this conversation,
> because this strikes me as a horrible idea. ?In particular, I write
> C extensions that release the GIL and scale just fine across multiple
> processors. ?Is the idea to force those kind of applications to use
> a new communications model?

That's one of the ideas. The goal is to get the Python processor to
help locate concurrency bugs. Since the threading module
communications model is "everything is shared by default", I'm pretty
sure it's going to break code that depends on it. But that's why it's
a proposal for Python 4 or something even further away.

    <mike


From solipsis at pitrou.net  Fri Nov  4 20:38:42 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 4 Nov 2011 20:38:42 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net>
	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>
Message-ID: <20111104203842.7de86289@pitrou.net>

On Sun, 30 Oct 2011 09:52:15 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Example with nested functions
> > -----------------------------
> >
> >>>> def f():
> > ... ? def g(): pass
> > ... ? return g
> > ...
> >>>> f.__qname__
> > 'f'
> >>>> f().__qname__
> > 'f.g'
> 
> For nested functions, I suggest adding something to the qname to
> directly indicate that the scope is hidden. Adding parentheses to the
> name of the outer function would probably work:
> 
>     f().g

I don't know, I find the "()" a bit too smart. I'd like Guido's
advice on the matter.

Regards

Antoine.




From guido at python.org  Fri Nov  4 21:49:23 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 4 Nov 2011 13:49:23 -0700
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111104203842.7de86289@pitrou.net>
References: <20111030001801.2f52ceb2@pitrou.net>
	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>
	<20111104203842.7de86289@pitrou.net>
Message-ID: <CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>

On Fri, Nov 4, 2011 at 12:38 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 30 Oct 2011 09:52:15 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> > Example with nested functions
>> > -----------------------------
>> >
>> >>>> def f():
>> > ...   def g(): pass
>> > ...   return g
>> > ...
>> >>>> f.__qname__
>> > 'f'
>> >>>> f().__qname__
>> > 'f.g'
>>
>> For nested functions, I suggest adding something to the qname to
>> directly indicate that the scope is hidden. Adding parentheses to the
>> name of the outer function would probably work:
>>
>>     f().g
>
> I don't know, I find the "()" a bit too smart. I'd like Guido's
> advice on the matter.

Hm. Both 'f.g' and 'f().g' for f().__qname__ are misleading, since both
look like valid expressions but neither will actually retrieve the intended
object. I'd be tempted to return something like '<local in f>.g' where 'f'
would be f.__qname__. And yes, if f.__qname__ were '<local in xyzzy>.f',
then f().__qname__ should be '<local in <local in xyzzy>.f>.g'.

Alternatively perhaps I could live with 'f.<locals>.g' and
'xyzzy.<locals>.f.<locals>.g'.

In either case, the use of <...> makes it plain that this should not be
taken literally as an expression -- unlike the case of nested classes,
where generally 'C.D.E' works to access E, if you have class C containing
class D which contains class E.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111104/2565c725/attachment.html>

From solipsis at pitrou.net  Fri Nov  4 22:53:06 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 4 Nov 2011 22:53:06 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net>
	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>
	<20111104203842.7de86289@pitrou.net>
	<CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>
Message-ID: <20111104225306.44131581@pitrou.net>

On Fri, 4 Nov 2011 13:49:23 -0700
Guido van Rossum <guido at python.org> wrote:
> 
> Hm. Both 'f.g' and 'f().g' for f().__qname__ are misleading, since both
> look like valid expressions but neither will actually retrieve the intended
> object.

Agreed.

> Alternatively perhaps I could live with 'f.<locals>.g' and
> 'xyzzy.<locals>.f.<locals>.g'.

Ah, thanks! I like this one.

Regards

Antoine.




From zuo at chopin.edu.pl  Sat Nov  5 00:15:23 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sat, 5 Nov 2011 00:15:23 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
 functions
In-Reply-To: <CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net>
	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>
	<20111104203842.7de86289@pitrou.net>
	<CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>
Message-ID: <20111104231523.GB2306@chopin.edu.pl>

Guido van Rossum dixit (2011-11-04, 13:49):

> On Fri, Nov 4, 2011 at 12:38 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sun, 30 Oct 2011 09:52:15 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
[snip]
> >> For nested functions, I suggest adding something to the qname to
> >> directly indicate that the scope is hidden. Adding parentheses to the
> >> name of the outer function would probably work:
> >>
> >>     f().g
> >
> > I don't know, I find the "()" a bit too smart. I'd like Guido's
> > advice on the matter.
> 
> Hm. Both 'f.g' and 'f().g' for f().__qname__ are misleading, since both
> look like valid expressions but neither will actually retrieve the intended
> object. I'd be tempted to return something like '<local in f>.g' where 'f'
> would be f.__qname__. And yes, if f.__qname__ were '<local in xyzzy>.f',
> then f().__qname__ should be '<local in <local in xyzzy>.f>.g'.
> 
> Alternatively perhaps I could live with 'f.<locals>.g' and
> 'xyzzy.<locals>.f.<locals>.g'.

And what about:

    <f locals>.g
and
    <<xyzzy locals>.f locals>.g

Cheers.
*j



From guido at python.org  Sat Nov  5 01:58:05 2011
From: guido at python.org (Guido van Rossum)
Date: Fri, 4 Nov 2011 17:58:05 -0700
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111104231523.GB2306@chopin.edu.pl>
References: <20111030001801.2f52ceb2@pitrou.net>
	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>
	<20111104203842.7de86289@pitrou.net>
	<CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>
	<20111104231523.GB2306@chopin.edu.pl>
Message-ID: <CAP7+vJ+i6q23ube6d3S1yPLe4t=On3=F_X3U03X0s5pehYR7XA@mail.gmail.com>

On Fri, Nov 4, 2011 at 4:15 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:

> Guido van Rossum dixit (2011-11-04, 13:49):
>
> > On Fri, Nov 4, 2011 at 12:38 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > > On Sun, 30 Oct 2011 09:52:15 +1000
> > > Nick Coghlan <ncoghlan at gmail.com> wrote:
> [snip]
> > >> For nested functions, I suggest adding something to the qname to
> > >> directly indicate that the scope is hidden. Adding parentheses to the
> > >> name of the outer function would probably work:
> > >>
> > >>     f().g
> > >
> > > I don't know, I find the "()" a bit too smart. I'd like Guido's
> > > advice on the matter.
> >
> > Hm. Both 'f.g' and 'f().g' for f().__qname__ are misleading, since both
> > look like valid expressions but neither will actually retrieve the
> intended
> > object. I'd be tempted to return something like '<local in f>.g' where
> 'f'
> > would be f.__qname__. And yes, if f.__qname__ were '<local in xyzzy>.f',
> > then f().__qname__ should be '<local in <local in xyzzy>.f>.g'.
> >
> > Alternatively perhaps I could live with 'f.<locals>.g' and
> > 'xyzzy.<locals>.f.<locals>.g'.
>
> And what about:
>
>    <f locals>.g
> and
>    <<xyzzy locals>.f locals>.g
>

In the end I like the non-nested version better.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111104/e90169bf/attachment.html>

From joshua.landau.ws at gmail.com  Sat Nov  5 02:43:40 2011
From: joshua.landau.ws at gmail.com (Joshua Landau)
Date: Sat, 5 Nov 2011 01:43:40 +0000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <mailman.13669.1320456473.27777.python-ideas@python.org>
References: <mailman.13669.1320456473.27777.python-ideas@python.org>
Message-ID: <CAN1F8qUO8fb8sWu6KYcJW05f1TJoQJFY0pZ+GDORArsoe-RzBQ@mail.gmail.com>

First of all - hello python-ideas. I liked the look of the archives, so I
thought I'd chip in, if it's OK.


>  And what about:
>>
>>    <f locals>.g
>> and
>>    <<xyzzy locals>.f locals>.g
>>
>
> In the end I like the non-nested version better.
>
xyzzy.<local f>.<local g>
or, preferably,
xyzzy.<instance.f>.<instance.g>
It's non-nested, shows the locality and maps the locality to the variable.

If not showing function instances is disliked, maybe drop the period
between the function and "<locals>" from yours:
xyzzy<locals>.f<locals>.g
or, again,
xyzzy<instance>.f<instance>.g

I think this is just cleaner than:
xyzzy.<locals>.f.<locals>.g
because it doesn't imply that xyzzy has a single set of locals - especially
using <instance>.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111105/2c5b03cf/attachment.html>

From ron3200 at gmail.com  Sat Nov  5 05:52:59 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Fri, 04 Nov 2011 23:52:59 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
 functions
In-Reply-To: <20111104231523.GB2306@chopin.edu.pl>
References: <20111030001801.2f52ceb2@pitrou.net>
	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>
	<20111104203842.7de86289@pitrou.net>
	<CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>
	<20111104231523.GB2306@chopin.edu.pl>
Message-ID: <1320468779.1911.39.camel@Gutsy>

On Sat, 2011-11-05 at 00:15 +0100, Jan Kaliszewski wrote:
> Guido van Rossum dixit (2011-11-04, 13:49):
> 
> > On Fri, Nov 4, 2011 at 12:38 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > > On Sun, 30 Oct 2011 09:52:15 +1000
> > > Nick Coghlan <ncoghlan at gmail.com> wrote:
> [snip]
> > >> For nested functions, I suggest adding something to the qname to
> > >> directly indicate that the scope is hidden. Adding parentheses to the
> > >> name of the outer function would probably work:
> > >>
> > >>     f().g
> > >
> > > I don't know, I find the "()" a bit too smart. I'd like Guido's
> > > advice on the matter.
> > 
> > Hm. Both 'f.g' and 'f().g' for f().__qname__ are misleading, since both
> > look like valid expressions but neither will actually retrieve the intended
> > object. I'd be tempted to return something like '<local in f>.g' where 'f'
> > would be f.__qname__. And yes, if f.__qname__ were '<local in xyzzy>.f',
> > then f().__qname__ should be '<local in <local in xyzzy>.f>.g'.
> > 
> > Alternatively perhaps I could live with 'f.<locals>.g' and
> > 'xyzzy.<locals>.f.<locals>.g'.

Too many dots I think.


> And what about:
> 
>     <f locals>.g
> and
>     <<xyzzy locals>.f locals>.g

Better, but I don't think you need to nest it to indicate that
relationship. I would use a colon instead of a dot to indicate a
namespace relationship.
   
    f.g    g is an attribute of f
    f[g]   g is in the container f
    f:g    g is in the namespace f

So I think I'd one of...

     <xyzzy locals>:<f locals>:g
     <xyzzy:f:g>

A slice is used a completely different context.


Cheers,
   Ron






From ncoghlan at gmail.com  Sat Nov  5 07:34:06 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 5 Nov 2011 16:34:06 +1000
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CAD=7U2C5sU63YEMUrX55fNkwpKyU0=kf7LMcV4sP_zvs+GetyQ@mail.gmail.com>
References: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
	<CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>
	<CACBComEFc-AjZ02jD9jsVFwMaFcf=kQ9T+9i68U-JD3tMBmnSA@mail.gmail.com>
	<CAD=7U2Bzw8RwzQ2ijLS2fUFzr4K4Ve8DR+KA33xiMNdErfJvnw@mail.gmail.com>
	<20111104190936.GA12074@idyll.org>
	<CAD=7U2C5sU63YEMUrX55fNkwpKyU0=kf7LMcV4sP_zvs+GetyQ@mail.gmail.com>
Message-ID: <CADiSq7cEQf=4+froyL8ng7fzbtMROCsg06if3ati-tLyF93PZw@mail.gmail.com>

On Sat, Nov 5, 2011 at 5:20 AM, Mike Meyer <mwm at mired.org> wrote:
> That's one of the ideas. The goal is to get the Python processor to
> help locate concurrency bugs. Since the threading module
> communications model is "everything is shared by default", I'm pretty
> sure it's going to break code that depends on it. But that's why it's
> a proposal for Python 4 or something even further away.

I'd suggest it's actually a proposal for a language that *isn't
Python*. There's no way an idea like this would ever be accepted into
Python itself without real world experimentation in another context.
It's going to be hard to get anyone interested enough to seriously try
it out, but I'd suggest going the "Python-based language" route (ala
Boo and Cython) rather than claiming the idea is Python as such.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Sat Nov  5 07:36:29 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 5 Nov 2011 16:36:29 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111104225306.44131581@pitrou.net>
References: <20111030001801.2f52ceb2@pitrou.net>
	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>
	<20111104203842.7de86289@pitrou.net>
	<CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>
	<20111104225306.44131581@pitrou.net>
Message-ID: <CADiSq7cfXbZ_SVRY0si2Xv1n2u6JYLuexYn82Fj-a98PuPZ8VQ@mail.gmail.com>

On Sat, Nov 5, 2011 at 7:53 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 4 Nov 2011 13:49:23 -0700
> Guido van Rossum <guido at python.org> wrote:
>> Alternatively perhaps I could live with 'f.<locals>.g' and
>> 'xyzzy.<locals>.f.<locals>.g'.
>
> Ah, thanks! I like this one.

Yep, I like that scheme, too.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From stephen at xemacs.org  Sat Nov  5 14:05:09 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 05 Nov 2011 22:05:09 +0900
Subject: [Python-ideas] Fwd: Concurrent safety?
In-Reply-To: <4EB427A8.8060704@mrabarnett.plus.com>
References: <20111030201143.481fdca2@bhuda.mired.org>
	<CAGu0Ant6jBHJBFRmT6UER0VsyYzJbtAKDhspYBBJLJEb5UPOrQ@mail.gmail.com>
	<CAD=7U2Dn2Vx9cAQ_=Xz0sMWDy4rLW3KeZghF0hgE2v70h63aFQ@mail.gmail.com>
	<CAGu0AnvQhzc4xiQ+ADOZqUquaRXdbAQ=3NCS7hyQGa5qq4vAog@mail.gmail.com>
	<20111031223048.6e5d2798@bhuda.mired.org>
	<CAD=7U2Bvw4fv4GpHbWm76FMGQYpFJaeUyw-uOwcSD_tEaCf8PQ@mail.gmail.com>
	<CAGu0AnuSG0JZMyKmS6pibV1M_Qnsr2K+-9vEUSUfM_GBkO3QLQ@mail.gmail.com>
	<CAD=7U2As0jm7adXgiYAGnH5QWX6KPDtWKcGsFEUiCyQGbz+jNw@mail.gmail.com>
	<CAGu0Anu-KCg-X5eHpJULjeukUZnO6O--d04t_rPDJNZ7B_NpKg@mail.gmail.com>
	<4EB34C86.10404@mrabarnett.plus.com>
	<87pqh8o85d.fsf@uwakimon.sk.tsukuba.ac.jp>
	<4EB427A8.8060704@mrabarnett.plus.com>
Message-ID: <87fwi2oh2y.fsf@uwakimon.sk.tsukuba.ac.jp>

MRAB writes:

 > my understanding is that in an implementation which uses references the
 > reference is actually an index into a table which contains the
 > addresses of the objects, therefore you would be locking in order of
 > increasing index.

Ah, OK.  So a reference is actually an object in the sense of an
identifiable region of memory, but it's not a Python object.  When I
see "reference" I think of C++ references or git refs, which are just
names, and an object can have many references.  But the scheme you
describe is like a C stdio file descriptor, and presumably is an
indirection which allows relocating Python objects in memory.  You can
even relocate the table in such a scheme.



From ram.rachum at gmail.com  Sat Nov  5 15:07:22 2011
From: ram.rachum at gmail.com (Ram Rachum)
Date: Sat, 5 Nov 2011 07:07:22 -0700 (PDT)
Subject: [Python-ideas] Need way to check if a `datetime.timedelta` is
	positive
Message-ID: <28750278.2158.1320502042916.JavaMail.geo-discussion-forums@yqhd1>

(Sorry for possible double-post, Google Groups was doing weird things.)

Hey,

I'm currently writing code that's manipulating `datetime.timedelta` 
objects. I get one and I need to check whether it's positive or negative.

I can't really think of an elegant way to do this.

Here's one:

    my_timedelta >= datetime.timedelta(0)

Another one:

    my_timedelta.total_seconds() >= 0


Perhaps we should allow a more elegant way? Possibly `my_timedelta >= 0` 
for 0 only?


Thanks,
Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111105/308dcb70/attachment.html>

From arnodel at gmail.com  Sat Nov  5 15:22:29 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Sat, 5 Nov 2011 14:22:29 +0000
Subject: [Python-ideas] Need way to check if a `datetime.timedelta` is
	positive
In-Reply-To: <28750278.2158.1320502042916.JavaMail.geo-discussion-forums@yqhd1>
References: <28750278.2158.1320502042916.JavaMail.geo-discussion-forums@yqhd1>
Message-ID: <CAJ6cK1a3rzgT8qe-88vMbb1o6hYtrdG92Jqz8NTMz_cmG06C+g@mail.gmail.com>

On 5 November 2011 14:07, Ram Rachum <ram.rachum at gmail.com> wrote:
> (Sorry for possible double-post, Google Groups was doing weird things.)
> Hey,
> I'm currently writing code that's manipulating `datetime.timedelta` objects.
> I get one and I need to check whether it's positive or negative.
> I can't really think of an elegant way to do this.
> Here's one:
> ? ? my_timedelta >= datetime.timedelta(0)
> Another one:
> ? ? my_timedelta.total_seconds() >= 0
>
> Perhaps we should allow a more elegant way? Possibly `my_timedelta >= 0` for
> 0 only?
>
> Thanks,
> Ram.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>

(That's more of a question for c.l.py)

Simpler:

my_timedelta.days >= 0

--
Arnaud


From ram.rachum at gmail.com  Sat Nov  5 15:26:00 2011
From: ram.rachum at gmail.com (cool-RR)
Date: Sat, 5 Nov 2011 16:26:00 +0200
Subject: [Python-ideas] Need way to check if a `datetime.timedelta` is
	positive
In-Reply-To: <CAJ6cK1a3rzgT8qe-88vMbb1o6hYtrdG92Jqz8NTMz_cmG06C+g@mail.gmail.com>
References: <28750278.2158.1320502042916.JavaMail.geo-discussion-forums@yqhd1>
	<CAJ6cK1a3rzgT8qe-88vMbb1o6hYtrdG92Jqz8NTMz_cmG06C+g@mail.gmail.com>
Message-ID: <CANXboVbMsiv_i+EizzJvdJo+DQAG5JWv5faFLmp33zCGqjwA9w@mail.gmail.com>

On Sat, Nov 5, 2011 at 4:22 PM, Arnaud Delobelle <arnodel at gmail.com> wrote:

> On 5 November 2011 14:07, Ram Rachum <ram.rachum at gmail.com> wrote:
> > (Sorry for possible double-post, Google Groups was doing weird things.)
> > Hey,
> > I'm currently writing code that's manipulating `datetime.timedelta`
> objects.
> > I get one and I need to check whether it's positive or negative.
> > I can't really think of an elegant way to do this.
> > Here's one:
> >     my_timedelta >= datetime.timedelta(0)
> > Another one:
> >     my_timedelta.total_seconds() >= 0
> >
> > Perhaps we should allow a more elegant way? Possibly `my_timedelta >= 0`
> for
> > 0 only?
> >
> > Thanks,
> > Ram.
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > http://mail.python.org/mailman/listinfo/python-ideas
> >
> >
>
> (That's more of a question for c.l.py)
>
> Simpler:
>
> my_timedelta.days >= 0
>
> --
> Arnaud
>

Shorter, yeah, but still weird. I'm making a calculation involving
microseconds and suddenly I deal with days? Pretty weird.


-- 
Sincerely,
Ram Rachum
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111105/813b04df/attachment.html>

From merwok at netwok.org  Sat Nov  5 17:32:58 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Sat, 05 Nov 2011 17:32:58 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111030001801.2f52ceb2@pitrou.net>
References: <20111030001801.2f52ceb2@pitrou.net>
Message-ID: <4EB5653A.4000302@netwok.org>

Hi,

> I would like to propose the following PEP for discussion and, if
> possible, acceptance. I think the proposal shouldn't be too
> controversial (I find it quite simple and straightforward myself :-)).
> [snip PEP]

+1.

For nested functions, I too think that 'f.<locals>.g' has too many dots;
I like '<local in f>.g' or '<f locals>.g'.

polka-dots-ly yours


From ericsnowcurrently at gmail.com  Sat Nov  5 18:04:47 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 5 Nov 2011 11:04:47 -0600
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <4EB5653A.4000302@netwok.org>
References: <20111030001801.2f52ceb2@pitrou.net> <4EB5653A.4000302@netwok.org>
Message-ID: <CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>

On Sat, Nov 5, 2011 at 10:32 AM, ?ric Araujo <merwok at netwok.org> wrote:
> Hi,
>
>> I would like to propose the following PEP for discussion and, if
>> possible, acceptance. I think the proposal shouldn't be too
>> controversial (I find it quite simple and straightforward myself :-)).
>> [snip PEP]
>
> +1.
>
> For nested functions, I too think that 'f.<locals>.g' has too many dots;
> I like '<local in f>.g' or '<f locals>.g'.

I like it too but don't think it's too many dots.

The function from which the locals came _could_ be rolled into the
brackets.  However, in the context of some object (like the class X to
which f belongs), 'X.f.<locals of f>.g' makes more sense in that case
than 'X.<locals of f>.g', since the locals is related to f and not X.
But, then the f is sort of redundant, so you go back to
'X.f.<locals>.g', and '<locals>' is still sort of unambiguous.

The disconnect is that <locals> is an externally anonymous namespace
resulting from a call, rather than bound to any external namespace
(like an object).  Perhaps it would be appropriate to use
'X.f().<locals>.g' to make that clear.

-eric


>
> polka-dots-ly yours
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From ron3200 at gmail.com  Sat Nov  5 19:01:58 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 05 Nov 2011 13:01:58 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
 functions
In-Reply-To: <CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net> <4EB5653A.4000302@netwok.org>
	<CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>
Message-ID: <1320516118.14236.11.camel@Gutsy>

On Sat, 2011-11-05 at 11:04 -0600, Eric Snow wrote:
> On Sat, Nov 5, 2011 at 10:32 AM, ?ric Araujo <merwok at netwok.org> wrote:
> > Hi,
> >
> >> I would like to propose the following PEP for discussion and, if
> >> possible, acceptance. I think the proposal shouldn't be too
> >> controversial (I find it quite simple and straightforward myself :-)).
> >> [snip PEP]
> >
> > +1.
> >
> > For nested functions, I too think that 'f.<locals>.g' has too many dots;
> > I like '<local in f>.g' or '<f locals>.g'.
> 
> I like it too but don't think it's too many dots.
> 
> The function from which the locals came _could_ be rolled into the
> brackets.  However, in the context of some object (like the class X to
> which f belongs), 'X.f.<locals of f>.g' makes more sense in that case
> than 'X.<locals of f>.g', since the locals is related to f and not X.
> But, then the f is sort of redundant, so you go back to
> 'X.f.<locals>.g', and '<locals>' is still sort of unambiguous.
> 
> The disconnect is that <locals> is an externally anonymous namespace
> resulting from a call, rather than bound to any external namespace
> (like an object).  Perhaps it would be appropriate to use
> 'X.f().<locals>.g' to make that clear.

I think if you consider locals in f as an implementation detail of f's
name space rather than a sub item of f, it's not as confusing.  It's
better to think of locals as being part of f, rather than in f.  That is
why <f locals> makes more sense than f.<locals>.  For example locals is
in f's frame object, so if you follow that reasoning you get.
f.<frame>.<locals>,  but I don't think we need all that.

Hmmm...  I think it actually should be spelled...

      f.<local g>

Following a pattern of...

      x                 object x
      x.f               f in object x
      x.f.<local g>     local g in f in x

That's both clear and concise.

Cheers,
   Ron
















From tjreedy at udel.edu  Sat Nov  5 19:51:34 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 05 Nov 2011 14:51:34 -0400
Subject: [Python-ideas] Need way to check if a `datetime.timedelta` is
	positive
In-Reply-To: <28750278.2158.1320502042916.JavaMail.geo-discussion-forums@yqhd1>
References: <28750278.2158.1320502042916.JavaMail.geo-discussion-forums@yqhd1>
Message-ID: <j940k3$c1d$1@dough.gmane.org>

On 11/5/2011 10:07 AM, Ram Rachum wrote:

> my_timedelta >= datetime.timedelta(0)

IMHO, This *is* the proper way. If doing repeatedly, put
TD0 = datetime.timedelta(0)
at the top.

-- 
Terry Jan Reedy



From ncoghlan at gmail.com  Sat Nov  5 23:31:17 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 6 Nov 2011 08:31:17 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <1320516118.14236.11.camel@Gutsy>
References: <20111030001801.2f52ceb2@pitrou.net> <4EB5653A.4000302@netwok.org>
	<CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>
	<1320516118.14236.11.camel@Gutsy>
Message-ID: <CADiSq7dOKv_kVknVA-OTfJt21Hi2C2G-u50yNAc2uivUsjnX3A@mail.gmail.com>

I think you're all overthinking this. We want a notation that:
1. Makes it clear attribute access won't work
2. Indicates where the nested definition really lives
3. Gives a reasonable error message if you do try to evaluate it

"f.<locals>.g" ticks all those boxes, so it's fine.

--
Nick Coghlan (via Gmail on Android, so likely to be more terse than usual)
On Nov 6, 2011 4:02 AM, "Ron Adam" <ron3200 at gmail.com> wrote:

> On Sat, 2011-11-05 at 11:04 -0600, Eric Snow wrote:
> > On Sat, Nov 5, 2011 at 10:32 AM, ?ric Araujo <merwok at netwok.org> wrote:
> > > Hi,
> > >
> > >> I would like to propose the following PEP for discussion and, if
> > >> possible, acceptance. I think the proposal shouldn't be too
> > >> controversial (I find it quite simple and straightforward myself :-)).
> > >> [snip PEP]
> > >
> > > +1.
> > >
> > > For nested functions, I too think that 'f.<locals>.g' has too many
> dots;
> > > I like '<local in f>.g' or '<f locals>.g'.
> >
> > I like it too but don't think it's too many dots.
> >
> > The function from which the locals came _could_ be rolled into the
> > brackets.  However, in the context of some object (like the class X to
> > which f belongs), 'X.f.<locals of f>.g' makes more sense in that case
> > than 'X.<locals of f>.g', since the locals is related to f and not X.
> > But, then the f is sort of redundant, so you go back to
> > 'X.f.<locals>.g', and '<locals>' is still sort of unambiguous.
> >
> > The disconnect is that <locals> is an externally anonymous namespace
> > resulting from a call, rather than bound to any external namespace
> > (like an object).  Perhaps it would be appropriate to use
> > 'X.f().<locals>.g' to make that clear.
>
> I think if you consider locals in f as an implementation detail of f's
> name space rather than a sub item of f, it's not as confusing.  It's
> better to think of locals as being part of f, rather than in f.  That is
> why <f locals> makes more sense than f.<locals>.  For example locals is
> in f's frame object, so if you follow that reasoning you get.
> f.<frame>.<locals>,  but I don't think we need all that.
>
> Hmmm...  I think it actually should be spelled...
>
>      f.<local g>
>
> Following a pattern of...
>
>      x                 object x
>      x.f               f in object x
>      x.f.<local g>     local g in f in x
>
> That's both clear and concise.
>
> Cheers,
>   Ron
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111106/301c8a2d/attachment.html>

From greg.ewing at canterbury.ac.nz  Sun Nov  6 01:09:39 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 06 Nov 2011 13:09:39 +1300
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <1320300930.10353.78.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy> <4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy>
Message-ID: <4EB5D043.8050106@canterbury.ac.nz>

Ron Adam wrote:
> On Thu, 2011-11-03 at 14:47 +1300, Greg Ewing wrote:
> 
>>However, if something other than 'yield' is used for coroutine
>>suspension -- such as a 'coyield' keyword or coyield() function --
>>then I think this problem becomes solvable. In a cogenerator
>>(i.e. a generator running in coroutine mode), 'coyield' would
>>do what 'yield' does in normal mode (simply suspend the frame),
>>and 'yield(value)' would raise StopIteration(value).
> 
> Well it sounds reasonable,  but how would that actually work?  What if
> the coroutine is paused at coyield, and you need to do a next rather
> than a conext?

That situation shouldn't occur, because if a generator is
suspended at a coyield, it's already in the middle of one
next() call, and you shouldn't be trying to start another
one until the first one is finished.

If you try, you should get an exception, just as happens
now if you try to invoke a generator's next() method
reentrantly:

Python 2.7 (r27:82500, Oct 15 2010, 21:14:33)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> def g():
...  G.next()
...  yield
...
 >>> G = g()
 >>> G.next()
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<stdin>", line 2, in g
ValueError: generator already executing

> And that is the whole problem... trying to make this all un_coconfusing
> to the average python programmer.  If it's coconfusing to us, they don't
> have a chance.  ;-)

Yes, I'm leaning back towards a completely separate protocol
now. We seem to need a number of new protocol features in
any case, and allowing the two protocols to overlap is just
creating needless confusion, I think.

-- 
Greg


From guido at python.org  Sun Nov  6 02:37:14 2011
From: guido at python.org (Guido van Rossum)
Date: Sat, 5 Nov 2011 18:37:14 -0700
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <CADiSq7dOKv_kVknVA-OTfJt21Hi2C2G-u50yNAc2uivUsjnX3A@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net> <4EB5653A.4000302@netwok.org>
	<CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>
	<1320516118.14236.11.camel@Gutsy>
	<CADiSq7dOKv_kVknVA-OTfJt21Hi2C2G-u50yNAc2uivUsjnX3A@mail.gmail.com>
Message-ID: <CAP7+vJ+DG=_hPRpw6qUc50FJKhSXD5QBko=AicQ_XwLQHS+X1A@mail.gmail.com>

On Sat, Nov 5, 2011 at 3:31 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I think you're all overthinking this. We want a notation that:
> 1. Makes it clear attribute access won't work
> 2. Indicates where the nested definition really lives
> 3. Gives a reasonable error message if you do try to evaluate it
>
> "f.<locals>.g" ticks all those boxes, so it's fine.
>

+1. Let's stop the bikeshedding.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111105/4245aed7/attachment.html>

From mwm at mired.org  Sun Nov  6 06:03:26 2011
From: mwm at mired.org (Mike Meyer)
Date: Sat, 5 Nov 2011 22:03:26 -0700
Subject: [Python-ideas] A concurrency survey of sorts
In-Reply-To: <CADiSq7cEQf=4+froyL8ng7fzbtMROCsg06if3ati-tLyF93PZw@mail.gmail.com>
References: <CACBComF9jZSG6QjMVSpdxc0DjhVd-m-xRFXJ=X08CVgGjOBZkQ@mail.gmail.com>
	<CAGmFidZqddEE0+RqS_YvRuZz2SCUKmQyNtR=VRhJ8doN6VUGXg@mail.gmail.com>
	<CACBComEFc-AjZ02jD9jsVFwMaFcf=kQ9T+9i68U-JD3tMBmnSA@mail.gmail.com>
	<CAD=7U2Bzw8RwzQ2ijLS2fUFzr4K4Ve8DR+KA33xiMNdErfJvnw@mail.gmail.com>
	<20111104190936.GA12074@idyll.org>
	<CAD=7U2C5sU63YEMUrX55fNkwpKyU0=kf7LMcV4sP_zvs+GetyQ@mail.gmail.com>
	<CADiSq7cEQf=4+froyL8ng7fzbtMROCsg06if3ati-tLyF93PZw@mail.gmail.com>
Message-ID: <20111105220326.2f937942@bhuda.mired.org>

On Sat, 5 Nov 2011 16:34:06 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Nov 5, 2011 at 5:20 AM, Mike Meyer <mwm at mired.org> wrote:
> > That's one of the ideas. The goal is to get the Python processor to
> > help locate concurrency bugs. Since the threading module
> > communications model is "everything is shared by default", I'm pretty
> > sure it's going to break code that depends on it. But that's why it's
> > a proposal for Python 4 or something even further away.
> I'd suggest it's actually a proposal for a language that *isn't
> Python*. There's no way an idea like this would ever be accepted into
> Python itself without real world experimentation in another context.
> It's going to be hard to get anyone interested enough to seriously try
> it out, but I'd suggest going the "Python-based language" route (ala
> Boo and Cython) rather than claiming the idea is Python as such.

That's been obvious from the start. But this does raise an interesting
question: assuming that this change could be made in a way that
the affect on most single-threaded code is minor (which is assuming a
*lot*), is the fact that it will break almost all concurrent code
enough to mean that it's really for a language other than Python?

     Thanks,
     <mike
-- 
Mike Meyer <mwm at mired.org>		http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org


From ron3200 at gmail.com  Sun Nov  6 08:20:31 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Sun, 06 Nov 2011 01:20:31 -0600
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <4EB5D043.8050106@canterbury.ac.nz>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy> <4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy> <4EB5D043.8050106@canterbury.ac.nz>
Message-ID: <1320564031.30197.100.camel@Gutsy>

On Sun, 2011-11-06 at 13:09 +1300, Greg Ewing wrote:
> Ron Adam wrote:
> > On Thu, 2011-11-03 at 14:47 +1300, Greg Ewing wrote:
> > 
> >>However, if something other than 'yield' is used for coroutine
> >>suspension -- such as a 'coyield' keyword or coyield() function --
> >>then I think this problem becomes solvable. In a cogenerator
> >>(i.e. a generator running in coroutine mode), 'coyield' would
> >>do what 'yield' does in normal mode (simply suspend the frame),
> >>and 'yield(value)' would raise StopIteration(value).
> > 
> > Well it sounds reasonable,  but how would that actually work?  What if
> > the coroutine is paused at coyield, and you need to do a next rather
> > than a conext?
> 
> That situation shouldn't occur, because if a generator is
> suspended at a coyield, it's already in the middle of one
> next() call, and you shouldn't be trying to start another
> one until the first one is finished.

Yes, I figured that out just a little before you posted this.

A coyield suspends it in between normal yield statements, and you dont
want to steel the value from the the next next(), .send(), or .throw()
call.  Which is why those won't work to continue as well.

What is needed is a different path out (and back) to the coroutine that
doesn't interfere with the standard yield behavior.  Or as you describe
it here.

> Yes, I'm leaning back towards a completely separate protocol
> now. We seem to need a number of new protocol features in
> any case, and allowing the two protocols to overlap is just
> creating needless confusion, I think.

I think adding a .resume() method to all generators would be good.  That
keeps it simple.  It would just raise an exception in the case of a
non-suspended generator.  Which may be useful in a loop as well.

Looking at ceval.c, I think Nick's suggestion of adding a new
WHY_SUSPEND would be good.  And along with it, a TARGET(GEN_SUSPEND)
block that sets up a SuspendIteration exception, and returns WHY_SUSPEND
to the generator object.

At that point it could save anything it needs before raising the
exception, and when the resume method is called, restore what it needs
before calling the frame evel loop.

I still haven't quite worked out how to get back to the original
next(), .send() or .throw() call.

Cheers,
   Ron


> > And that is the whole problem... trying to make this all un_coconfusing
> > to the average python programmer.  If it's coconfusing to us, they don't
> > have a chance.  ;-)
> 






From ubershmekel at gmail.com  Sun Nov  6 09:56:36 2011
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Sun, 6 Nov 2011 10:56:36 +0200
Subject: [Python-ideas] Renaming the "test" package
Message-ID: <CANSw7KxjTv6CX2rrY1Jc+s3GQRSfnLG0AgWNoM9kXLnQCA_3Cw@mail.gmail.com>

For python4, the "test" package should actually be named
__pythontest__ or something similar. There are 2 million results for
""test.py"" on google.

What do you think?

--Yuval


From pyideas at rebertia.com  Sun Nov  6 10:16:02 2011
From: pyideas at rebertia.com (Chris Rebert)
Date: Sun, 6 Nov 2011 01:16:02 -0800
Subject: [Python-ideas] Renaming the "test" package
In-Reply-To: <CANSw7KxjTv6CX2rrY1Jc+s3GQRSfnLG0AgWNoM9kXLnQCA_3Cw@mail.gmail.com>
References: <CANSw7KxjTv6CX2rrY1Jc+s3GQRSfnLG0AgWNoM9kXLnQCA_3Cw@mail.gmail.com>
Message-ID: <CAMZYqRRi3oGei7ZgRyMhcx4kYMLOnWCEXyLXkkzTN0=rBu4miw@mail.gmail.com>

On Sun, Nov 6, 2011 at 1:56 AM, Yuval Greenfield <ubershmekel at gmail.com> wrote:
> For python4, the "test" package should actually be named
> __pythontest__ or something similar. There are 2 million results for
> ""test.py"" on google.

I don't get what problem you're suggesting there is that such a rename
would solve.
Python has these honking great things called namespaces (in this case,
packages); they make the existence of multiple Python files with the
same name a non-issue (just don't use non-absolute imports).

Cheers,
Chris
--
http://rebertia.com


From ncoghlan at gmail.com  Sun Nov  6 13:21:01 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 6 Nov 2011 22:21:01 +1000
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
	protocol
In-Reply-To: <1320564031.30197.100.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy>
	<4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy>
	<4EB5D043.8050106@canterbury.ac.nz>
	<1320564031.30197.100.camel@Gutsy>
Message-ID: <CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>

On Sun, Nov 6, 2011 at 5:20 PM, Ron Adam <ron3200 at gmail.com> wrote:
> What is needed is a different path out (and back) to the coroutine that
> doesn't interfere with the standard yield behavior. ?Or as you describe
> it here.

Did my post of the thread+queue based implementation of a coroutine
API design concept not go through? It demonstrated exactly the need
for a separate I/O channel independent of the call/return and
next/send/throw/yield channels.

>> Yes, I'm leaning back towards a completely separate protocol
>> now. We seem to need a number of new protocol features in
>> any case, and allowing the two protocols to overlap is just
>> creating needless confusion, I think.
>
> I think adding a .resume() method to all generators would be good. ?That
> keeps it simple. ?It would just raise an exception in the case of a
> non-suspended generator. ?Which may be useful in a loop as well.
>
> Looking at ceval.c, I think Nick's suggestion of adding a new
> WHY_SUSPEND would be good. ?And along with it, a TARGET(GEN_SUSPEND)
> block that sets up a SuspendIteration exception, and returns WHY_SUSPEND
> to the generator object.

The whole point of a separate WHY_SUSPEND is that you *wouldn't*
unwind the stack at all - you'd just leave it in place and return to
the point that called into the coroutine in the first place.

Exceptions are too destructive to the stack to be usable for this task
(as soon as you hit an except or finally clause, the execution state
in the frame gets modified to invoke them, thus meaning you can no
longer resume that stack correctly)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ron3200 at gmail.com  Sun Nov  6 16:03:04 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Sun, 06 Nov 2011 09:03:04 -0600
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy> <4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy> <4EB5D043.8050106@canterbury.ac.nz>
	<1320564031.30197.100.camel@Gutsy>
	<CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>
Message-ID: <1320591784.5049.10.camel@Gutsy>

On Sun, 2011-11-06 at 22:21 +1000, Nick Coghlan wrote:
> On Sun, Nov 6, 2011 at 5:20 PM, Ron Adam <ron3200 at gmail.com> wrote:
> > What is needed is a different path out (and back) to the coroutine that
> > doesn't interfere with the standard yield behavior.  Or as you describe
> > it here.
> 
> Did my post of the thread+queue based implementation of a coroutine
> API design concept not go through? It demonstrated exactly the need
> for a separate I/O channel independent of the call/return and
> next/send/throw/yield channels.

Ok, I found it and the link to the implantation.  

Thanks for the reminder, I'll check it out tonight when I get back.


> >> Yes, I'm leaning back towards a completely separate protocol
> >> now. We seem to need a number of new protocol features in
> >> any case, and allowing the two protocols to overlap is just
> >> creating needless confusion, I think.
> >
> > I think adding a .resume() method to all generators would be good.  That
> > keeps it simple.  It would just raise an exception in the case of a
> > non-suspended generator.  Which may be useful in a loop as well.
> >
> > Looking at ceval.c, I think Nick's suggestion of adding a new
> > WHY_SUSPEND would be good.  And along with it, a TARGET(GEN_SUSPEND)
> > block that sets up a SuspendIteration exception, and returns WHY_SUSPEND
> > to the generator object.
> 
> The whole point of a separate WHY_SUSPEND is that you *wouldn't*
> unwind the stack at all - you'd just leave it in place and return to
> the point that called into the coroutine in the first place.

Well, that was what I was trying for.  In any event, I'm learning a lot
more by trying to actually do it, than just speculate about it.  I'm at
the point of learning just how exceptions interact with the frames.  And
I'm pretty sure I'll just confirm what you say below, but with a much
better understanding of how it all works.

Cheers,
   Ron

> Exceptions are too destructive to the stack to be usable for this task
> (as soon as you hit an except or finally clause, the execution state
> in the frame gets modified to invoke them, thus meaning you can no
> longer resume that stack correctly)
> 
> Cheers,
> Nick.
> 




From greg.ewing at canterbury.ac.nz  Sun Nov  6 21:43:17 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 07 Nov 2011 09:43:17 +1300
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <1320564031.30197.100.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy> <4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy> <4EB5D043.8050106@canterbury.ac.nz>
	<1320564031.30197.100.camel@Gutsy>
Message-ID: <4EB6F165.7050800@canterbury.ac.nz>

Ron Adam wrote:

> I still haven't quite worked out how to get back to the original
> next(), .send() or .throw() call.

Keeping the protocols fully separated requires more than
just adding methods to the generator. It also requires either
its own version of the yield-from chain or some other way
of keeping track of the stack of active generators. Once
you have that, it becomes clearer how to get back to the
right place.

-- 
Greg


From ethan at stoneleaf.us  Sun Nov  6 22:22:29 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 06 Nov 2011 13:22:29 -0800
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <1320468779.1911.39.camel@Gutsy>
References: <20111030001801.2f52ceb2@pitrou.net>	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>	<20111104203842.7de86289@pitrou.net>	<CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>	<20111104231523.GB2306@chopin.edu.pl>
	<1320468779.1911.39.camel@Gutsy>
Message-ID: <4EB6FA95.10801@stoneleaf.us>

Ron Adam wrote:
> Better, but I don't think you need to nest it to indicate that
> relationship. I would use a colon instead of a dot to indicate a
> namespace relationship.
>    
>     f.g    g is an attribute of f

Being an attribute of an object /is/ being in that object's namespace 
(functions being the obvious exception).

~Ethan~


From ethan at stoneleaf.us  Sun Nov  6 22:18:38 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 06 Nov 2011 13:18:38 -0800
Subject: [Python-ideas] PEP 3155 - Qualified name for classes
	and	functions
In-Reply-To: <CAP7+vJ+i6q23ube6d3S1yPLe4t=On3=F_X3U03X0s5pehYR7XA@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net>	<CADiSq7dV-Gv0ubJMLAR43fQVU7QetGPaTpQaYJ+4Fd9arXvO9w@mail.gmail.com>	<20111104203842.7de86289@pitrou.net>	<CAP7+vJ+Pm09uhDo6Je9MSEAaTvC+gaW5sXSNo_vj1-wfy2q0VQ@mail.gmail.com>	<20111104231523.GB2306@chopin.edu.pl>
	<CAP7+vJ+i6q23ube6d3S1yPLe4t=On3=F_X3U03X0s5pehYR7XA@mail.gmail.com>
Message-ID: <4EB6F9AE.7000605@stoneleaf.us>

Guido van Rossum wrote:
> In the end I like the non-nested version better.

+1

~Ethan~


From solipsis at pitrou.net  Mon Nov  7 01:04:35 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 7 Nov 2011 01:04:35 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net> <4EB5653A.4000302@netwok.org>
	<CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>
	<1320516118.14236.11.camel@Gutsy>
	<CADiSq7dOKv_kVknVA-OTfJt21Hi2C2G-u50yNAc2uivUsjnX3A@mail.gmail.com>
	<CAP7+vJ+DG=_hPRpw6qUc50FJKhSXD5QBko=AicQ_XwLQHS+X1A@mail.gmail.com>
Message-ID: <20111107010435.7d6be84a@pitrou.net>

On Sat, 5 Nov 2011 18:37:14 -0700
Guido van Rossum <guido at python.org> wrote:
> On Sat, Nov 5, 2011 at 3:31 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> > I think you're all overthinking this. We want a notation that:
> > 1. Makes it clear attribute access won't work
> > 2. Indicates where the nested definition really lives
> > 3. Gives a reasonable error message if you do try to evaluate it
> >
> > "f.<locals>.g" ticks all those boxes, so it's fine.
> >
> 
> +1. Let's stop the bikeshedding.

I've now updated the PEP and the implementation for "f.<locals>.g" to be
the __qname__ value in that case.

Regards

Antoine.




From greg at krypto.org  Mon Nov  7 02:38:00 2011
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 6 Nov 2011 17:38:00 -0800
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
Message-ID: <CAGE7PNL62MwACyd05JqXi+R-WPMKPQ=bmzN-Q=zcq6HVErW_fQ@mail.gmail.com>

On Tue, Nov 1, 2011 at 1:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> No, the traceback info is added by the eval loop itself. Remember that
> when you raise an exception *type* (rather than an instance), the
> exception doesn't get instantiated until it gets caught somewhere -
> the eval loop maintains the unwinding stack for the traceback as part
> of the thread state until it is time to attach it to the exception
> object.
>

I did not know that.  Is there a good reason for doing this?  It seems
unnecessarily complicated.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111106/95726310/attachment.html>

From ncoghlan at gmail.com  Mon Nov  7 02:46:43 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 7 Nov 2011 11:46:43 +1000
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
In-Reply-To: <CAGE7PNL62MwACyd05JqXi+R-WPMKPQ=bmzN-Q=zcq6HVErW_fQ@mail.gmail.com>
References: <CAGE7PNL62MwACyd05JqXi+R-WPMKPQ=bmzN-Q=zcq6HVErW_fQ@mail.gmail.com>
Message-ID: <CADiSq7d+nNryst2qAWktUMLMzSYz7TS7majAz8MvRk_TAO4k7Q@mail.gmail.com>

On Mon, Nov 7, 2011 at 11:38 AM, Gregory P. Smith <greg at krypto.org> wrote:
>
> On Tue, Nov 1, 2011 at 1:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> No, the traceback info is added by the eval loop itself. Remember that
>> when you raise an exception *type* (rather than an instance), the
>> exception doesn't get instantiated until it gets caught somewhere -
>> the eval loop maintains the unwinding stack for the traceback as part
>> of the thread state until it is time to attach it to the exception
>> object.
>
> I did not know that. ?Is there a good reason for doing this? ?It seems
> unnecessarily complicated.

You'll have to ask Guido that one - it's been like that since long
before I got involved in hacking on the interpreter.

It's possibly a lingering artifact of the old "exceptions are just
strings" design, since traceback storage didn't get added to exception
instances until Py3k.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From guido at python.org  Mon Nov  7 02:52:14 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 6 Nov 2011 17:52:14 -0800
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
In-Reply-To: <CADiSq7d+nNryst2qAWktUMLMzSYz7TS7majAz8MvRk_TAO4k7Q@mail.gmail.com>
References: <CAGE7PNL62MwACyd05JqXi+R-WPMKPQ=bmzN-Q=zcq6HVErW_fQ@mail.gmail.com>
	<CADiSq7d+nNryst2qAWktUMLMzSYz7TS7majAz8MvRk_TAO4k7Q@mail.gmail.com>
Message-ID: <CAP7+vJLD1uT3uxH-9Q-mDx-+17jCC2fGjSKBF-XSCvGHkajj2Q@mail.gmail.com>

On Sun, Nov 6, 2011 at 5:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Mon, Nov 7, 2011 at 11:38 AM, Gregory P. Smith <greg at krypto.org> wrote:
> >
> > On Tue, Nov 1, 2011 at 1:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >>
> >> No, the traceback info is added by the eval loop itself. Remember that
> >> when you raise an exception *type* (rather than an instance), the
> >> exception doesn't get instantiated until it gets caught somewhere -
> >> the eval loop maintains the unwinding stack for the traceback as part
> >> of the thread state until it is time to attach it to the exception
> >> object.
> >
> > I did not know that.  Is there a good reason for doing this?  It seems
> > unnecessarily complicated.
>
> You'll have to ask Guido that one - it's been like that since long
> before I got involved in hacking on the interpreter.
>
> It's possibly a lingering artifact of the old "exceptions are just
> strings" design, since traceback storage didn't get added to exception
> instances until Py3k.
>

No, it was actually introduced when exceptions became classes. It is an
optimization that we deemed important at the time: to avoid instantiating
the class when it's going to be caught by C code that doesn't care about
the instance. A common example is StopIteration, but there are probably
plenty of other situations like it. We may even have benchmarked for-loops
with and without this -- while the exception only happens once per
for-loop, there are a lot of for-loops, many of which iterate over small
sequences, and it adds up.

I'm not sure that it's still that important -- in fact I'm not sure Python
3 still has this behavior.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111106/292b82c6/attachment.html>

From ncoghlan at gmail.com  Mon Nov  7 03:03:46 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 7 Nov 2011 12:03:46 +1000
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
In-Reply-To: <CAP7+vJLD1uT3uxH-9Q-mDx-+17jCC2fGjSKBF-XSCvGHkajj2Q@mail.gmail.com>
References: <CAGE7PNL62MwACyd05JqXi+R-WPMKPQ=bmzN-Q=zcq6HVErW_fQ@mail.gmail.com>
	<CADiSq7d+nNryst2qAWktUMLMzSYz7TS7majAz8MvRk_TAO4k7Q@mail.gmail.com>
	<CAP7+vJLD1uT3uxH-9Q-mDx-+17jCC2fGjSKBF-XSCvGHkajj2Q@mail.gmail.com>
Message-ID: <CADiSq7dWx5TuKXrTgxwS9Sotat-=kSPLvMGzJQqNCxXrTo=rjQ@mail.gmail.com>

On Mon, Nov 7, 2011 at 11:52 AM, Guido van Rossum <guido at python.org> wrote:
> No, it was actually introduced when exceptions became classes. It is an
> optimization that we deemed important at the time: to avoid instantiating
> the class when it's going to be caught by C code that doesn't care about the
> instance. A common example is StopIteration, but there are probably plenty
> of other situations like it. We may even have benchmarked for-loops with and
> without this -- while the exception only happens once per for-loop, there
> are a lot of for-loops, many of which iterate over small sequences, and it
> adds up.

Ah, thanks - interesting to know.

> I'm not sure that it's still that important -- in fact I'm not sure Python 3
> still has this behavior.

I'm fairly sure it does (it was the 3.x version of ceval that I was
reading to remind myself of how the stack unwinding process actually
works and the call to PyErr_NormalizeException() is still in there).
It's just that without the multiple argument forms of the raise
statement, pure Python code can only exploit it for exceptions without
any arguments (so it still works for the StopIteration optimisation).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From solipsis at pitrou.net  Mon Nov  7 04:04:53 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 7 Nov 2011 04:04:53 +0100
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
References: <CAGE7PNL62MwACyd05JqXi+R-WPMKPQ=bmzN-Q=zcq6HVErW_fQ@mail.gmail.com>
	<CADiSq7d+nNryst2qAWktUMLMzSYz7TS7majAz8MvRk_TAO4k7Q@mail.gmail.com>
	<CAP7+vJLD1uT3uxH-9Q-mDx-+17jCC2fGjSKBF-XSCvGHkajj2Q@mail.gmail.com>
	<CADiSq7dWx5TuKXrTgxwS9Sotat-=kSPLvMGzJQqNCxXrTo=rjQ@mail.gmail.com>
Message-ID: <20111107040453.594180e6@pitrou.net>

On Mon, 7 Nov 2011 12:03:46 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> > I'm not sure that it's still that important -- in fact I'm not sure Python 3
> > still has this behavior.
> 
> I'm fairly sure it does (it was the 3.x version of ceval that I was
> reading to remind myself of how the stack unwinding process actually
> works and the call to PyErr_NormalizeException() is still in there).
> It's just that without the multiple argument forms of the raise
> statement, pure Python code can only exploit it for exceptions without
> any arguments (so it still works for the StopIteration optimisation).

It's too late for me to go and take a look, but I'm not sure it does.
Unless the performance loss is significant, I would be in favour of
simplifying all this and always instantiating exceptions. Exception
handling in the eval loop and exception "normalization" is a nasty area
of the interpreter.

Regards

Antoine.




From cs at zip.com.au  Mon Nov  7 08:52:55 2011
From: cs at zip.com.au (Cameron Simpson)
Date: Mon, 7 Nov 2011 18:52:55 +1100
Subject: [Python-ideas] raising an exception type doesn't instantiate it
	until it gets	caught
In-Reply-To: <20111107040453.594180e6@pitrou.net>
References: <20111107040453.594180e6@pitrou.net>
Message-ID: <20111107075255.GA16482@cskk.homeip.net>

On 07Nov2011 04:04, Antoine Pitrou <solipsis at pitrou.net> wrote:
| On Mon, 7 Nov 2011 12:03:46 +1000
| Nick Coghlan <ncoghlan at gmail.com> wrote:
| > > I'm not sure that it's still that important -- in fact I'm not sure Python 3
| > > still has this behavior.
| > 
| > I'm fairly sure it does (it was the 3.x version of ceval that I was
| > reading to remind myself of how the stack unwinding process actually
| > works and the call to PyErr_NormalizeException() is still in there).
| > It's just that without the multiple argument forms of the raise
| > statement, pure Python code can only exploit it for exceptions without
| > any arguments (so it still works for the StopIteration optimisation).
| 
| It's too late for me to go and take a look, but I'm not sure it does.
| Unless the performance loss is significant, I would be in favour of
| simplifying all this and always instantiating exceptions. Exception
| handling in the eval loop and exception "normalization" is a nasty area
| of the interpreter.

I presume StopIteration would get instantiated to a singleton, like
NoneType to None? Just asking.
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Login incorrect.
Only perfect spellers may
enter this system.
- Haiku Error Messages http://www.salonmagazine.com/21st/chal/1998/02/10chal2.html


From steve at pearwood.info  Mon Nov  7 11:15:33 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 07 Nov 2011 21:15:33 +1100
Subject: [Python-ideas] raising an exception type doesn't instantiate
 it	until it gets	caught
In-Reply-To: <20111107075255.GA16482@cskk.homeip.net>
References: <20111107040453.594180e6@pitrou.net>
	<20111107075255.GA16482@cskk.homeip.net>
Message-ID: <4EB7AFC5.3040509@pearwood.info>

Cameron Simpson wrote:

> I presume StopIteration would get instantiated to a singleton, like
> NoneType to None? Just asking.


Are you asking about what it should be, or what it is? Either way:

 >>> a = StopIteration('spam')
 >>> b = StopIteration('ham')
 >>> a is b
False



-- 
Steven


From solipsis at pitrou.net  Mon Nov  7 11:35:44 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 7 Nov 2011 11:35:44 +0100
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
References: <20111107040453.594180e6@pitrou.net>
	<20111107075255.GA16482@cskk.homeip.net>
Message-ID: <20111107113544.6a413544@pitrou.net>

On Mon, 7 Nov 2011 18:52:55 +1100
Cameron Simpson <cs at zip.com.au> wrote:
> On 07Nov2011 04:04, Antoine Pitrou <solipsis at pitrou.net> wrote:
> | On Mon, 7 Nov 2011 12:03:46 +1000
> | Nick Coghlan <ncoghlan at gmail.com> wrote:
> | > > I'm not sure that it's still that important -- in fact I'm not sure Python 3
> | > > still has this behavior.
> | > 
> | > I'm fairly sure it does (it was the 3.x version of ceval that I was
> | > reading to remind myself of how the stack unwinding process actually
> | > works and the call to PyErr_NormalizeException() is still in there).
> | > It's just that without the multiple argument forms of the raise
> | > statement, pure Python code can only exploit it for exceptions without
> | > any arguments (so it still works for the StopIteration optimisation).
> | 
> | It's too late for me to go and take a look, but I'm not sure it does.
> | Unless the performance loss is significant, I would be in favour of
> | simplifying all this and always instantiating exceptions. Exception
> | handling in the eval loop and exception "normalization" is a nasty area
> | of the interpreter.
> 
> I presume StopIteration would get instantiated to a singleton, like
> NoneType to None? Just asking.

It is impossible to use singletons for exception instances now that the
traceback is stored on them.
(there was a bug with MemoryError which was a singleton: a raised
memory error would make its traceback immortal, making memory pressure
even worse)

Regards

Antoine.




From ncoghlan at gmail.com  Mon Nov  7 13:01:24 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 7 Nov 2011 22:01:24 +1000
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
In-Reply-To: <20111107075255.GA16482@cskk.homeip.net>
References: <20111107040453.594180e6@pitrou.net>
	<20111107075255.GA16482@cskk.homeip.net>
Message-ID: <CADiSq7eM8OOerLH4PN+hNEKk9i5dw94tdzugkC-qYgC2Lx3RCA@mail.gmail.com>

On Mon, Nov 7, 2011 at 5:52 PM, Cameron Simpson <cs at zip.com.au> wrote:
>
> I presume StopIteration would get instantiated to a singleton, like
> NoneType to None? Just asking.

Even without the traceback issue Antoine mentioned, it's already the
case that StopIteration isn't a singleton in 2.x. Various pieces of
code (e.g. contextlib.contextmanager) rely on being able to tell
whether they're getting a specific StopIteration instance back or a
new one.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From barry at python.org  Mon Nov  7 16:34:24 2011
From: barry at python.org (Barry Warsaw)
Date: Mon, 7 Nov 2011 10:34:24 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net>
Message-ID: <20111107103424.4c2e2ef1@resist.wooz.org>

On Oct 30, 2011, at 12:18 AM, Antoine Pitrou wrote:

>I would like to propose the following PEP for discussion and, if
>possible, acceptance. I think the proposal shouldn't be too
>controversial (I find it quite simple and straightforward myself :-)).

Nice PEP, and +1 for the concept (with Guido's preferred format).  However,
"qname" is pretty obscure and I only guessed what the "q" stood for by reading
the title of the PEP.

It seems to me that this attribute represents the dotted path from module
globals to the object.  You have to be careful not to confuse this with a file
system path, so something like __dotted_name__, __dotted_path__, or
__full_name__ perhaps.  I don't much care, but I do think cryptic
abbreviations should be avoided.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111107/02f38a05/attachment.pgp>

From barry at python.org  Mon Nov  7 16:40:34 2011
From: barry at python.org (Barry Warsaw)
Date: Mon, 7 Nov 2011 10:40:34 -0500
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
References: <CAGE7PNL62MwACyd05JqXi+R-WPMKPQ=bmzN-Q=zcq6HVErW_fQ@mail.gmail.com>
	<CADiSq7d+nNryst2qAWktUMLMzSYz7TS7majAz8MvRk_TAO4k7Q@mail.gmail.com>
	<CAP7+vJLD1uT3uxH-9Q-mDx-+17jCC2fGjSKBF-XSCvGHkajj2Q@mail.gmail.com>
Message-ID: <20111107104034.1a808263@resist.wooz.org>

On Nov 06, 2011, at 05:52 PM, Guido van Rossum wrote:

>No, it was actually introduced when exceptions became classes. It is an
>optimization that we deemed important at the time: to avoid instantiating
>the class when it's going to be caught by C code that doesn't care about
>the instance. A common example is StopIteration, but there are probably
>plenty of other situations like it. We may even have benchmarked for-loops
>with and without this -- while the exception only happens once per
>for-loop, there are a lot of for-loops, many of which iterate over small
>sequences, and it adds up.
>
>I'm not sure that it's still that important -- in fact I'm not sure Python
>3 still has this behavior.

I did some performance testing back when this was introduced.  They numbers
aren't relevant any more, but the basic idea was to time things like
dictionary access with .get() vs. __getitem__() both when the key was in the
dictionary and when it was missing.  There were some other timing tests IIRC.
IIRC, instantiating the exception in every case was a fairly significant hit.

It would be good to see some updated tests and numbers before anything was
changed in the interpreter.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111107/2d027078/attachment.pgp>

From merwok at netwok.org  Mon Nov  7 17:19:34 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Mon, 07 Nov 2011 17:19:34 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111107103424.4c2e2ef1@resist.wooz.org>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
Message-ID: <4EB80516.8060500@netwok.org>

Le 07/11/2011 16:34, Barry Warsaw a ?crit :
> Nice PEP, and +1 for the concept (with Guido's preferred format).  However,
> "qname" is pretty obscure and I only guessed what the "q" stood for by reading
> the title of the PEP.
OTOH, qname will be easily found in search engine (one of my griefs with
the ?packaging? module name).

> It seems to me that this attribute represents the dotted path from module
> globals to the object.  You have to be careful not to confuse this with a file
> system path, so something like __dotted_name__, __dotted_path__, or
> __full_name__ perhaps.  I don't much care, but I do think cryptic
> abbreviations should be avoided.
-1 on these propositions: A dotted name or path includes the module name
for me (or for things like packaging.util.resolve_name).  I?m less
affirmative about ?full name?; it could mean ?full name in one module?,
so that __module__ + __fullname__ == the fully qualified name.  This
terminology is not terrific: I don?t think it would be good to have a
distinction between ?full name? (a.b) and ?fully qualified name?
(module.a.b).  I think ?qualified name? and ?fully qualified name? are
better terms: ?qualified? has IMO less assumptions than full (you ask
yourself ?in what way is it qualified?? and go to the docs, instead of
thinking ?ah, it?s the full name including the module? or ?ah, it?s like
__name__ but better, and without the module?), and ?fully qualified?
builds on ?qualified?.

(I?m trying to be constructive, not bikeshedding; tell me if I fail.)

__qualname__ is less cryptic and not too long; __qualifiedname__ is even
less cryptic.

Cheers


From barry at python.org  Mon Nov  7 17:25:53 2011
From: barry at python.org (Barry Warsaw)
Date: Mon, 7 Nov 2011 11:25:53 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
 functions
In-Reply-To: <4EB80516.8060500@netwok.org>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<4EB80516.8060500@netwok.org>
Message-ID: <20111107112553.35c7fffd@resist.wooz.org>

On Nov 07, 2011, at 05:19 PM, ?ric Araujo wrote:

>OTOH, qname will be easily found in search engine (one of my griefs with
>the ?packaging? module name).

Are you sure about that?  I.e. have you tried it?

>-1 on these propositions: A dotted name or path includes the module name
>for me (or for things like packaging.util.resolve_name).  I?m less
>affirmative about ?full name?; it could mean ?full name in one module?,
>so that __module__ + __fullname__ == the fully qualified name.  This
>terminology is not terrific: I don?t think it would be good to have a
>distinction between ?full name? (a.b) and ?fully qualified name?
>(module.a.b).  I think ?qualified name? and ?fully qualified name? are
>better terms: ?qualified? has IMO less assumptions than full (you ask
>yourself ?in what way is it qualified?? and go to the docs, instead of
>thinking ?ah, it?s the full name including the module? or ?ah, it?s like
>__name__ but better, and without the module?), and ?fully qualified?
>builds on ?qualified?.
>
>(I?m trying to be constructive, not bikeshedding; tell me if I fail.)
>
>__qualname__ is less cryptic and not too long; __qualifiedname__ is even
>less cryptic.

"scoped" could also be used instead of "qualified".

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111107/cd110a03/attachment.pgp>

From solipsis at pitrou.net  Mon Nov  7 17:23:06 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 7 Nov 2011 17:23:06 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<4EB80516.8060500@netwok.org>
	<20111107112553.35c7fffd@resist.wooz.org>
Message-ID: <20111107172306.25854c71@pitrou.net>

On Mon, 7 Nov 2011 11:25:53 -0500
Barry Warsaw <barry at python.org> wrote:

> On Nov 07, 2011, at 05:19 PM, ?ric Araujo wrote:
> 
> >OTOH, qname will be easily found in search engine (one of my griefs with
> >the ?packaging? module name).
> 
> Are you sure about that?  I.e. have you tried it?
> 
> >-1 on these propositions: A dotted name or path includes the module name
> >for me (or for things like packaging.util.resolve_name).  I?m less
> >affirmative about ?full name?; it could mean ?full name in one module?,
> >so that __module__ + __fullname__ == the fully qualified name.  This
> >terminology is not terrific: I don?t think it would be good to have a
> >distinction between ?full name? (a.b) and ?fully qualified name?
> >(module.a.b).  I think ?qualified name? and ?fully qualified name? are
> >better terms: ?qualified? has IMO less assumptions than full (you ask
> >yourself ?in what way is it qualified?? and go to the docs, instead of
> >thinking ?ah, it?s the full name including the module? or ?ah, it?s like
> >__name__ but better, and without the module?), and ?fully qualified?
> >builds on ?qualified?.
> >
> >(I?m trying to be constructive, not bikeshedding; tell me if I fail.)
> >
> >__qualname__ is less cryptic and not too long; __qualifiedname__ is even
> >less cryptic.
> 
> "scoped" could also be used instead of "qualified".

How about __sname__?
Ok, sorry :-)

Antoine.




From merwok at netwok.org  Mon Nov  7 18:00:57 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Mon, 07 Nov 2011 18:00:57 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111107112553.35c7fffd@resist.wooz.org>
References: <20111030001801.2f52ceb2@pitrou.net>	<20111107103424.4c2e2ef1@resist.wooz.org>	<4EB80516.8060500@netwok.org>
	<20111107112553.35c7fffd@resist.wooz.org>
Message-ID: <4EB80EC9.4080505@netwok.org>

Le 07/11/2011 17:25, Barry Warsaw a ?crit :
> On Nov 07, 2011, at 05:19 PM, ?ric Araujo wrote:
>> OTOH, qname will be easily found in search engine
> Are you sure about that?  I.e. have you tried it?
I did, and it made me remember why I didn?t find ?qname? cryptic: It is
used by XML specs (Antoine, you could mention that in the PEP).  Top
Google results for ?python qname? are XML-related.

More:
http://en.wikipedia.org/wiki/QName
http://en.wikipedia.org/wiki/Fully_qualified_name

(#include <disclaimer about wikipedia not being authoritative>)

> "scoped" could also be used instead of "qualified".
You truly are a great name generator!  +1 to either.

Cheers


From cs at zip.com.au  Mon Nov  7 21:05:29 2011
From: cs at zip.com.au (Cameron Simpson)
Date: Tue, 8 Nov 2011 07:05:29 +1100
Subject: [Python-ideas] raising an exception type doesn't instantiate it
	until it gets	caught
In-Reply-To: <4EB7AFC5.3040509@pearwood.info>
	<20111107113544.6a413544@pitrou.net>
	<CADiSq7eM8OOerLH4PN+hNEKk9i5dw94tdzugkC-qYgC2Lx3RCA@mail.gmail.com>
References: <4EB7AFC5.3040509@pearwood.info>
	<20111107113544.6a413544@pitrou.net>
	<CADiSq7eM8OOerLH4PN+hNEKk9i5dw94tdzugkC-qYgC2Lx3RCA@mail.gmail.com>
Message-ID: <20111107200529.GA10386@cskk.homeip.net>

I wrote, naively:
| > I presume StopIteration would get instantiated to a singleton, like
| > NoneType to None? Just asking.

On 07Nov2011 22:01, Nick Coghlan <ncoghlan at gmail.com> wrote:
| Even without the traceback issue Antoine mentioned, it's already the
| case that StopIteration isn't a singleton in 2.x. Various pieces of
| code (e.g. contextlib.contextmanager) rely on being able to tell
| whether they're getting a specific StopIteration instance back or a
| new one.

Interesting.

Off topic digression:

I've been slightly uncomfortable about exceptions as control flow for a
while, basicly when writing code like this:

  try:
    x = G.next()
  except StopIteration:
    # G is empty!

in that I don't entirely know that the StopIteration came from G of from
some buggy code deeper inside G that let a StopIteration out, eg by
mangling a try/except like the above. In most circumstances with other
exceptions, while you might _expect_ the exception to come from the
source you expect you don't care so much because it will indicate
failure of the operation anyway. Report or die, you don't proceed as if
the op was good. But with StopIteration one is reading "G is empty"
directly into the situation and acting as though it is normal (exit the
event loop or whatever it may imply).

On 07Nov2011 11:35, Antoine Pitrou <solipsis at pitrou.net> wrote:
| It is impossible to use singletons for exception instances now that the
| traceback is stored on them.

Ah. I had somehow thought the exception itself and the traceback were
distinct items, presented in a tuple.

On 07Nov2011 21:15, Steven D'Aprano <steve at pearwood.info> wrote:
| Are you asking about what it should be, or what it is?

The former.

| Either way:
| >>> a = StopIteration('spam')
| >>> b = StopIteration('ham')
| >>> a is b
| False

Since my question was about the proposed new behaviour when just a type
was raised, the above test wouldn't educate me. Though it does address the
behaviour of the type instantation in general.

Cheers,
-- 
Cameron Simpson <cs at zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

Carpe Datum     - John Sloan <jsloan at ncar.ucar.edu>


From greg at krypto.org  Mon Nov  7 22:43:55 2011
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 7 Nov 2011 13:43:55 -0800
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
In-Reply-To: <20111107200529.GA10386@cskk.homeip.net>
References: <4EB7AFC5.3040509@pearwood.info>
	<20111107113544.6a413544@pitrou.net>
	<CADiSq7eM8OOerLH4PN+hNEKk9i5dw94tdzugkC-qYgC2Lx3RCA@mail.gmail.com>
	<20111107200529.GA10386@cskk.homeip.net>
Message-ID: <CAGE7PN+3ez=Ckf6X4=-GcEe6yxp9ZdfzpByUV0ApXL9BCfAKug@mail.gmail.com>

On Mon, Nov 7, 2011 at 12:05 PM, Cameron Simpson <cs at zip.com.au> wrote:

> I wrote, naively:
> | > I presume StopIteration would get instantiated to a singleton, like
> | > NoneType to None? Just asking.
>
> On 07Nov2011 22:01, Nick Coghlan <ncoghlan at gmail.com> wrote:
> | Even without the traceback issue Antoine mentioned, it's already the
> | case that StopIteration isn't a singleton in 2.x. Various pieces of
> | code (e.g. contextlib.contextmanager) rely on being able to tell
> | whether they're getting a specific StopIteration instance back or a
> | new one.
>
> Interesting.
>
> Off topic digression:
>
> I've been slightly uncomfortable about exceptions as control flow for a
> while, basicly when writing code like this:
>
>  try:
>    x = G.next()
>  except StopIteration:
>    # G is empty!
>
> in that I don't entirely know that the StopIteration came from G of from
> some buggy code deeper inside G that let a StopIteration out, eg by
> mangling a try/except like the above. In most circumstances with other
> exceptions, while you might _expect_ the exception to come from the
> source you expect you don't care so much because it will indicate
> failure of the operation anyway. Report or die, you don't proceed as if
> the op was good. But with StopIteration one is reading "G is empty"
> directly into the situation and acting as though it is normal (exit the
> event loop or whatever it may imply).
>

Agreed.  Use of exceptions for this in the language feels like it was a
convenient way to do it but as the conditions aren't really
*exceptional*at all it'd be nice if there were a lighter weight
mechanism that could
skip the unneeded parts of the exception raising and handling mechanism for
the implementation.  We don't need the traceback to be stored in these
situations.

This existing logic to instantiate and associate the traceback with it only
if caught is one way to implement doing exactly that. Any other ideas?

Hackish things like a class attribute on classes being raised as an
exception, or a LightweightException class being part of its class
heirarchy used to signify if that exception should take the full path or
the fast path come to mind but could be considered equally surprising.

I'm not sure any of this is worth it but it would simplify the eval loop.
We're speaking implementation details of CPython here, not an actual change
to the language itself. (*)

-gps

(*) Please beat anybody who writes code that depends on this somewhat odd
exception instantiation timing behavior side effect over the head with a
frozen herring.



> On 07Nov2011 11:35, Antoine Pitrou <solipsis at pitrou.net> wrote:
> | It is impossible to use singletons for exception instances now that the
> | traceback is stored on them.
>
> Ah. I had somehow thought the exception itself and the traceback were
> distinct items, presented in a tuple.
>
> On 07Nov2011 21:15, Steven D'Aprano <steve at pearwood.info> wrote:
> | Are you asking about what it should be, or what it is?
>
> The former.
>
> | Either way:
> | >>> a = StopIteration('spam')
> | >>> b = StopIteration('ham')
> | >>> a is b
> | False
>
> Since my question was about the proposed new behaviour when just a type
> was raised, the above test wouldn't educate me. Though it does address the
> behaviour of the type instantation in general.
>
> Cheers,
> --
> Cameron Simpson <cs at zip.com.au> DoD#743
> http://www.cskk.ezoshosting.com/cs/
>
> Carpe Datum     - John Sloan <jsloan at ncar.ucar.edu>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111107/52e890b7/attachment.html>

From zuo at chopin.edu.pl  Mon Nov  7 22:48:16 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 7 Nov 2011 22:48:16 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
 functions
In-Reply-To: <4EB80EC9.4080505@netwok.org>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<4EB80516.8060500@netwok.org>
	<20111107112553.35c7fffd@resist.wooz.org>
	<4EB80EC9.4080505@netwok.org>
Message-ID: <20111107214816.GC2332@chopin.edu.pl>

?ric Araujo dixit (2011-11-07, 18:00):

> Le 07/11/2011 17:25, Barry Warsaw a ?crit :
> > On Nov 07, 2011, at 05:19 PM, ?ric Araujo wrote:
> >> OTOH, qname will be easily found in search engine
> > Are you sure about that?  I.e. have you tried it?
> I did, and it made me remember why I didn?t find ?qname? cryptic: It is
> used by XML specs (Antoine, you could mention that in the PEP).  Top
> Google results for ?python qname? are XML-related.
> 
> More:
> http://en.wikipedia.org/wiki/QName
> http://en.wikipedia.org/wiki/Fully_qualified_name
> 
> (#include <disclaimer about wikipedia not being authoritative>)
> 
> > "scoped" could also be used instead of "qualified".
> You truly are a great name generator!  +1 to either.

But without the underscore in the middle, please. We have
__getattribute__ and __deepcopy__, not __get_attribute__ and
__deep_copy__.

Cheers.
*j



From anacrolix at gmail.com  Mon Nov  7 23:01:24 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Tue, 8 Nov 2011 09:01:24 +1100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111107214816.GC2332@chopin.edu.pl>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<4EB80516.8060500@netwok.org>
	<20111107112553.35c7fffd@resist.wooz.org>
	<4EB80EC9.4080505@netwok.org> <20111107214816.GC2332@chopin.edu.pl>
Message-ID: <CAB4yi1Odc+KcbfNH9WNW0e+53FCjd4dsK9Z4SkZnv7QaVSKrJg@mail.gmail.com>

+1 qualname
On Nov 8, 2011 8:48 AM, "Jan Kaliszewski" <zuo at chopin.edu.pl> wrote:

> ?ric Araujo dixit (2011-11-07, 18:00):
>
> > Le 07/11/2011 17:25, Barry Warsaw a ?crit :
> > > On Nov 07, 2011, at 05:19 PM, ?ric Araujo wrote:
> > >> OTOH, qname will be easily found in search engine
> > > Are you sure about that?  I.e. have you tried it?
> > I did, and it made me remember why I didn?t find ?qname? cryptic: It is
> > used by XML specs (Antoine, you could mention that in the PEP).  Top
> > Google results for ?python qname? are XML-related.
> >
> > More:
> > http://en.wikipedia.org/wiki/QName
> > http://en.wikipedia.org/wiki/Fully_qualified_name
> >
> > (#include <disclaimer about wikipedia not being authoritative>)
> >
> > > "scoped" could also be used instead of "qualified".
> > You truly are a great name generator!  +1 to either.
>
> But without the underscore in the middle, please. We have
> __getattribute__ and __deepcopy__, not __get_attribute__ and
> __deep_copy__.
>
> Cheers.
> *j
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111108/ef07a48e/attachment.html>

From fuzzyman at gmail.com  Tue Nov  8 00:05:21 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Mon, 7 Nov 2011 23:05:21 +0000
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
In-Reply-To: <CAGE7PN+3ez=Ckf6X4=-GcEe6yxp9ZdfzpByUV0ApXL9BCfAKug@mail.gmail.com>
References: <4EB7AFC5.3040509@pearwood.info>
	<20111107113544.6a413544@pitrou.net>
	<CADiSq7eM8OOerLH4PN+hNEKk9i5dw94tdzugkC-qYgC2Lx3RCA@mail.gmail.com>
	<20111107200529.GA10386@cskk.homeip.net>
	<CAGE7PN+3ez=Ckf6X4=-GcEe6yxp9ZdfzpByUV0ApXL9BCfAKug@mail.gmail.com>
Message-ID: <CAKCKLWxq1xcfxXvFT_31Li5MgfMmVA82KDM22xa+jSx=iKOR7A@mail.gmail.com>

On 7 November 2011 21:43, Gregory P. Smith <greg at krypto.org> wrote:

>
>
> On Mon, Nov 7, 2011 at 12:05 PM, Cameron Simpson <cs at zip.com.au> wrote:
>
>> I wrote, naively:
>> | > I presume StopIteration would get instantiated to a singleton, like
>> | > NoneType to None? Just asking.
>>
>> On 07Nov2011 22:01, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> | Even without the traceback issue Antoine mentioned, it's already the
>> | case that StopIteration isn't a singleton in 2.x. Various pieces of
>> | code (e.g. contextlib.contextmanager) rely on being able to tell
>> | whether they're getting a specific StopIteration instance back or a
>> | new one.
>>
>> Interesting.
>>
>> Off topic digression:
>>
>> I've been slightly uncomfortable about exceptions as control flow for a
>> while, basicly when writing code like this:
>>
>>  try:
>>    x = G.next()
>>  except StopIteration:
>>    # G is empty!
>>
>> in that I don't entirely know that the StopIteration came from G of from
>> some buggy code deeper inside G that let a StopIteration out, eg by
>> mangling a try/except like the above. In most circumstances with other
>> exceptions, while you might _expect_ the exception to come from the
>> source you expect you don't care so much because it will indicate
>> failure of the operation anyway. Report or die, you don't proceed as if
>> the op was good. But with StopIteration one is reading "G is empty"
>> directly into the situation and acting as though it is normal (exit the
>> event loop or whatever it may imply).
>>
>
> Agreed.  Use of exceptions for this in the language feels like it was a
> convenient way to do it but as the conditions aren't really *exceptional*at all it'd be nice if there were a lighter weight mechanism that could
> skip the unneeded parts of the exception raising and handling mechanism for
> the implementation.  We don't need the traceback to be stored in these
> situations.
>
> This existing logic to instantiate and associate the traceback with it
> only if caught is one way to implement doing exactly that. Any other ideas?
>
> Hackish things like a class attribute on classes being raised as an
> exception, or a LightweightException class being part of its class
> heirarchy used to signify if that exception should take the full path or
> the fast path come to mind but could be considered equally surprising.
>
> I'm not sure any of this is worth it but it would simplify the eval loop.
> We're speaking implementation details of CPython here, not an actual change
> to the language itself. (*)
>
> -gps
>
> (*) Please beat anybody who writes code that depends on this somewhat odd
> exception instantiation timing behavior side effect over the head with a
> frozen herring.
>


Having the interpreter instantiate the exception for you allows you to do
wonderful things like this:

>>> class Foo(Exception):
...  def __new__(cls, *args):
...   return object()
...
>>> try:
...  raise Foo
... except Exception as e:
...  print (e)
...
<object object at 0x100634280>

(I know this has nothing to do with the topic being debated but for some
reason this code tickles me. Plus it used to segfault Python 3...)

All the best,


Michael



>
>
>
>> On 07Nov2011 11:35, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> | It is impossible to use singletons for exception instances now that the
>> | traceback is stored on them.
>>
>> Ah. I had somehow thought the exception itself and the traceback were
>> distinct items, presented in a tuple.
>>
>> On 07Nov2011 21:15, Steven D'Aprano <steve at pearwood.info> wrote:
>> | Are you asking about what it should be, or what it is?
>>
>> The former.
>>
>> | Either way:
>> | >>> a = StopIteration('spam')
>> | >>> b = StopIteration('ham')
>> | >>> a is b
>> | False
>>
>> Since my question was about the proposed new behaviour when just a type
>> was raised, the above test wouldn't educate me. Though it does address the
>> behaviour of the type instantation in general.
>>
>> Cheers,
>> --
>> Cameron Simpson <cs at zip.com.au> DoD#743
>> http://www.cskk.ezoshosting.com/cs/
>>
>> Carpe Datum     - John Sloan <jsloan at ncar.ucar.edu>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111107/f31ed2d7/attachment.html>

From fuzzyman at gmail.com  Tue Nov  8 00:06:58 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Mon, 7 Nov 2011 23:06:58 +0000
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
In-Reply-To: <CAKCKLWxq1xcfxXvFT_31Li5MgfMmVA82KDM22xa+jSx=iKOR7A@mail.gmail.com>
References: <4EB7AFC5.3040509@pearwood.info>
	<20111107113544.6a413544@pitrou.net>
	<CADiSq7eM8OOerLH4PN+hNEKk9i5dw94tdzugkC-qYgC2Lx3RCA@mail.gmail.com>
	<20111107200529.GA10386@cskk.homeip.net>
	<CAGE7PN+3ez=Ckf6X4=-GcEe6yxp9ZdfzpByUV0ApXL9BCfAKug@mail.gmail.com>
	<CAKCKLWxq1xcfxXvFT_31Li5MgfMmVA82KDM22xa+jSx=iKOR7A@mail.gmail.com>
Message-ID: <CAKCKLWzZ6qhE7iv1CaLhJQ-u5ObTDh7vRyH7z_d8=T6OE4KTGw@mail.gmail.com>

On 7 November 2011 23:05, Michael Foord <fuzzyman at gmail.com> wrote:

>
>
> On 7 November 2011 21:43, Gregory P. Smith <greg at krypto.org> wrote:
>
>>
>>
>> On Mon, Nov 7, 2011 at 12:05 PM, Cameron Simpson <cs at zip.com.au> wrote:
>>
>>> I wrote, naively:
>>> | > I presume StopIteration would get instantiated to a singleton, like
>>> | > NoneType to None? Just asking.
>>>
>>> On 07Nov2011 22:01, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> | Even without the traceback issue Antoine mentioned, it's already the
>>> | case that StopIteration isn't a singleton in 2.x. Various pieces of
>>> | code (e.g. contextlib.contextmanager) rely on being able to tell
>>> | whether they're getting a specific StopIteration instance back or a
>>> | new one.
>>>
>>> Interesting.
>>>
>>> Off topic digression:
>>>
>>> I've been slightly uncomfortable about exceptions as control flow for a
>>> while, basicly when writing code like this:
>>>
>>>  try:
>>>    x = G.next()
>>>  except StopIteration:
>>>    # G is empty!
>>>
>>> in that I don't entirely know that the StopIteration came from G of from
>>> some buggy code deeper inside G that let a StopIteration out, eg by
>>> mangling a try/except like the above. In most circumstances with other
>>> exceptions, while you might _expect_ the exception to come from the
>>> source you expect you don't care so much because it will indicate
>>> failure of the operation anyway. Report or die, you don't proceed as if
>>> the op was good. But with StopIteration one is reading "G is empty"
>>> directly into the situation and acting as though it is normal (exit the
>>> event loop or whatever it may imply).
>>>
>>
>> Agreed.  Use of exceptions for this in the language feels like it was a
>> convenient way to do it but as the conditions aren't really *exceptional*at all it'd be nice if there were a lighter weight mechanism that could
>> skip the unneeded parts of the exception raising and handling mechanism for
>> the implementation.  We don't need the traceback to be stored in these
>> situations.
>>
>> This existing logic to instantiate and associate the traceback with it
>> only if caught is one way to implement doing exactly that. Any other ideas?
>>
>> Hackish things like a class attribute on classes being raised as an
>> exception, or a LightweightException class being part of its class
>> heirarchy used to signify if that exception should take the full path or
>> the fast path come to mind but could be considered equally surprising.
>>
>> I'm not sure any of this is worth it but it would simplify the eval
>> loop.  We're speaking implementation details of CPython here, not an actual
>> change to the language itself. (*)
>>
>> -gps
>>
>> (*) Please beat anybody who writes code that depends on this somewhat odd
>> exception instantiation timing behavior side effect over the head with a
>> frozen herring.
>>
>
>
> Having the interpreter instantiate the exception for you allows you to do
> wonderful things like this:
>
> >>> class Foo(Exception):
> ...  def __new__(cls, *args):
> ...   return object()
> ...
> >>> try:
> ...  raise Foo
> ... except Exception as e:
> ...  print (e)
> ...
> <object object at 0x100634280>
>
> (I know this has nothing to do with the topic being debated but for some
> reason this code tickles me. Plus it used to segfault Python 3...)
>


Ooh... this one segfaults Python 3.2 - I wonder if that's been fixed yet.

>>> class Foo(Exception):
...  def __new__(cls, *args):
...   return 'string exception'
...
>>> try:
...  raise Foo
... except Exception as e:
...  print (e)
...
Segmentation fault: 11

All the best,

Michael Foord


>
> All the best,
>
>
> Michael
>
>
>
>>
>>
>>
>>> On 07Nov2011 11:35, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> | It is impossible to use singletons for exception instances now that the
>>> | traceback is stored on them.
>>>
>>> Ah. I had somehow thought the exception itself and the traceback were
>>> distinct items, presented in a tuple.
>>>
>>> On 07Nov2011 21:15, Steven D'Aprano <steve at pearwood.info> wrote:
>>> | Are you asking about what it should be, or what it is?
>>>
>>> The former.
>>>
>>> | Either way:
>>> | >>> a = StopIteration('spam')
>>> | >>> b = StopIteration('ham')
>>> | >>> a is b
>>> | False
>>>
>>> Since my question was about the proposed new behaviour when just a type
>>> was raised, the above test wouldn't educate me. Though it does address
>>> the
>>> behaviour of the type instantation in general.
>>>
>>> Cheers,
>>> --
>>> Cameron Simpson <cs at zip.com.au> DoD#743
>>> http://www.cskk.ezoshosting.com/cs/
>>>
>>> Carpe Datum     - John Sloan <jsloan at ncar.ucar.edu>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> http://mail.python.org/mailman/listinfo/python-ideas
>>>
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>>
>
>
> --
>
> http://www.voidspace.org.uk/
>
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
>
> May you share freely, never taking more than you give.
> -- the sqlite blessing http://www.sqlite.org/different.html
>
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111107/ac0e12d7/attachment.html>

From fuzzyman at gmail.com  Tue Nov  8 00:20:38 2011
From: fuzzyman at gmail.com (Michael Foord)
Date: Mon, 7 Nov 2011 23:20:38 +0000
Subject: [Python-ideas] raising an exception type doesn't instantiate it
 until it gets caught
In-Reply-To: <CAKCKLWzZ6qhE7iv1CaLhJQ-u5ObTDh7vRyH7z_d8=T6OE4KTGw@mail.gmail.com>
References: <4EB7AFC5.3040509@pearwood.info>
	<20111107113544.6a413544@pitrou.net>
	<CADiSq7eM8OOerLH4PN+hNEKk9i5dw94tdzugkC-qYgC2Lx3RCA@mail.gmail.com>
	<20111107200529.GA10386@cskk.homeip.net>
	<CAGE7PN+3ez=Ckf6X4=-GcEe6yxp9ZdfzpByUV0ApXL9BCfAKug@mail.gmail.com>
	<CAKCKLWxq1xcfxXvFT_31Li5MgfMmVA82KDM22xa+jSx=iKOR7A@mail.gmail.com>
	<CAKCKLWzZ6qhE7iv1CaLhJQ-u5ObTDh7vRyH7z_d8=T6OE4KTGw@mail.gmail.com>
Message-ID: <CAKCKLWzVyEKfuzy+eQ3OvjdV651GZhbA-VVFpgWz-faFNJ0rsg@mail.gmail.com>

On 7 November 2011 23:06, Michael Foord <fuzzyman at gmail.com> wrote:

>
>
> On 7 November 2011 23:05, Michael Foord <fuzzyman at gmail.com> wrote:
>
>>
>>
>> On 7 November 2011 21:43, Gregory P. Smith <greg at krypto.org> wrote:
>>
>>>
>>>
>>> On Mon, Nov 7, 2011 at 12:05 PM, Cameron Simpson <cs at zip.com.au> wrote:
>>>
>>>> I wrote, naively:
>>>> | > I presume StopIteration would get instantiated to a singleton, like
>>>> | > NoneType to None? Just asking.
>>>>
>>>> On 07Nov2011 22:01, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>> | Even without the traceback issue Antoine mentioned, it's already the
>>>> | case that StopIteration isn't a singleton in 2.x. Various pieces of
>>>> | code (e.g. contextlib.contextmanager) rely on being able to tell
>>>> | whether they're getting a specific StopIteration instance back or a
>>>> | new one.
>>>>
>>>> Interesting.
>>>>
>>>> Off topic digression:
>>>>
>>>> I've been slightly uncomfortable about exceptions as control flow for a
>>>> while, basicly when writing code like this:
>>>>
>>>>  try:
>>>>    x = G.next()
>>>>  except StopIteration:
>>>>    # G is empty!
>>>>
>>>> in that I don't entirely know that the StopIteration came from G of from
>>>> some buggy code deeper inside G that let a StopIteration out, eg by
>>>> mangling a try/except like the above. In most circumstances with other
>>>> exceptions, while you might _expect_ the exception to come from the
>>>> source you expect you don't care so much because it will indicate
>>>> failure of the operation anyway. Report or die, you don't proceed as if
>>>> the op was good. But with StopIteration one is reading "G is empty"
>>>> directly into the situation and acting as though it is normal (exit the
>>>> event loop or whatever it may imply).
>>>>
>>>
>>> Agreed.  Use of exceptions for this in the language feels like it was a
>>> convenient way to do it but as the conditions aren't really *exceptional
>>> * at all it'd be nice if there were a lighter weight mechanism that
>>> could skip the unneeded parts of the exception raising and handling
>>> mechanism for the implementation.  We don't need the traceback to be stored
>>> in these situations.
>>>
>>> This existing logic to instantiate and associate the traceback with it
>>> only if caught is one way to implement doing exactly that. Any other ideas?
>>>
>>> Hackish things like a class attribute on classes being raised as an
>>> exception, or a LightweightException class being part of its class
>>> heirarchy used to signify if that exception should take the full path or
>>> the fast path come to mind but could be considered equally surprising.
>>>
>>> I'm not sure any of this is worth it but it would simplify the eval
>>> loop.  We're speaking implementation details of CPython here, not an actual
>>> change to the language itself. (*)
>>>
>>> -gps
>>>
>>> (*) Please beat anybody who writes code that depends on this somewhat
>>> odd exception instantiation timing behavior side effect over the head with
>>> a frozen herring.
>>>
>>
>>
>> Having the interpreter instantiate the exception for you allows you to do
>> wonderful things like this:
>>
>> >>> class Foo(Exception):
>> ...  def __new__(cls, *args):
>> ...   return object()
>> ...
>> >>> try:
>> ...  raise Foo
>> ... except Exception as e:
>> ...  print (e)
>> ...
>> <object object at 0x100634280>
>>
>> (I know this has nothing to do with the topic being debated but for some
>> reason this code tickles me. Plus it used to segfault Python 3...)
>>
>
>
> Ooh... this one segfaults Python 3.2 - I wonder if that's been fixed yet.
>
> >>> class Foo(Exception):
> ...  def __new__(cls, *args):
> ...   return 'string exception'
> ...
> >>> try:
> ...  raise Foo
> ... except Exception as e:
> ...  print (e)
> ...
> Segmentation fault: 11
>


Yeah, fixed on 3.2 branch and trunk. Sorry for all the noise. I finally
managed to get Python head to compile on OS X Lion, yay!

Michael


>
> All the best,
>
> Michael Foord
>
>
>>
>> All the best,
>>
>>
>> Michael
>>
>>
>>
>>>
>>>
>>>
>>>> On 07Nov2011 11:35, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>> | It is impossible to use singletons for exception instances now that
>>>> the
>>>> | traceback is stored on them.
>>>>
>>>> Ah. I had somehow thought the exception itself and the traceback were
>>>> distinct items, presented in a tuple.
>>>>
>>>> On 07Nov2011 21:15, Steven D'Aprano <steve at pearwood.info> wrote:
>>>> | Are you asking about what it should be, or what it is?
>>>>
>>>> The former.
>>>>
>>>> | Either way:
>>>> | >>> a = StopIteration('spam')
>>>> | >>> b = StopIteration('ham')
>>>> | >>> a is b
>>>> | False
>>>>
>>>> Since my question was about the proposed new behaviour when just a type
>>>> was raised, the above test wouldn't educate me. Though it does address
>>>> the
>>>> behaviour of the type instantation in general.
>>>>
>>>> Cheers,
>>>> --
>>>> Cameron Simpson <cs at zip.com.au> DoD#743
>>>> http://www.cskk.ezoshosting.com/cs/
>>>>
>>>> Carpe Datum     - John Sloan <jsloan at ncar.ucar.edu>
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> http://mail.python.org/mailman/listinfo/python-ideas
>>>>
>>>
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> http://mail.python.org/mailman/listinfo/python-ideas
>>>
>>>
>>
>>
>> --
>>
>> http://www.voidspace.org.uk/
>>
>> May you do good and not evil
>>
>> May you find forgiveness for yourself and forgive others
>>
>> May you share freely, never taking more than you give.
>> -- the sqlite blessing http://www.sqlite.org/different.html
>>
>>
>>
>
>
> --
>
> http://www.voidspace.org.uk/
>
> May you do good and not evil
> May you find forgiveness for yourself and forgive others
>
> May you share freely, never taking more than you give.
> -- the sqlite blessing http://www.sqlite.org/different.html
>
>
>


-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111107/bf70ae76/attachment.html>

From ncoghlan at gmail.com  Tue Nov  8 02:16:42 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 8 Nov 2011 11:16:42 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111107103424.4c2e2ef1@resist.wooz.org>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
Message-ID: <CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>

On Tue, Nov 8, 2011 at 1:34 AM, Barry Warsaw <barry at python.org> wrote:
> Nice PEP, and +1 for the concept (with Guido's preferred format). ?However,
> "qname" is pretty obscure and I only guessed what the "q" stood for by reading
> the title of the PEP.
>
> It seems to me that this attribute represents the dotted path from module
> globals to the object. ?You have to be careful not to confuse this with a file
> system path, so something like __dotted_name__, __dotted_path__, or
> __full_name__ perhaps. ?I don't much care, but I do think cryptic
> abbreviations should be avoided.

For both this PEP and PEP 395 (module aliasing), we need a new term
for a name that is "usually the same as __name__, but different in
some cases where __name__ lacks necessary detail".

"qualified name" meets that criteria in both cases by covering all
situations where __name__ underspecifies things. For classes and
functions, __name__ leaves out the nesting information, so you can't
reliable locate the definition based solely on __module__ and
__name__. In the case of modules, we occasionally alter __name__ for
other purposes (such as indicating the main module or to ensure
objects get serialised with the correct module information when the
file layout doesn't match the nominal module layout). The new
qualified names will address all those cases without causing backwards
compatibility issues for existing uses of __name__. Hence, -1 on terms
other than "qualified", because they don't fit the module aliasing use
case (which involves name that are neither scoped, nor necessarily
dotted) and would require me to come up with yet another term for what
is essentially the same concept.

The question then is whether or not to introduce "qname" as an
officially documented abbreviation for "qualified name". And to that I
say, "Hell, yes".

1. "qualifiedname" and "qualified_name" is just too damn long. I want
an abbreviation.
2. "qualname" is too hard to say
3. People are going to come up with an abbreviation anyway, so we may
as well make it an official one so we get to benefit as well

Being able to talk about "names and qnames" without ambiguity will
make all of the affected code much easier to update and discuss than
if we insist on using the full "qualified name" term every time.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From wuwei23 at gmail.com  Tue Nov  8 03:44:12 2011
From: wuwei23 at gmail.com (alex23)
Date: Mon, 7 Nov 2011 18:44:12 -0800 (PST)
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
Message-ID: <60c446b4-f8e3-4f72-a597-7eef7a1dc20e@z15g2000prn.googlegroups.com>

On Nov 8, 11:16?am, Nick Coghlan <ncogh... at gmail.com> wrote:
> For both this PEP and PEP 395 (module aliasing), we need a new term
> for a name that is "usually the same as __name__, but different in
> some cases where __name__ lacks necessary detail".

__aka__ ? :)


From ron3200 at gmail.com  Tue Nov  8 06:30:53 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 07 Nov 2011 23:30:53 -0600
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy> <4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy> <4EB5D043.8050106@canterbury.ac.nz>
	<1320564031.30197.100.camel@Gutsy>
	<CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>
Message-ID: <1320730253.22449.39.camel@Gutsy>

On Sun, 2011-11-06 at 22:21 +1000, Nick Coghlan wrote:
> On Sun, Nov 6, 2011 at 5:20 PM, Ron Adam <ron3200 at gmail.com> wrote:
> > What is needed is a different path out (and back) to the coroutine that
> > doesn't interfere with the standard yield behavior.  Or as you describe
> > it here.
> 
> Did my post of the thread+queue based implementation of a coroutine
> API design concept not go through? It demonstrated exactly the need
> for a separate I/O channel independent of the call/return and
> next/send/throw/yield channels.

I'm looking at it now.  BTW the resume method is missing the 'return
data' at the end.

Yes, I think it will be useful as a way to figure out the best API.

It looks like the API is almost the same as the generator interface,
with different spellings.

What do you think about an inverted generator API?

You almost have that, but the throw is in the thread object, and not
reachable from the thread.

Generator API.

   Outside generator   <-->   Inside generator

      next()                     yield
      .send()
      .throw()


Inverted generator API

   Outside cothread    <-->   Inside cothread

      .resume()                  suspend()
                                 throw() 

Where resume works like yield, (yield to cothread), and suspend() works
like .send().  Throw() raises an exception at the resume() call,
like .throw() raises an exception at the yield in a generator.

Cheers,
   Ron




From ncoghlan at gmail.com  Tue Nov  8 06:46:31 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 8 Nov 2011 15:46:31 +1000
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
	protocol
In-Reply-To: <1320730253.22449.39.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy>
	<4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy>
	<4EB5D043.8050106@canterbury.ac.nz>
	<1320564031.30197.100.camel@Gutsy>
	<CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>
	<1320730253.22449.39.camel@Gutsy>
Message-ID: <CADiSq7e7JUegZuRF6dghJZGYqXx8HC1wEBXCCiR7qjthJuZECw@mail.gmail.com>

On Tue, Nov 8, 2011 at 3:30 PM, Ron Adam <ron3200 at gmail.com> wrote:
> Generator API.
>
> ? Outside generator ? <--> ? Inside generator
>
> ? ? ?next() ? ? ? ? ? ? ? ? ? ? yield
> ? ? ?.send()
> ? ? ?.throw()
>
>
> Inverted generator API
>
> ? Outside cothread ? ?<--> ? Inside cothread
>
> ? ? ?.resume() ? ? ? ? ? ? ? ? ?suspend()
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? throw()
>
> Where resume works like yield, (yield to cothread), and suspend() works
> like .send(). ?Throw() raises an exception at the resume() call,
> like .throw() raises an exception at the yield in a generator.

No, that doesn't make any sense. When the coroutine throws an
exception internally it's done - we don't *want* to preserve the stack
any more, because something broke and we won't be resuming it.
Instead, we let the exception bubble up the stack and if nothing
handles it, we pass it back to the thread that called resume().

The reason we need an explicit throw() is that the data request (or
whatever it was we suspended to wait for) might fail - in that case,
the thread calling resume() needs to be able to indicate this to the
cothread by resuming with an exception.

The flow control parallels are like this:

  Inside the generator/cothread:
    yield -> cothread.suspend() # Wait to be resumed
    return -> return # Finish normally
    raise -> raise  # Bail out with an error

  Outside the generator/cothread
    send() -> resume() # Resume execution normally (optionally providing data)
    throw() -> throw() # Resume execution with an exception

Don't worry about next() in this context, since it's just an alternate
spelling for send(None).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ron3200 at gmail.com  Tue Nov  8 09:24:49 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Tue, 08 Nov 2011 02:24:49 -0600
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <CADiSq7e7JUegZuRF6dghJZGYqXx8HC1wEBXCCiR7qjthJuZECw@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy> <4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy> <4EB5D043.8050106@canterbury.ac.nz>
	<1320564031.30197.100.camel@Gutsy>
	<CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>
	<1320730253.22449.39.camel@Gutsy>
	<CADiSq7e7JUegZuRF6dghJZGYqXx8HC1wEBXCCiR7qjthJuZECw@mail.gmail.com>
Message-ID: <1320740689.25015.101.camel@Gutsy>

On Tue, 2011-11-08 at 15:46 +1000, Nick Coghlan wrote:
> On Tue, Nov 8, 2011 at 3:30 PM, Ron Adam <ron3200 at gmail.com> wrote:
> > Generator API.
> >
> >   Outside generator   <-->   Inside generator
> >
> >      next()                     yield
> >      .send()
> >      .throw()
> >
> >
> > Inverted generator API
> >
> >   Outside cothread    <-->   Inside cothread
> >
> >      .resume()                  suspend()
> >                                 throw()
> >
> > Where resume works like yield, (yield to cothread), and suspend() works
> > like .send().  Throw() raises an exception at the resume() call,
> > like .throw() raises an exception at the yield in a generator.
> 
> No, that doesn't make any sense.

Probably because I didn't explain it well enough.

> When the coroutine throws an
> exception internally it's done - we don't *want* to preserve the stack
> any more, because something broke and we won't be resuming it.

You mean throw as in a natural occurring exception rather than one
explicitly thrown. Different thing.

In the case of raise, (or throws due to an error.) true, but that's not
how throw() would work in an inverse generator API.  If we throw an
exception from the *inside*, it's not a coroutine error, it's re-raised
at the handler in the resume() call, not in the coroutine itself.  That
could work with generators as well.  Wish it did.

The reason that makes sense to do in coroutines is we most likely
already have a try except structure in the coroutine handler to catch
the exit and return status exceptions, so why not take advantage of that
and make it possibly for the coroutines to send out exceptions for other
things. with a throw() from inside the coroutine.  (and not unwind the
stack like a raise would.)

For the same reason you throw() an exception into a generator, you could
throw an exception out of a coroutine.  You don't *need* to do that with
either of them.  The alternative is to pass through the normal data
channel and parse, test, and/or filter it out once it gets to the other
side.  An try-except around a data input can be very efficient at doing
that with a lot less work.

> Instead, we let the exception bubble up the stack and if nothing
> handles it, we pass it back to the thread that called resume().

Right, and we can't resume from there in that case.

> The reason we need an explicit throw() is that the data request (or
> whatever it was we suspended to wait for) might fail - in that case,
> the thread calling resume() needs to be able to indicate this to the
> cothread by resuming with an exception.

Yes and no... Yes, that works, and no because it could work just as well
the other way around.

If generators had a throw keyword...

def adder(count):
   exc = None
   total = n = 0
   while n < count:
      try:
          if exc is None:
             x = yield 
          else:
             x = throw exc    # reraise exc in .send(), not here.
                              # suspends here, and waits for new x.
          total += x          # <-- error may be here.
          exc = None
          n += 1
      except Exception as e:
          exc = e
   yield total

In this case, the exception wouldn't bubble out, but be reraised at
the .send() where it can be handled.

I think that generators not being able to handle these types of things
gracefully is a reason not to use them with coroutines. A program based
on generator coroutines that you get an unexpected exception from needs
to be completely restarted. That makes sense for small iterators, but
not for larger programs.


> The flow control parallels are like this:
> 
>   Inside the generator/cothread:
>     yield -> cothread.suspend() # Wait to be resumed
>     return -> return # Finish normally
>     raise -> raise  # Bail out with an error
> 
>   Outside the generator/cothread
>     send() -> resume() # Resume execution normally (optionally providing data)
>     throw() -> throw() # Resume execution with an exception
> 
> Don't worry about next() in this context, since it's just an alternate
> spelling for send(None).

Yes, about the next.  And yes, this is the design in your cothread
module example. :)

Cheers,
   Ron




From ncoghlan at gmail.com  Tue Nov  8 11:43:51 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 8 Nov 2011 20:43:51 +1000
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
	protocol
In-Reply-To: <1320740689.25015.101.camel@Gutsy>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz>
	<1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz>
	<1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy>
	<4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy>
	<4EB5D043.8050106@canterbury.ac.nz>
	<1320564031.30197.100.camel@Gutsy>
	<CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>
	<1320730253.22449.39.camel@Gutsy>
	<CADiSq7e7JUegZuRF6dghJZGYqXx8HC1wEBXCCiR7qjthJuZECw@mail.gmail.com>
	<1320740689.25015.101.camel@Gutsy>
Message-ID: <CADiSq7d2ra-3Aiq0qMUy+EsViaG7_-PT=7SyrD1hA5966BPjkA@mail.gmail.com>

On Tue, Nov 8, 2011 at 6:24 PM, Ron Adam <ron3200 at gmail.com> wrote:
> On Tue, 2011-11-08 at 15:46 +1000, Nick Coghlan wrote:
>> No, that doesn't make any sense.
>
> Probably because I didn't explain it well enough.

No, it doesn't make any sense because you're taking a symmetric
coroutine concept and attempting to apply it to an asymmetric
coroutine API design.

*If* this was a symmetric coroutine design (like greenlets), then
*yes* it would make sense to offer resume() and throw() as your
control flow APIs (there wouldn't be a suspend() at all in a symmetric
design, except perhaps as a convenience wrapper around invoking
resume() on the coroutine that called resume() or throw() on you).

With an asymmetric design, you only have 3 sensible options: terminate
naturally, terminate with an exception or suspend execution. Look at
the API design again - like the corresponding generator methods,
cothread.resume() and cothread.throw() both use "returned normally" to
indicate that the coroutine is suspended and still has more to do. If
they raise an exception, it means the coroutine is done, either
because it failed or because it finished normally (with the latter
case being distinguished by a specific exception type, just as it is
for generators). You're suggesting it would be reasonable to raise an
exception *and* still expect the coroutine to eventually be resumed
again. How is the caller of resume() or throw() meant to distinguish
that new case from the two terminating cases?

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From barry at python.org  Tue Nov  8 16:55:40 2011
From: barry at python.org (Barry Warsaw)
Date: Tue, 8 Nov 2011 10:55:40 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
Message-ID: <20111108105540.6ca5d579@resist.wooz.org>

In general, one problem with abbreviations is that they are more difficult for
non-native English speakers to understand and use.  I've often heard such
complaints from acquaintances for whom English is not their primary language.
The other problem is that while *you* know what the 'q' stands for because you
derived it from the underlying concept, someone who stumbles over it in the
opposite direction will not know what it means.  Maybe they'll search for it,
but otherwise, it'll just be a meaningless combination of characters.

Python has always valued readability over writing convenience, and I think
this is one of Guido's founding brilliant insights: code is read far more
often then it is written.  And yet, he managed to find elegant ways of
expressing code clearly without being overly verbose.

For these reasons, I strongly believe that this attribute should not be
abbreviated.

If the spelled out name is too long, find another one that conveys the same
information in fewer characters.  Several have been proposed and it's not hard
to find others.  E.g. __name_details__.

(I personally do not have a problem with underscores separating the words.
Again, it makes it more clear, where as mashwords can be difficult to scan.)

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111108/ac1a4070/attachment.pgp>

From matt at whoosh.ca  Tue Nov  8 16:57:45 2011
From: matt at whoosh.ca (Matt Chaput)
Date: Tue, 08 Nov 2011 10:57:45 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111108105540.6ca5d579@resist.wooz.org>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
Message-ID: <4EB95179.1030004@whoosh.ca>

On 08/11/2011 10:55 AM, Barry Warsaw wrote:
> If the spelled out name is too long, find another one that conveys the same
> information in fewer characters.  Several have been proposed and it's not hard
> to find others.  E.g. __name_details__.

Sorry I haven't been paying attention to this thread, but has 
__fullname__ been considered?

Cheers,

Matt



From solipsis at pitrou.net  Tue Nov  8 17:14:14 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 8 Nov 2011 17:14:14 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
Message-ID: <20111108171414.2f222781@pitrou.net>

On Tue, 8 Nov 2011 10:55:40 -0500
Barry Warsaw <barry at python.org> wrote:
> In general, one problem with abbreviations is that they are more difficult for
> non-native English speakers to understand and use.  I've often heard such
> complaints from acquaintances for whom English is not their primary language.
> The other problem is that while *you* know what the 'q' stands for because you
> derived it from the underlying concept, someone who stumbles over it in the
> opposite direction will not know what it means.  Maybe they'll search for it,
> but otherwise, it'll just be a meaningless combination of characters.
> 
> Python has always valued readability over writing convenience, and I think
> this is one of Guido's founding brilliant insights: code is read far more
> often then it is written.  And yet, he managed to find elegant ways of
> expressing code clearly without being overly verbose.
> 
> For these reasons, I strongly believe that this attribute should not be
> abbreviated.
> 
> If the spelled out name is too long, find another one that conveys the same
> information in fewer characters.  Several have been proposed and it's not hard
> to find others.  E.g. __name_details__.

If we go that way, I'd still prefer __qualname__ (but I'm fine with
__qname__ :-)).

Regards

Antoine.




From ron3200 at gmail.com  Tue Nov  8 17:56:13 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Tue, 08 Nov 2011 10:56:13 -0600
Subject: [Python-ideas] Cofunctions - Getting away from the iterator
 protocol
In-Reply-To: <CADiSq7d2ra-3Aiq0qMUy+EsViaG7_-PT=7SyrD1hA5966BPjkA@mail.gmail.com>
References: <4EA8BD66.6010807@canterbury.ac.nz>
	<CADiSq7dFcsiEnJwxkvcOb3aCPMuGg1Jbx2qe7okNUH6kZPN-4g@mail.gmail.com>
	<4EA9FED3.6050505@pearwood.info>
	<CADiSq7dZ18Hb3vnj_4KfQJKaSz5X5fU=QwraUstSE-gu8LYowg@mail.gmail.com>
	<4EADBEA7.9000608@canterbury.ac.nz>
	<CADiSq7d09EigTLZiSGvXy=HVOot92WGd1v0n2CLCcbuWLktMjQ@mail.gmail.com>
	<CAB4yi1Neiq8tdFOdQaHqCRFkOaz4z4hfzcW06ZQtyi-igwN0ZQ@mail.gmail.com>
	<CADiSq7emGhvcji8e48=TC=RZLQhMWb2-EV1ODGeKg-q2Rh6NDQ@mail.gmail.com>
	<CAB4yi1PdMh48XUvxPoVHCSXVHUsU28zSiejFQfrTv0_n1g8U3A@mail.gmail.com>
	<4EAE5F83.9040305@canterbury.ac.nz> <1320083850.5984.115.camel@Gutsy>
	<4EAF0DEE.1020500@canterbury.ac.nz> <1320106977.6637.50.camel@Gutsy>
	<CADiSq7fsLcM2mSaA8QdvNOW27-7OkkUkonAsajJF0cezLJt2oA@mail.gmail.com>
	<1320123506.9456.57.camel@Gutsy> <j8o3hk$afm$1@dough.gmane.org>
	<CADiSq7dHGDOT=_XU=j97ZCogFTRwJjaBLaGO1HmZrQBtt=D4Jw@mail.gmail.com>
	<1320250872.14285.65.camel@Gutsy> <4EB1F2BD.7070202@canterbury.ac.nz>
	<1320300930.10353.78.camel@Gutsy> <4EB5D043.8050106@canterbury.ac.nz>
	<1320564031.30197.100.camel@Gutsy>
	<CADiSq7eX9WkqH_e_O9XJ+SucKSH0ojR0P6KsjVAS-_WmBBQGnw@mail.gmail.com>
	<1320730253.22449.39.camel@Gutsy>
	<CADiSq7e7JUegZuRF6dghJZGYqXx8HC1wEBXCCiR7qjthJuZECw@mail.gmail.com>
	<1320740689.25015.101.camel@Gutsy>
	<CADiSq7d2ra-3Aiq0qMUy+EsViaG7_-PT=7SyrD1hA5966BPjkA@mail.gmail.com>
Message-ID: <1320771373.26276.60.camel@Gutsy>

On Tue, 2011-11-08 at 20:43 +1000, Nick Coghlan wrote:
> On Tue, Nov 8, 2011 at 6:24 PM, Ron Adam <ron3200 at gmail.com> wrote:
> > On Tue, 2011-11-08 at 15:46 +1000, Nick Coghlan wrote:
> >> No, that doesn't make any sense.
> >
> > Probably because I didn't explain it well enough.
> 
> No, it doesn't make any sense because you're taking a symmetric
> coroutine concept and attempting to apply it to an asymmetric
> coroutine API design.

I'm not really thinking that way. I'm just looking at what I want to do,
and what I'd like to have to do it nicely in python.

> *If* this was a symmetric coroutine design (like greenlets), then
> *yes* it would make sense to offer resume() and throw() as your
> control flow APIs

Yes.

> (there wouldn't be a suspend() at all in a symmetric
> design, except perhaps as a convenience wrapper around invoking
> resume() on the coroutine that called resume() or throw() on you).

You lost me here with suspend() not being needed except as a wrapper
around resume().  I think the reason you don't want to do that is
because you may want to .resume() multiple sub-coroutines, but you only
suspend() the one your in.

The way I see it, coroutines are a bit like little box's where there is
a boundary we can't cross in the way we normally would with exceptions.
Which is where the throw() method comes in.  A throw keyword would go
the other direction. (out instead of in.)  It wouldn't raise the
exception and let it bubble out as in the case of an unexpected error.
It would handle expected exception conditions, so continuing from there,
does make sense.

My feeling is the reason we don't already have that, has less to do with
asymmetric/symmetric design principles, and more to do with not having a
strong enough need (possibly until now) to justify a new keyword.

An API design that doesn't require new keywords wouldn't have that
limitation.  So we could offer both and let the programmer design his
coroutines as symmetrical or asymmetrical.


> With an asymmetric design, you only have 3 sensible options: terminate
> naturally, terminate with an exception or suspend execution. Look at
> the API design again - like the corresponding generator methods,
> cothread.resume() and cothread.throw() both use "returned normally" to
> indicate that the coroutine is suspended and still has more to do. If
> they raise an exception, it means the coroutine is done, either
> because it failed or because it finished normally (with the latter
> case being distinguished by a specific exception type, just as it is
> for generators). You're suggesting it would be reasonable to raise an
> exception *and* still expect the coroutine to eventually be resumed
> again. How is the caller of resume() or throw() meant to distinguish
> that new case from the two terminating cases?

Asymmetric, and symmetric wording is a description of how it's used.  We
don't necessarily need to limit python programmers to one or the other
exclusively.

Let me come up with some working examples.  I'm not too far from that
now.  Then we can discuss and experiment with it in a much more
practical way.

Cheers,
   Ron




From g.rodola at gmail.com  Tue Nov  8 19:20:45 2011
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Tue, 8 Nov 2011 19:20:45 +0100
Subject: [Python-ideas] shutil.tail(file, lines)
Message-ID: <CAFYqXL_WCb8gckm6qSjQPs+13F7QM7JHTWhLrkAAXe4JFD6L4A@mail.gmail.com>

This is something I need to do every once in a while and I think it
would be a good addition for shutil module.
Here:
http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-python-similar-to-tail
...is a nice implementation which appears to be a good compromise in
terms of speed and memory consumption (it reads the file in chunks, no
more than 1024 bytes per-read).

What do you think?

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/


From g.rodola at gmail.com  Tue Nov  8 19:27:31 2011
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Tue, 8 Nov 2011 19:27:31 +0100
Subject: [Python-ideas] shutil.tail(file, lines)
In-Reply-To: <CAFYqXL_WCb8gckm6qSjQPs+13F7QM7JHTWhLrkAAXe4JFD6L4A@mail.gmail.com>
References: <CAFYqXL_WCb8gckm6qSjQPs+13F7QM7JHTWhLrkAAXe4JFD6L4A@mail.gmail.com>
Message-ID: <CAFYqXL8BFBCCRQYe5weHwLTSgM_oW6fqSOkBSqEgjD0f0Ou0Mg@mail.gmail.com>

Edit: after a better look it seems data gets stored in memory
increasingly. We can see whether that is fixable someway though.

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/


2011/11/8 Giampaolo Rodol? <g.rodola at gmail.com>:
> This is something I need to do every once in a while and I think it
> would be a good addition for shutil module.
> Here:
> http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-python-similar-to-tail
> ...is a nice implementation which appears to be a good compromise in
> terms of speed and memory consumption (it reads the file in chunks, no
> more than 1024 bytes per-read).
>
> What do you think?
>
> --- Giampaolo
> http://code.google.com/p/pyftpdlib/
> http://code.google.com/p/psutil/
>


From jxo6948 at rit.edu  Tue Nov  8 19:30:16 2011
From: jxo6948 at rit.edu (John O'Connor)
Date: Tue, 8 Nov 2011 13:30:16 -0500
Subject: [Python-ideas] shutil.tail(file, lines)
In-Reply-To: <CAFYqXL8BFBCCRQYe5weHwLTSgM_oW6fqSOkBSqEgjD0f0Ou0Mg@mail.gmail.com>
References: <CAFYqXL_WCb8gckm6qSjQPs+13F7QM7JHTWhLrkAAXe4JFD6L4A@mail.gmail.com>
	<CAFYqXL8BFBCCRQYe5weHwLTSgM_oW6fqSOkBSqEgjD0f0Ou0Mg@mail.gmail.com>
Message-ID: <CABCbifV3YibVDi24x+LL-Y-FfKZOZym2MCCDjAg9rcQzROAa4g@mail.gmail.com>

There is also a more general solution: the reverse file iterator, started
here but never
finished: http://bugs.python.org/issue1677872

- John


On Tue, Nov 8, 2011 at 1:27 PM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:

> Edit: after a better look it seems data gets stored in memory
> increasingly. We can see whether that is fixable someway though.
>
> --- Giampaolo
> http://code.google.com/p/pyftpdlib/
> http://code.google.com/p/psutil/
>
>
> 2011/11/8 Giampaolo Rodol? <g.rodola at gmail.com>:
> > This is something I need to do every once in a while and I think it
> > would be a good addition for shutil module.
> > Here:
> >
> http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-python-similar-to-tail
> > ...is a nice implementation which appears to be a good compromise in
> > terms of speed and memory consumption (it reads the file in chunks, no
> > more than 1024 bytes per-read).
> >
> > What do you think?
> >
> > --- Giampaolo
> > http://code.google.com/p/pyftpdlib/
> > http://code.google.com/p/psutil/
> >
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111108/3c866a91/attachment.html>

From g.rodola at gmail.com  Tue Nov  8 19:43:28 2011
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Tue, 8 Nov 2011 19:43:28 +0100
Subject: [Python-ideas] shutil.tail(file, lines)
In-Reply-To: <CABCbifV3YibVDi24x+LL-Y-FfKZOZym2MCCDjAg9rcQzROAa4g@mail.gmail.com>
References: <CAFYqXL_WCb8gckm6qSjQPs+13F7QM7JHTWhLrkAAXe4JFD6L4A@mail.gmail.com>
	<CAFYqXL8BFBCCRQYe5weHwLTSgM_oW6fqSOkBSqEgjD0f0Ou0Mg@mail.gmail.com>
	<CABCbifV3YibVDi24x+LL-Y-FfKZOZym2MCCDjAg9rcQzROAa4g@mail.gmail.com>
Message-ID: <CAFYqXL__a9OZt9u2La98xFS9CHgeuhcth+qE9js3ZMXtUxGgig@mail.gmail.com>

2011/11/8 John O'Connor <jxo6948 at rit.edu>:
> There is also a more general solution: the reverse file iterator,?started
> here but never
> finished:?http://bugs.python.org/issue1677872
> - John

Oh, nice!
That would certainly be better.

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/


From anacrolix at gmail.com  Tue Nov  8 22:24:26 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Wed, 9 Nov 2011 08:24:26 +1100
Subject: [Python-ideas] shutil.tail(file, lines)
In-Reply-To: <CAFYqXL__a9OZt9u2La98xFS9CHgeuhcth+qE9js3ZMXtUxGgig@mail.gmail.com>
References: <CAFYqXL_WCb8gckm6qSjQPs+13F7QM7JHTWhLrkAAXe4JFD6L4A@mail.gmail.com>
	<CAFYqXL8BFBCCRQYe5weHwLTSgM_oW6fqSOkBSqEgjD0f0Ou0Mg@mail.gmail.com>
	<CABCbifV3YibVDi24x+LL-Y-FfKZOZym2MCCDjAg9rcQzROAa4g@mail.gmail.com>
	<CAFYqXL__a9OZt9u2La98xFS9CHgeuhcth+qE9js3ZMXtUxGgig@mail.gmail.com>
Message-ID: <CAB4yi1MDJgcuUnMdADoBmL2J0o=++v=X-WnHXvaYQcamMPfTbQ@mail.gmail.com>

+1 This is an excellent idea. I reimplement this very often.

On Wed, Nov 9, 2011 at 5:43 AM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
> 2011/11/8 John O'Connor <jxo6948 at rit.edu>:
>> There is also a more general solution: the reverse file iterator,?started
>> here but never
>> finished:?http://bugs.python.org/issue1677872
>> - John
>
> Oh, nice!
> That would certainly be better.
>
> --- Giampaolo
> http://code.google.com/p/pyftpdlib/
> http://code.google.com/p/psutil/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From anacrolix at gmail.com  Tue Nov  8 22:26:13 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Wed, 9 Nov 2011 08:26:13 +1100
Subject: [Python-ideas] shutil.tail(file, lines)
In-Reply-To: <CAB4yi1MDJgcuUnMdADoBmL2J0o=++v=X-WnHXvaYQcamMPfTbQ@mail.gmail.com>
References: <CAFYqXL_WCb8gckm6qSjQPs+13F7QM7JHTWhLrkAAXe4JFD6L4A@mail.gmail.com>
	<CAFYqXL8BFBCCRQYe5weHwLTSgM_oW6fqSOkBSqEgjD0f0Ou0Mg@mail.gmail.com>
	<CABCbifV3YibVDi24x+LL-Y-FfKZOZym2MCCDjAg9rcQzROAa4g@mail.gmail.com>
	<CAFYqXL__a9OZt9u2La98xFS9CHgeuhcth+qE9js3ZMXtUxGgig@mail.gmail.com>
	<CAB4yi1MDJgcuUnMdADoBmL2J0o=++v=X-WnHXvaYQcamMPfTbQ@mail.gmail.com>
Message-ID: <CAB4yi1Mp2P_qaYZ6BHG0S67n6ZGMTSU9fHQoP+3gtXe+rQ_=Eg@mail.gmail.com>

Addendum: I implement the form that continues tailing, waiting for new
data. I think the naive form "last X lines" is a little trivial to put
in stdlib.

On Wed, Nov 9, 2011 at 8:24 AM, Matt Joiner <anacrolix at gmail.com> wrote:
> +1 This is an excellent idea. I reimplement this very often.
>
> On Wed, Nov 9, 2011 at 5:43 AM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
>> 2011/11/8 John O'Connor <jxo6948 at rit.edu>:
>>> There is also a more general solution: the reverse file iterator,?started
>>> here but never
>>> finished:?http://bugs.python.org/issue1677872
>>> - John
>>
>> Oh, nice!
>> That would certainly be better.
>>
>> --- Giampaolo
>> http://code.google.com/p/pyftpdlib/
>> http://code.google.com/p/psutil/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>


From ncoghlan at gmail.com  Tue Nov  8 23:09:57 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 9 Nov 2011 08:09:57 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <4EB95179.1030004@whoosh.ca>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
	<4EB95179.1030004@whoosh.ca>
Message-ID: <CADiSq7c0nJPEK+D1rVsMixwZfmSmCzF02fp3rTievFDLLd16NQ@mail.gmail.com>

Yeah, although I forget exactly when. The problem with it is that it is
wrong (since the full name would include the module info)

--
Nick Coghlan (via Gmail on Android, so likely to be more terse than usual)
On Nov 9, 2011 2:00 AM, "Matt Chaput" <matt at whoosh.ca> wrote:

> On 08/11/2011 10:55 AM, Barry Warsaw wrote:
>
>> If the spelled out name is too long, find another one that conveys the
>> same
>> information in fewer characters.  Several have been proposed and it's not
>> hard
>> to find others.  E.g. __name_details__.
>>
>
> Sorry I haven't been paying attention to this thread, but has __fullname__
> been considered?
>
> Cheers,
>
> Matt
>
> ______________________________**_________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111109/5492736e/attachment.html>

From solipsis at pitrou.net  Tue Nov  8 23:14:07 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 8 Nov 2011 23:14:07 +0100
Subject: [Python-ideas] shutil.tail(file, lines)
References: <CAFYqXL_WCb8gckm6qSjQPs+13F7QM7JHTWhLrkAAXe4JFD6L4A@mail.gmail.com>
Message-ID: <20111108231407.6ab1d503@pitrou.net>

On Tue, 8 Nov 2011 19:20:45 +0100
Giampaolo Rodol? <g.rodola at gmail.com>
wrote:
> This is something I need to do every once in a while and I think it
> would be a good addition for shutil module.
> Here:
> http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-python-similar-to-tail
> ...is a nice implementation which appears to be a good compromise in
> terms of speed and memory consumption (it reads the file in chunks, no
> more than 1024 bytes per-read).

Well, is it supposed to be a text file or a binary file?
With a binary file the above approach is ok (you can use an adaptative
average line length if you want to be smarter).
With a text file and a generic encoding (possible weird or nasty) you
have no other solution than reading the file from the start.

Regards

Antoine.




From casevh at gmail.com  Wed Nov  9 07:17:26 2011
From: casevh at gmail.com (Case Van Horsen)
Date: Tue, 8 Nov 2011 22:17:26 -0800
Subject: [Python-ideas] Add additional special method lookups to math module
Message-ID: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>

Currently the functions round(), math.ceil(), math.floor(), and
math.trunc() all check for the existence of a special method
(__round__, __ceil__, __floor__, and __trunc__). Would it be possible
to enhance the math and cmath modules to check for the existence of a
special method for (almost) functions? For example, math.sin(obj)
would first check for obj.__sin__.

Rationale

I'm in the final stages of adding support for the MPFR
(multiple-precision floating point) and MPC (multiple-precision
complex) libraries to the next-generation of gmpy, currently known as
gmpy2. If the special method checks are added to the math and cmath
modules, then the new mpfr() and mpc() types can easily substitute for
the existing float/complex types in code that uses the math or cmath
module.

Thoughts?


From jimjjewett at gmail.com  Wed Nov  9 20:32:09 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 9 Nov 2011 14:32:09 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111108105540.6ca5d579@resist.wooz.org>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
Message-ID: <CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>

On 11/8/11, Barry Warsaw <barry at python.org> wrote:
> ... one problem with abbreviations is that they are more difficult for
> non-native English speakers to understand and use.

qname seems to be well established for XML.  Do you fear that the q
doesn't look enough like a prefix, and they won't recognize it as a
type of name, or only that they won't know what makes this type of
name special?

> Python has always valued readability over writing convenience, and I think
> this is one of Guido's founding brilliant insights: code is read far more
> often then it is written.  And yet, he managed to find elegant ways of
> expressing code clearly without being overly verbose.

Frankly, I wouldn't know precisely what a "qualified name" is, and I
can only guess based on my painful experience with other systems -- of
which XML is by far the least ugly.  I'm not sure a standard
abbreviation makes things any worse.  ("Fully Qualified Name" isn't as
bad, but clearly runs afoul of succinctness.)

-jJ


From jimjjewett at gmail.com  Wed Nov  9 20:43:41 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 9 Nov 2011 14:43:41 -0500
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
Message-ID: <CA+OGgf557ihXUVOGS03mm+C_==VwW8LYa4nTmoA7EW8ru3U9ig@mail.gmail.com>

I like the idea, but there is a question of namespace size,
particularly if there may be other meanings for some of the names.

I would therefore prefer __math_sin__, __math_arctan__, etc.  (Does it
need to be even more specific than math?)

-jJ

On 11/9/11, Case Van Horsen <casevh at gmail.com> wrote:
> Currently the functions round(), math.ceil(), math.floor(), and
> math.trunc() all check for the existence of a special method
> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
> to enhance the math and cmath modules to check for the existence of a
> special method for (almost) functions? For example, math.sin(obj)
> would first check for obj.__sin__.
>
> Rationale
>
> I'm in the final stages of adding support for the MPFR
> (multiple-precision floating point) and MPC (multiple-precision
> complex) libraries to the next-generation of gmpy, currently known as
> gmpy2. If the special method checks are added to the math and cmath
> modules, then the new mpfr() and mpc() types can easily substitute for
> the existing float/complex types in code that uses the math or cmath
> module.
>
> Thoughts?
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From jeremy at jeremysanders.net  Wed Nov  9 21:06:46 2011
From: jeremy at jeremysanders.net (Jeremy Sanders)
Date: Wed, 09 Nov 2011 20:06:46 +0000
Subject: [Python-ideas] Add additional special method lookups to math
	module
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
Message-ID: <j9emgm$hvg$1@dough.gmane.org>

Case Van Horsen wrote:

> Currently the functions round(), math.ceil(), math.floor(), and
> math.trunc() all check for the existence of a special method
> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
> to enhance the math and cmath modules to check for the existence of a
> special method for (almost) functions? For example, math.sin(obj)
> would first check for obj.__sin__.

Wouldn't duct typing the functions in math and cmath be equivalent and 
wouldn't slow down normal users?

Howver, I'm not convinced that it's very nice changing the behaviour of 
built-in modules depending on the presence another module anyway, I'd just 
supply another module with the same api - people could always mess around 
with the import path if they really want to override the default.

import math

def mysin(x):
   xxxx
math.sin = mysin

Jeremy



From masklinn at masklinn.net  Wed Nov  9 21:15:25 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 9 Nov 2011 21:15:25 +0100
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <j9emgm$hvg$1@dough.gmane.org>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
Message-ID: <F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>

On 2011-11-09, at 21:06 , Jeremy Sanders wrote:
> Case Van Horsen wrote:
>> Currently the functions round(), math.ceil(), math.floor(), and
>> math.trunc() all check for the existence of a special method
>> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
>> to enhance the math and cmath modules to check for the existence of a
>> special method for (almost) functions? For example, math.sin(obj)
>> would first check for obj.__sin__.
> 
> Wouldn't duct typing the functions in math and cmath be equivalent and 
> wouldn't slow down normal users?
> 
> Howver, I'm not convinced that it's very nice changing the behaviour of 
> built-in modules depending on the presence another module anyway, I'd just 
> supply another module with the same api - people could always mess around 
> with the import path if they really want to override the default.
> 
> import math
> 
> def mysin(x):
>   xxxx
> math.sin = mysin
> 
> Jeremy
> 
This would mean custom numerical types wouldn't be drop-in compatible with existing numerical *code*, which I am under the impression is what Case Van Horsen wants (and is a desirable outcome).

Furthermore, your second paragraph is not correct: Case does not propose "changing the behavior of built-in modules depending on the presence of another module", Case proposes adding *method hooks* to existing math and cmath functions. These would be protocols allowing custom numerical types to implement `math`/`cmath`'s operation in a sensible manner, as is already possible for four methods he mentions, as well as a number of other Python operations[0], including prominent numerical ones[1].

[0] http://docs.python.org/reference/datamodel.html
[1] http://docs.python.org/reference/datamodel.html#emulating-numeric-types

From jeremy at jeremysanders.net  Wed Nov  9 21:32:20 2011
From: jeremy at jeremysanders.net (Jeremy Sanders)
Date: Wed, 09 Nov 2011 20:32:20 +0000
Subject: [Python-ideas] Add additional special method lookups to math
	module
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
	<F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
Message-ID: <j9eo0l$u3r$1@dough.gmane.org>

Masklinn wrote:

> This would mean custom numerical types wouldn't be drop-in compatible with
> existing numerical *code*, which I am under the impression is what Case
> Van Horsen wants (and is a desirable outcome).

I think it looks too much like hidden magic to me. Explicit is better than 
implicit.

If I were doing this, I wouldn't change the current behaviour by default. 
I'd include a function which would change the built-in functions, if 
required.

Jeremy




From g.brandl at gmx.net  Wed Nov  9 21:35:23 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 09 Nov 2011 21:35:23 +0100
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
Message-ID: <j9eo65$vu8$1@dough.gmane.org>

Am 09.11.2011 07:17, schrieb Case Van Horsen:
> Currently the functions round(), math.ceil(), math.floor(), and
> math.trunc() all check for the existence of a special method
> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
> to enhance the math and cmath modules to check for the existence of a
> special method for (almost) functions? For example, math.sin(obj)
> would first check for obj.__sin__.
> 
> Rationale
> 
> I'm in the final stages of adding support for the MPFR
> (multiple-precision floating point) and MPC (multiple-precision
> complex) libraries to the next-generation of gmpy, currently known as
> gmpy2. If the special method checks are added to the math and cmath
> modules, then the new mpfr() and mpc() types can easily substitute for
> the existing float/complex types in code that uses the math or cmath
> module.
> 
> Thoughts?

Just a data point: numpy's ufuncs like sin() look for an attribute of
the same name ("sin") if you give them objects that aren't arrays or
Python/numpy numbers.

Georg



From masklinn at masklinn.net  Wed Nov  9 21:37:32 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 9 Nov 2011 21:37:32 +0100
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <j9eo0l$u3r$1@dough.gmane.org>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
	<F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
	<j9eo0l$u3r$1@dough.gmane.org>
Message-ID: <DAA27B99-430F-4845-A2C3-50AD7FB41BA0@masklinn.net>

On 2011-11-09, at 21:32 , Jeremy Sanders wrote:
> Masklinn wrote:
>> This would mean custom numerical types wouldn't be drop-in compatible with
>> existing numerical *code*, which I am under the impression is what Case
>> Van Horsen wants (and is a desirable outcome).
> 
> I think it looks too much like hidden magic to me. Explicit is better than 
> implicit.
> 
It does not look any more like hidden magic than overriding "__add__" does.

> If I were doing this, I wouldn't change the current behaviour by default. 
> I'd include a function which would change the built-in functions, if 
> required.
That makes no sense.


From arnodel at gmail.com  Wed Nov  9 21:47:17 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Wed, 9 Nov 2011 20:47:17 +0000
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
Message-ID: <CAJ6cK1Zbp2XgpAAVujfkfH4TSmdVV6zW2tzHB4MRpYtyX1tHaQ@mail.gmail.com>

On 9 November 2011 06:17, Case Van Horsen <casevh at gmail.com> wrote:
> Currently the functions round(), math.ceil(), math.floor(), and
> math.trunc() all check for the existence of a special method
> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
> to enhance the math and cmath modules to check for the existence of a
> special method for (almost) functions? For example, math.sin(obj)
> would first check for obj.__sin__.
>
> Rationale
>
> I'm in the final stages of adding support for the MPFR
> (multiple-precision floating point) and MPC (multiple-precision
> complex) libraries to the next-generation of gmpy, currently known as
> gmpy2. If the special method checks are added to the math and cmath
> modules, then the new mpfr() and mpc() types can easily substitute for
> the existing float/complex types in code that uses the math or cmath
> module.
>
> Thoughts?

I have been faced with a very similar situation recently.

I am adding Python scripting capability to a Dynamic Geometry
application called GeoGebra (www.geogebra.org).  GeoGebra has its own
numeric types.  I have wrapped them in Python classes so that all
arithmetic operations work correctly on them but it would be a big
improvement if the standard analytic functions in the math module
could work on them as well.

So this would be a welcome addition for me (although, as GeoGebra is a
Java application, I am using Jython, so I would have to wait a while
to see this coming my way :).

-- 
Arnaud


From p.f.moore at gmail.com  Wed Nov  9 21:51:20 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 9 Nov 2011 20:51:20 +0000
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
	<F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
Message-ID: <CACac1F-LxxY1mgS3sHkHDEVX_nwDusFxRJoe08b0757g9DJbXw@mail.gmail.com>

On 9 November 2011 20:15, Masklinn <masklinn at masklinn.net> wrote:
> This would mean custom numerical types wouldn't be drop-in compatible with existing numerical *code*,
> which I am under the impression is what Case Van Horsen wants (and is a desirable outcome).
>
> Furthermore, your second paragraph is not correct: Case does not propose "changing the behavior of
> built-in modules depending on the presence of another module", Case proposes adding *method hooks*
> to existing math and cmath functions. These would be protocols allowing custom numerical types to
> implement `math`/`cmath`'s operation in a sensible manner, as is already possible for four methods he
> mentions, as well as a number of other Python operations[0], including prominent numerical ones[1].

I like the idea of being able to extend the math functions to allow
user-defined numeric types to behave like built in ones. (In fact, if
this were possible, complex could use it and a large proportion of the
cmath module would be unneeded). However, I have to say that for me
this is a purely theoretical issue - I've never had the need for the
functionality myself in real code, and I doubt I ever will. Given the
proliferation of special methods that would be needed, plus the
runtime overhead of extra checking, I think the cost is likely too
high.

It's also worth noting the preamble text in the math module
documentation - the functions there deliberately only handle floats,
and are thin wrappers over the C standard library. To get any sort of
generic behaviour, users need to use cmath, by design. So as stated
the proposal is unlikely to get much support.

Some alternative suggestions:

1. As already noted, you could include a gmpmath module with the
relevant functions, and users could monkeypatch the math module if
they wanted to do so.

2. To make such monkeypatching easier, publish a module that exposes a
context manager to do the job:
       with monkeypatch(math, 'sin', gmpmath.sin):
           your code here...

3. Write a genericmath module that provides the "generic" versions
you're proposing, and see how much use it gets - if it's sufficiently
popular, you have a better case then for folding the functionality
into at least cmath.

4. As Georg mentioned, numpy's ufuncs have a feature like this, so
support that and people can use your types with numpy. That may be
enough (depending on your expected user base).

Overall, I think the idea of being able to use user-defined types
interchangeably with built-in ones is a good one, but it's not
something Python goes out of its way to support. If the idea of
generic functions had taken off, then this is an area where they would
have fit nicely - but once again, there wasn't much enthusiasm from
the core developers for addressing the types of problem they were
aimed at.

Paul.


From jeremy at jeremysanders.net  Wed Nov  9 22:04:40 2011
From: jeremy at jeremysanders.net (Jeremy Sanders)
Date: Wed, 09 Nov 2011 21:04:40 +0000
Subject: [Python-ideas] Add additional special method lookups to math
	module
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
Message-ID: <j9ept8$du2$1@dough.gmane.org>

Jeremy Sanders wrote:

> Wouldn't duct typing the functions in math and cmath be equivalent and
> wouldn't slow down normal users?

s/duct typing/monkey patching/ - I think I was thinking of animals and duck 
came into my mind rather than monkey.

Jeremy




From jeremy at jeremysanders.net  Wed Nov  9 22:10:19 2011
From: jeremy at jeremysanders.net (Jeremy Sanders)
Date: Wed, 09 Nov 2011 21:10:19 +0000
Subject: [Python-ideas] Add additional special method lookups to math
	module
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
	<F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
	<j9eo0l$u3r$1@dough.gmane.org>
	<DAA27B99-430F-4845-A2C3-50AD7FB41BA0@masklinn.net>
Message-ID: <j9eq7s$h0b$1@dough.gmane.org>

Masklinn wrote:

> That makes no sense.

I think it does. I could write code relying on the math.XXX failing for 
types other than the built in numeric types, or for particular exceptions 
for particular calls. Also, if you override a standard module as a quick way 
to get existing code working, you can get lots of existing code paths which 
might fail in interesting ways, especially if there weren't unit tests for 
these cases. It would make more sense if the possibility was documented in 
the math documentation.

If you go and change the behaviour of built-in functions just by importing a 
module it could lead to some weird bugs. I know monkey patching can do this 
currently, but doing it by default seems dangerous to me.

Jeremy




From masklinn at masklinn.net  Wed Nov  9 22:32:07 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 9 Nov 2011 22:32:07 +0100
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <j9eq7s$h0b$1@dough.gmane.org>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
	<F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
	<j9eo0l$u3r$1@dough.gmane.org>
	<DAA27B99-430F-4845-A2C3-50AD7FB41BA0@masklinn.net>
	<j9eq7s$h0b$1@dough.gmane.org>
Message-ID: <73289562-327E-4771-9D10-4D6ED251477E@masklinn.net>

On 2011-11-09, at 22:10 , Jeremy Sanders wrote:
> Masklinn wrote:
>> That makes no sense.
> I think it does.
No, you're objecting to a local, scoped and type-based customization of operations (something already common in Python) by asking for a global functional reimplementation in user code instead.

> I could write code relying on the math.XXX failing for 
> types other than the built in numeric types, or for particular exceptions 
> for particular calls.
I fail to see how that's an argument against local customization and for global reimplementation.

> Also, if you override a standard module as a quick way 
> to get existing code working, you can get lots of existing code paths which 
> might fail in interesting ways, especially if there weren't unit tests for 
> these cases. It would make more sense if the possibility was documented in 
> the math documentation.
> 
> If you go and change the behaviour of built-in functions just by importing a 
> module it could lead to some weird bugs. I know monkey patching can do this 
> currently, but doing it by default seems dangerous to me.
Nothing in these two paragraphs makes any sense in the context of the current discussion. *Nobody* (but you) argues for "changing the behavior of built-in functions just by importing a module", or for "overriding a standard module".

I think you have misunderstood what Case Van Horsen is suggesting.

From ncoghlan at gmail.com  Wed Nov  9 22:33:04 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 10 Nov 2011 07:33:04 +1000
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <CACac1F-LxxY1mgS3sHkHDEVX_nwDusFxRJoe08b0757g9DJbXw@mail.gmail.com>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
	<F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
	<CACac1F-LxxY1mgS3sHkHDEVX_nwDusFxRJoe08b0757g9DJbXw@mail.gmail.com>
Message-ID: <CADiSq7de5LQJ7v+Eh38pkU2EZ=a0d-yq39Z1sVCbvON+rBrG7Q@mail.gmail.com>

The big problem with generics in the past has been clearly articulating use
cases. A genericmath and updated pprint could help a lot on that front.

--
Nick Coghlan (via Gmail on Android, so likely to be more terse than usual)
On Nov 10, 2011 6:52 AM, "Paul Moore" <p.f.moore at gmail.com> wrote:

> On 9 November 2011 20:15, Masklinn <masklinn at masklinn.net> wrote:
> > This would mean custom numerical types wouldn't be drop-in compatible
> with existing numerical *code*,
> > which I am under the impression is what Case Van Horsen wants (and is a
> desirable outcome).
> >
> > Furthermore, your second paragraph is not correct: Case does not propose
> "changing the behavior of
> > built-in modules depending on the presence of another module", Case
> proposes adding *method hooks*
> > to existing math and cmath functions. These would be protocols allowing
> custom numerical types to
> > implement `math`/`cmath`'s operation in a sensible manner, as is already
> possible for four methods he
> > mentions, as well as a number of other Python operations[0], including
> prominent numerical ones[1].
>
> I like the idea of being able to extend the math functions to allow
> user-defined numeric types to behave like built in ones. (In fact, if
> this were possible, complex could use it and a large proportion of the
> cmath module would be unneeded). However, I have to say that for me
> this is a purely theoretical issue - I've never had the need for the
> functionality myself in real code, and I doubt I ever will. Given the
> proliferation of special methods that would be needed, plus the
> runtime overhead of extra checking, I think the cost is likely too
> high.
>
> It's also worth noting the preamble text in the math module
> documentation - the functions there deliberately only handle floats,
> and are thin wrappers over the C standard library. To get any sort of
> generic behaviour, users need to use cmath, by design. So as stated
> the proposal is unlikely to get much support.
>
> Some alternative suggestions:
>
> 1. As already noted, you could include a gmpmath module with the
> relevant functions, and users could monkeypatch the math module if
> they wanted to do so.
>
> 2. To make such monkeypatching easier, publish a module that exposes a
> context manager to do the job:
>       with monkeypatch(math, 'sin', gmpmath.sin):
>           your code here...
>
> 3. Write a genericmath module that provides the "generic" versions
> you're proposing, and see how much use it gets - if it's sufficiently
> popular, you have a better case then for folding the functionality
> into at least cmath.
>
> 4. As Georg mentioned, numpy's ufuncs have a feature like this, so
> support that and people can use your types with numpy. That may be
> enough (depending on your expected user base).
>
> Overall, I think the idea of being able to use user-defined types
> interchangeably with built-in ones is a good one, but it's not
> something Python goes out of its way to support. If the idea of
> generic functions had taken off, then this is an area where they would
> have fit nicely - but once again, there wasn't much enthusiasm from
> the core developers for addressing the types of problem they were
> aimed at.
>
> Paul.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111110/a847fa89/attachment.html>

From casevh at gmail.com  Wed Nov  9 22:42:26 2011
From: casevh at gmail.com (Case Van Horsen)
Date: Wed, 9 Nov 2011 13:42:26 -0800
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <CA+OGgf557ihXUVOGS03mm+C_==VwW8LYa4nTmoA7EW8ru3U9ig@mail.gmail.com>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<CA+OGgf557ihXUVOGS03mm+C_==VwW8LYa4nTmoA7EW8ru3U9ig@mail.gmail.com>
Message-ID: <CANerV6mQAzLZqW6Sane_3qYWvWJ3oHotK7Rb3UR4OGLXDhQ+JQ@mail.gmail.com>

On Wed, Nov 9, 2011 at 11:43 AM, Jim Jewett <jimjjewett at gmail.com> wrote:
> I like the idea, but there is a question of namespace size,
> particularly if there may be other meanings for some of the names.
>
> I would therefore prefer __math_sin__, __math_arctan__, etc. ?(Does it
> need to be even more specific than math?)
>
> -jJ
>
Good idea. Creating a naming convention for functions that return real
or complex values would be needed. math.sin(obj) would look for
obj.__math_sin__. cmath.sin(obj) would look for obj.__cmath_sin__.

gmpy2 supports integer, rational, real, and complex numbers. The
integer, rational, and real objects would need both the __math_XXX__
and __cmath_XXX__ methods. The complex object would only support the
__cmath_XXX__ methods.

There is ambiguity with functions that take more than one argument:
atan2, copysign, fmod, hypot, and pow. If only one of the arguments
supports the special method, then that special method should be used.
If both of the arguments supports the special method AND both
arguments are of the same type, then the special method should be
used.

But what if both arguments are different types? I think it should try
the special method associated with the first argument and if that
return NotImplemented, then it should try the special method
associated with the second argument.

If the special methods ever return NotImplemented, then the normal
math/cmath handling should make a final attempt.

casevh


From barry at python.org  Wed Nov  9 23:04:38 2011
From: barry at python.org (Barry Warsaw)
Date: Wed, 9 Nov 2011 17:04:38 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
 functions
In-Reply-To: <CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
	<CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>
Message-ID: <20111109170438.0626d40b@limelight.wooz.org>

On Nov 09, 2011, at 02:32 PM, Jim Jewett wrote:

>qname seems to be well established for XML.  Do you fear that the q
>doesn't look enough like a prefix, and they won't recognize it as a
>type of name, or only that they won't know what makes this type of
>name special?

I think it will be easy to misread __qname__ as __name__ and it won't be
obviously clear what the difference between them is.

>> Python has always valued readability over writing convenience, and I think
>> this is one of Guido's founding brilliant insights: code is read far more
>> often then it is written.  And yet, he managed to find elegant ways of
>> expressing code clearly without being overly verbose.
>
>Frankly, I wouldn't know precisely what a "qualified name" is, and I
>can only guess based on my painful experience with other systems -- of
>which XML is by far the least ugly.  I'm not sure a standard
>abbreviation makes things any worse.  ("Fully Qualified Name" isn't as
>bad, but clearly runs afoul of succinctness.)

Isn't that a problem with the basic terminology then?  If you don't know what
a "qualified name" is you probably won't know what a "qname" is, and you
definitely won't make that connection.  I think that's more reason to find the
right terminology and spell it out.

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111109/5cefdffc/attachment.pgp>

From casevh at gmail.com  Wed Nov  9 23:31:02 2011
From: casevh at gmail.com (Case Van Horsen)
Date: Wed, 9 Nov 2011 14:31:02 -0800
Subject: [Python-ideas] Add additional special method lookups to math
	module
Message-ID: <CANerV6keP2ZE4nEEpwNj28sKE7AX474NyJC9mMr=Mf8+A7E+TA@mail.gmail.com>

Paul Moore wrote:
> On 9 November 2011 20:15, Masklinn <masklinn at masklinn.net> wrote:
>> This would mean custom numerical types wouldn't be drop-in compatible with existing numerical *code*,
>> which I am under the impression is what Case Van Horsen wants (and is a desirable outcome).
>>
>> Furthermore, your second paragraph is not correct: Case does not propose "changing the behavior of
>> built-in modules depending on the presence of another module", Case proposes adding *method hooks*
>> to existing math and cmath functions. These would be protocols allowing custom numerical types to
>> implement `math`/`cmath`'s operation in a sensible manner, as is already possible for four methods he
>> mentions, as well as a number of other Python operations[0], including prominent numerical ones[1].
>
Correct. If a type does not implement one of the special methods, then
there would be no change to the behaviour of the math/cmath modules.

> I like the idea of being able to extend the math functions to allow
> user-defined numeric types to behave like built in ones. (In fact, if
> this were possible, complex could use it and a large proportion of the
> cmath module would be unneeded). However, I have to say that for me
> this is a purely theoretical issue - I've never had the need for the
> functionality myself in real code, and I doubt I ever will. Given the
> proliferation of special methods that would be needed, plus the
> runtime overhead of extra checking, I think the cost is likely too
> high.
>
> It's also worth noting the preamble text in the math module
> documentation - the functions there deliberately only handle floats,
> and are thin wrappers over the C standard library. To get any sort of
> generic behaviour, users need to use cmath, by design. So as stated
> the proposal is unlikely to get much support.
>
> Some alternative suggestions:
>
> 1. As already noted, you could include a gmpmath module with the
> relevant functions, and users could monkeypatch the math module if
> they wanted to do so.
>
The gmpy2 module provides a superset of the functions in math/cmath.
I'm trying to avoid name collisions if someone replaces "from math
import *" with "from gmpy2 import *". I know they shouldn't do
that....

> 2. To make such monkeypatching easier, publish a module that exposes a
> context manager to do the job:
> ? ? ? with monkeypatch(math, 'sin', gmpmath.sin):
> ? ? ? ? ? your code here...
>
> 3. Write a genericmath module that provides the "generic" versions
> you're proposing, and see how much use it gets - if it's sufficiently
> popular, you have a better case then for folding the functionality
> into at least cmath.
I'll try to put a prototype together. One issue with merging math and
cmath is that math.functions raise an exception instead of returning a
complex result. If it becomes a single module, should it return
complex values instead of raising an exception? That could be a major
change in someone's code. (gmpy2 uses a context manager to
enable/disable the return of a complex result from a real function.)
>
> 4. As Georg mentioned, numpy's ufuncs have a feature like this, so
> support that and people can use your types with numpy. That may be
> enough (depending on your expected user base).
>

I will look at numpy's ufuncs.

> Overall, I think the idea of being able to use user-defined types
> interchangeably with built-in ones is a good one, but it's not
> something Python goes out of its way to support. If the idea of
> generic functions had taken off, then this is an area where they would
> have fit nicely - but once again, there wasn't much enthusiasm from
> the core developers for addressing the types of problem they were
> aimed at.
>
> Paul.
>
casevh


From steve at pearwood.info  Thu Nov 10 00:02:29 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 10 Nov 2011 10:02:29 +1100
Subject: [Python-ideas] Add additional special method lookups to math
 module
In-Reply-To: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
Message-ID: <4EBB0685.4030601@pearwood.info>

Case Van Horsen wrote:
> Currently the functions round(), math.ceil(), math.floor(), and
> math.trunc() all check for the existence of a special method
> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
> to enhance the math and cmath modules to check for the existence of a
> special method for (almost) functions? For example, math.sin(obj)
> would first check for obj.__sin__.

I would not object to this.

The only function I can honestly say I have had a concrete use-case for 
is math.sqrt. This comes up from time to time, e.g.:

http://bytes.com/topic/python/answers/463861-how-overload-sqrt-module
http://permalink.gmane.org/gmane.comp.python.general/694849

However, how far should we go? Does every function in the math module 
require a dunder method, e.g. __degrees__ ? What happens if we add more 
functions, say math.bessel? Do we really expect that all numeric types 
must support a __bessel__ method? I suspect that this proposal is 
actually bigger than it seems at first glance.

We can:

* Do nothing (the status quo). If you write a numeric type, you can 
support a small number of mathematical operations, such as + and 
math.floor, but not others, such as math.sqrt or math.sin.

* Officially recommend that people monkey-patch the math module if they 
want to write a drop-in replacement for numeric types. I consider this 
unspeakable, but mention it for completeness since others have raised 
the possibility.

* Add support for dunder methods in an ad hoc manner, when and as 
requested, without making any promises about any other functions.

* Add support for dunder methods in a systematic way. This would require 
distinguishing between fundamental operations that should support dunder 
methods, and those that shouldn't (if any). This will probably need a PEP.

* Instead of having to decide what operations should be supported ahead 
of time, perhaps there is a way for types to register themselves with 
the math module, e.g. say "I support sin, but not sinh". Somewhat akin 
to the way ABCs work, at least conceptually. One advantage of that may 
be that numeric classes won't have to use dunder methods for supporting 
the math module, e.g. MyNumber might register sin rather than __sin__.


-- 
Steven



From ncoghlan at gmail.com  Thu Nov 10 01:24:18 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 10 Nov 2011 10:24:18 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111109170438.0626d40b@limelight.wooz.org>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
	<CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>
	<20111109170438.0626d40b@limelight.wooz.org>
Message-ID: <CADiSq7d2TkAQm=TPqwFBddfFbwfwgMO2Rp+KYTMhibKaANGg8A@mail.gmail.com>

On Thu, Nov 10, 2011 at 8:04 AM, Barry Warsaw <barry at python.org> wrote:
> Isn't that a problem with the basic terminology then? ?If you don't know what
> a "qualified name" is you probably won't know what a "qname" is, and you
> definitely won't make that connection. ?I think that's more reason to find the
> right terminology and spell it out.

There may not *be* 'right terminology' for what a qname is going to
be, since we're basically inventing the concept to cover some gaps in
the meaning of __name__. I'm setting the mnemonic bar at "once you
know what a Python qname is, is the name suggestive enough to let you
remember it?", not at the much higher "if you encounter the Python
qname term without knowing what it means, are you going to be able to
guess correctly without looking it up?". I'm also considering the
reverse problem of  "if you encounter the Python qname term without
knowing what it means, are you going to *think* you know what it means
without looking it up and be incorrect?".

For me, it's that last point that rules out ideas like "full name" and
"fully qualified" name - the proposed names deliberately *aren't*
fully qualified in the class and function cases (since they still
leave out the module information, storing that on a separate
attribute).

For the former considerations, I think the differences between
existing __name__ attributes and the new attributes are too subtle and
obscure to ever allow completely intuitive terminology. For classes
and functions, __name__ refers to the name given to the relevant
constructor (either directly or syntactically). This is insufficient
to correctly identify classes and functions that are not defined at
the top level in their respective modules, so the new attribute
addresses that. For modules, __name__ normally refers to the name that
was used to import the module, *unless* it has been altered for some
reason (either to mark the main module or else to redirect
serialisation operations to a public API instead of an implementation
module). The new attribute then allows those alterations of __name__
to take place without losing the information about which module was
actually imported to create the module namespace.

"qname" and "qualified name" are nice, neutral terminology that
suggest something closely related to "name" with being identical. In
all proposed use cases, they preserve information that would otherwise
have been lost.

That said, in earlier drafts of PEP 395 (module aliasing), I did
suggest the term "implementation name". As in the current PEP,
"__name__" would retain its current semantics, while "__impl_name__"
would be what was actually imported when creating the module.

Logic that needed to figure out an objects full name would then look
something like:

    mod_name = obj.__module__
    if mod_name == "__main__":
        mod_name = sys.modules[mod_name].__impl_name__
    full_name = mod_name + "." + obj.__impl_name__

Introspection logic would use either or both, depending on the context
(e.g. pydoc would likely favour using __name__, while inspect would
prefer __impl_name__ for things like getsource()).

It seems to me that "implementation name" would work as a term for
Antoine's PEP as well, since it's about providing enough information
to locate the actual implementation of the class or function relative
to the top level of the module.

The term "implementation name" is also a bit more suggestive of the
intended purpose of these attributes than the completely neutral
"qualified name".

That means my current order of naming preference is:

   - "implementation name" & "__impl_name__"
   - "qualified name" & "__qname__"
   - "qualified name" & "__qualname__"
   - "qualified name" & "__qual_name__"

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From greg.ewing at canterbury.ac.nz  Thu Nov 10 01:29:13 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 10 Nov 2011 13:29:13 +1300
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <4EBB0685.4030601@pearwood.info>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<4EBB0685.4030601@pearwood.info>
Message-ID: <4EBB1AD9.5080907@canterbury.ac.nz>

On 10/11/11 12:02, Steven D'Aprano wrote:

> However, how far should we go? Does every function in the math module require
> a dunder method, e.g. __degrees__ ? What happens if we add more functions, say
> math.bessel?

Another approach to all this would be to provide generic implementations
of the math functions in terms of the basic arithmetic operations. While
this wouldn't always be as fast as using type-specific methods, it would
at least work with all number-like types and would extend easily to any
new functions we might add without having to update every existing
numeric type.

It might even be possible to support type-specific accelerations of these
generic implementations using a __polynomial__ method that takes a list
of coefficients, or maybe a function for generating the coefficients.

-- 
Greg


From casevh at gmail.com  Thu Nov 10 01:55:16 2011
From: casevh at gmail.com (Case Van Horsen)
Date: Wed, 9 Nov 2011 16:55:16 -0800
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <4EBB0685.4030601@pearwood.info>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<4EBB0685.4030601@pearwood.info>
Message-ID: <CANerV6=L6N_UZmQPzB2hCrj+fsTY40BsyFApg0zkGe25h9oZkQ@mail.gmail.com>

On Wed, Nov 9, 2011 at 3:02 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Case Van Horsen wrote:
>>
>> Currently the functions round(), math.ceil(), math.floor(), and
>> math.trunc() all check for the existence of a special method
>> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
>> to enhance the math and cmath modules to check for the existence of a
>> special method for (almost) functions? For example, math.sin(obj)
>> would first check for obj.__sin__.
>
> I would not object to this.
>
> The only function I can honestly say I have had a concrete use-case for is
> math.sqrt. This comes up from time to time, e.g.:
>
> http://bytes.com/topic/python/answers/463861-how-overload-sqrt-module
> http://permalink.gmane.org/gmane.comp.python.general/694849
>
> However, how far should we go? Does every function in the math module
> require a dunder method, e.g. __degrees__ ? What happens if we add more
> functions, say math.bessel? Do we really expect that all numeric types must
> support a __bessel__ method? I suspect that this proposal is actually bigger
> than it seems at first glance.

It would be completely optional for a numeric type to support these
methods. If they're not supported, the numeric type is converted to a
float and then math.function proceeds as it currently does.

>
> We can:
>
> * Do nothing (the status quo). If you write a numeric type, you can support
> a small number of mathematical operations, such as + and math.floor, but not
> others, such as math.sqrt or math.sin.
>
> * Officially recommend that people monkey-patch the math module if they want
> to write a drop-in replacement for numeric types. I consider this
> unspeakable, but mention it for completeness since others have raised the
> possibility.
>
> * Add support for dunder methods in an ad hoc manner, when and as requested,
> without making any promises about any other functions.

I hacked mathmodule.c FUNC1 macro to perform the lookup the many of
math module functions. It was only about 15 lines of code (but it
doesn't check if NotImplemented is returned.) Unfortunately, it
increases the running time of math.sin, for example, by 15%. I need to
look at the Identifier API to see if that helps the performance but I
don't think 15% is a penalty everyone should pay.

>
> * Add support for dunder methods in a systematic way. This would require
> distinguishing between fundamental operations that should support dunder
> methods, and those that shouldn't (if any). This will probably need a PEP.
>
Especially true if new nb_slots need to be created to avoid the
performance impact. I was hoping the performance impact of the special
method lookup was negligible so it could be a low impact change.

> * Instead of having to decide what operations should be supported ahead of
> time, perhaps there is a way for types to register themselves with the math
> module, e.g. say "I support sin, but not sinh". Somewhat akin to the way
> ABCs work, at least conceptually. One advantage of that may be that numeric
> classes won't have to use dunder methods for supporting the math module,
> e.g. MyNumber might register sin rather than __sin__.

As long as they support __float__, they'll continue to work with the
math module.
>
>
> --
> Steven
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From solipsis at pitrou.net  Thu Nov 10 02:00:42 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 10 Nov 2011 02:00:42 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
	<CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>
	<20111109170438.0626d40b@limelight.wooz.org>
	<CADiSq7d2TkAQm=TPqwFBddfFbwfwgMO2Rp+KYTMhibKaANGg8A@mail.gmail.com>
Message-ID: <20111110020042.20580ba3@pitrou.net>

On Thu, 10 Nov 2011 10:24:18 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> It seems to me that "implementation name" would work as a term for
> Antoine's PEP as well, since it's about providing enough information
> to locate the actual implementation of the class or function relative
> to the top level of the module.

I don't really follow this reasoning. There is no other object than the
"actual implementation"; __name__ and __q[ual]name__ denote the same
thing. "Implementation name" sounds like a misnomer to me.

If __qname__ is too elliptic, let's settle on __qualname__?

Regards

Antoine.




From solipsis at pitrou.net  Thu Nov 10 02:01:52 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 10 Nov 2011 02:01:52 +0100
Subject: [Python-ideas] Add additional special method lookups to math
	module
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<4EBB0685.4030601@pearwood.info>
	<CANerV6=L6N_UZmQPzB2hCrj+fsTY40BsyFApg0zkGe25h9oZkQ@mail.gmail.com>
Message-ID: <20111110020152.55c39c18@pitrou.net>

On Wed, 9 Nov 2011 16:55:16 -0800
Case Van Horsen <casevh at gmail.com> wrote:

> On Wed, Nov 9, 2011 at 3:02 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> > Case Van Horsen wrote:
> >>
> >> Currently the functions round(), math.ceil(), math.floor(), and
> >> math.trunc() all check for the existence of a special method
> >> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
> >> to enhance the math and cmath modules to check for the existence of a
> >> special method for (almost) functions? For example, math.sin(obj)
> >> would first check for obj.__sin__.
> >
> > I would not object to this.
> >
> > The only function I can honestly say I have had a concrete use-case for is
> > math.sqrt. This comes up from time to time, e.g.:
> >
> > http://bytes.com/topic/python/answers/463861-how-overload-sqrt-module
> > http://permalink.gmane.org/gmane.comp.python.general/694849
> >
> > However, how far should we go? Does every function in the math module
> > require a dunder method, e.g. __degrees__ ? What happens if we add more
> > functions, say math.bessel? Do we really expect that all numeric types must
> > support a __bessel__ method? I suspect that this proposal is actually bigger
> > than it seems at first glance.
> 
> It would be completely optional for a numeric type to support these
> methods. If they're not supported, the numeric type is converted to a
> float and then math.function proceeds as it currently does.
> 
> >
> > We can:
> >
> > * Do nothing (the status quo). If you write a numeric type, you can support
> > a small number of mathematical operations, such as + and math.floor, but not
> > others, such as math.sqrt or math.sin.
> >
> > * Officially recommend that people monkey-patch the math module if they want
> > to write a drop-in replacement for numeric types. I consider this
> > unspeakable, but mention it for completeness since others have raised the
> > possibility.
> >
> > * Add support for dunder methods in an ad hoc manner, when and as requested,
> > without making any promises about any other functions.
> 
> I hacked mathmodule.c FUNC1 macro to perform the lookup the many of
> math module functions. It was only about 15 lines of code (but it
> doesn't check if NotImplemented is returned.) Unfortunately, it
> increases the running time of math.sin, for example, by 15%. I need to
> look at the Identifier API to see if that helps the performance but I
> don't think 15% is a penalty everyone should pay.

You can special-case floats using PyFloat_CheckExact and see if that
helps.

Regards

Antoine.




From tjreedy at udel.edu  Thu Nov 10 02:32:56 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 09 Nov 2011 20:32:56 -0500
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <4EBB1AD9.5080907@canterbury.ac.nz>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<4EBB0685.4030601@pearwood.info>
	<4EBB1AD9.5080907@canterbury.ac.nz>
Message-ID: <j9f9kb$abt$1@dough.gmane.org>

On 11/9/2011 7:29 PM, Greg Ewing wrote:
> On 10/11/11 12:02, Steven D'Aprano wrote:
>
>> However, how far should we go? Does every function in the math module
>> require
>> a dunder method, e.g. __degrees__ ? What happens if we add more
>> functions, say
>> math.bessel?
>
> Another approach to all this would be to provide generic implementations
> of the math functions in terms of the basic arithmetic operations.

Perhaps in a gmath (generic math) module.

 > While
> this wouldn't always be as fast as using type-specific methods, it would
> at least work with all number-like types and would extend easily to any
> new functions we might add without having to update every existing
> numeric type.
>
> It might even be possible to support type-specific accelerations of these
> generic implementations using a __polynomial__ method that takes a list
> of coefficients, or maybe a function for generating the coefficients.

-- 
Terry Jan Reedy



From ncoghlan at gmail.com  Thu Nov 10 02:38:31 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 10 Nov 2011 11:38:31 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111110020042.20580ba3@pitrou.net>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
	<CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>
	<20111109170438.0626d40b@limelight.wooz.org>
	<CADiSq7d2TkAQm=TPqwFBddfFbwfwgMO2Rp+KYTMhibKaANGg8A@mail.gmail.com>
	<20111110020042.20580ba3@pitrou.net>
Message-ID: <CADiSq7eCaxRJ32y5yvpKJ7Z6NDZtTbHxocatXA4XYBi8DkLYGQ@mail.gmail.com>

On Thu, Nov 10, 2011 at 11:00 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 10 Nov 2011 10:24:18 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> It seems to me that "implementation name" would work as a term for
>> Antoine's PEP as well, since it's about providing enough information
>> to locate the actual implementation of the class or function relative
>> to the top level of the module.
>
> I don't really follow this reasoning. There is no other object than the
> "actual implementation"; __name__ and __q[ual]name__ denote the same
> thing. "Implementation name" sounds like a misnomer to me.

Yeah, on further reflection, I agree that the connotations suggesting
a separate implementation object may be too strong for that to be a
reasonable term. That's probably why I dropped it in the first place.

> If __qname__ is too elliptic, let's settle on __qualname__?

"q name" is easy to say, "qual name" is relatively hard to say - if
we're going to abbreviate, it should be to something pronounceable.
I'd say try to summarise this naming discussion into the PEP, but
otherwise stick to the "qualified name" and "__qname__" proposal
(unless/until Guido says otherwise).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From casevh at gmail.com  Thu Nov 10 02:39:37 2011
From: casevh at gmail.com (Case Van Horsen)
Date: Wed, 9 Nov 2011 17:39:37 -0800
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <20111110020152.55c39c18@pitrou.net>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<4EBB0685.4030601@pearwood.info>
	<CANerV6=L6N_UZmQPzB2hCrj+fsTY40BsyFApg0zkGe25h9oZkQ@mail.gmail.com>
	<20111110020152.55c39c18@pitrou.net>
Message-ID: <CANerV6k=Nou-UKr5+-LB9jYS_MOYZsfK09Q47PQzjT5yXNN8pA@mail.gmail.com>

On Wed, Nov 9, 2011 at 5:01 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 9 Nov 2011 16:55:16 -0800
> Case Van Horsen <casevh at gmail.com> wrote:
>
>> On Wed, Nov 9, 2011 at 3:02 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>> > Case Van Horsen wrote:
>> >>
>> >> Currently the functions round(), math.ceil(), math.floor(), and
>> >> math.trunc() all check for the existence of a special method
>> >> (__round__, __ceil__, __floor__, and __trunc__). Would it be possible
>> >> to enhance the math and cmath modules to check for the existence of a
>> >> special method for (almost) functions? For example, math.sin(obj)
>> >> would first check for obj.__sin__.
>> >
>> > I would not object to this.
>> >
>> > The only function I can honestly say I have had a concrete use-case for is
>> > math.sqrt. This comes up from time to time, e.g.:
>> >
>> > http://bytes.com/topic/python/answers/463861-how-overload-sqrt-module
>> > http://permalink.gmane.org/gmane.comp.python.general/694849
>> >
>> > However, how far should we go? Does every function in the math module
>> > require a dunder method, e.g. __degrees__ ? What happens if we add more
>> > functions, say math.bessel? Do we really expect that all numeric types must
>> > support a __bessel__ method? I suspect that this proposal is actually bigger
>> > than it seems at first glance.
>>
>> It would be completely optional for a numeric type to support these
>> methods. If they're not supported, the numeric type is converted to a
>> float and then math.function proceeds as it currently does.
>>
>> >
>> > We can:
>> >
>> > * Do nothing (the status quo). If you write a numeric type, you can support
>> > a small number of mathematical operations, such as + and math.floor, but not
>> > others, such as math.sqrt or math.sin.
>> >
>> > * Officially recommend that people monkey-patch the math module if they want
>> > to write a drop-in replacement for numeric types. I consider this
>> > unspeakable, but mention it for completeness since others have raised the
>> > possibility.
>> >
>> > * Add support for dunder methods in an ad hoc manner, when and as requested,
>> > without making any promises about any other functions.
>>
>> I hacked mathmodule.c FUNC1 macro to perform the lookup the many of
>> math module functions. It was only about 15 lines of code (but it
>> doesn't check if NotImplemented is returned.) Unfortunately, it
>> increases the running time of math.sin, for example, by 15%. I need to
>> look at the Identifier API to see if that helps the performance but I
>> don't think 15% is a penalty everyone should pay.
>
> You can special-case floats using PyFloat_CheckExact and see if that
> helps.

Thanks. That decreases the penalty to about 1.5%. I'll finish hacking
mathmodule.c and gmpy2 and run some additional tests.

casevh
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From solipsis at pitrou.net  Thu Nov 10 02:36:49 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 10 Nov 2011 02:36:49 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
 functions
In-Reply-To: <CADiSq7eCaxRJ32y5yvpKJ7Z6NDZtTbHxocatXA4XYBi8DkLYGQ@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
	<CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>
	<20111109170438.0626d40b@limelight.wooz.org>
	<CADiSq7d2TkAQm=TPqwFBddfFbwfwgMO2Rp+KYTMhibKaANGg8A@mail.gmail.com>
	<20111110020042.20580ba3@pitrou.net>
	<CADiSq7eCaxRJ32y5yvpKJ7Z6NDZtTbHxocatXA4XYBi8DkLYGQ@mail.gmail.com>
Message-ID: <1320889009.3370.8.camel@localhost.localdomain>

Le jeudi 10 novembre 2011 ? 11:38 +1000, Nick Coghlan a ?crit :
> On Thu, Nov 10, 2011 at 11:00 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Thu, 10 Nov 2011 10:24:18 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
> >>
> >> It seems to me that "implementation name" would work as a term for
> >> Antoine's PEP as well, since it's about providing enough information
> >> to locate the actual implementation of the class or function relative
> >> to the top level of the module.
> >
> > I don't really follow this reasoning. There is no other object than the
> > "actual implementation"; __name__ and __q[ual]name__ denote the same
> > thing. "Implementation name" sounds like a misnomer to me.
> 
> Yeah, on further reflection, I agree that the connotations suggesting
> a separate implementation object may be too strong for that to be a
> reasonable term. That's probably why I dropped it in the first place.
> 
> > If __qname__ is too elliptic, let's settle on __qualname__?
> 
> "q name" is easy to say, "qual name" is relatively hard to say - if
> we're going to abbreviate, it should be to something pronounceable.

This must depend where you come from. I have no problem pronouncing
"qualname" with a gross French accent, and I'm sure Victor would
understand me :-)

Regards

Antoine.




From jimjjewett at gmail.com  Thu Nov 10 03:17:53 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 9 Nov 2011 21:17:53 -0500
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <73289562-327E-4771-9D10-4D6ED251477E@masklinn.net>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
	<F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
	<j9eo0l$u3r$1@dough.gmane.org>
	<DAA27B99-430F-4845-A2C3-50AD7FB41BA0@masklinn.net>
	<j9eq7s$h0b$1@dough.gmane.org>
	<73289562-327E-4771-9D10-4D6ED251477E@masklinn.net>
Message-ID: <CA+OGgf44f-3Us+dpDpdSpD_fReCzf6qTFh4mt+=gS5vEgN+tPA@mail.gmail.com>

On 11/9/11, Masklinn <masklinn at masklinn.net> wrote:
> On 2011-11-09, at 22:10 , Jeremy Sanders wrote:
>> Masklinn wrote:

>> If you go and change the behaviour of built-in functions just by importing
>> a  module it could lead to some weird bugs. I know monkey patching can
>> do this  currently, but doing it by default seems dangerous to me.
> Nothing in these two paragraphs makes any sense in the context of the
> current discussion. *Nobody* (but you) argues for "changing the behavior of
> built-in functions just by importing a module", or for "overriding a
> standard module".

Several people have suggested that the standard library module not be
changed, but that users be encouraged to monkeypatch instead.

In other words, if I import funkymathtype, it should (as part of the
import process) monkeypatch math (and cmath) to replace the builtin
functions with something that handles its own types as well.  Short of
generic functions, I happen to think that is a bad idea, but it was
suggested.

And frankly, "import funkymathtype.math as math" probably is the right
answer, in the short run, but that is still pretty fragile against
other modules having imported math themselves.

-jJ


-jJ


From jimjjewett at gmail.com  Thu Nov 10 03:45:38 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 9 Nov 2011 21:45:38 -0500
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <CADiSq7d2TkAQm=TPqwFBddfFbwfwgMO2Rp+KYTMhibKaANGg8A@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
	<CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>
	<20111109170438.0626d40b@limelight.wooz.org>
	<CADiSq7d2TkAQm=TPqwFBddfFbwfwgMO2Rp+KYTMhibKaANGg8A@mail.gmail.com>
Message-ID: <CA+OGgf7mAqCRTzKCHiy-mtmKqbraEGOvdPkvC-fkaUdbXcm9nQ@mail.gmail.com>

On 11/9/11, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Thu, Nov 10, 2011 at 8:04 AM, Barry Warsaw <barry at python.org> wrote:
>> Isn't that a problem with the basic terminology then?  If you don't know
>> what a "qualified name" is you probably won't know what a "qname" is,
>> and you definitely won't make that connection.  I think that's more
>> reason to find the right terminology and spell it out.

I know that a "qualified name" is a special kind of name, and that if
I have to worry about it, then I should expect subtle bugs with either
false matches, false mismatches, or both.

I cannot imagine a more specific definition that is both true and
sufficiently simple that I don't have to look it up.

But frankly, that seems to be true of the proposed names as well
(usually but not always equal to __name__, often but not always
sufficient to figure out where the object was defined, etc).

The only cognitive grief is about figuring out when I *do* need to use
a qname instead of a regular name; I don't see a rule of thumb like
"Handles user-generated strings" or "Multiple namespaces are being
used".

> There may not *be* 'right terminology' for what a qname is going to
> be, since we're basically inventing the concept to cover some gaps in
> the meaning of __name__.

And so I suspect there isn't a simple term.  (There might be several
simple terms that each describe some but not all of the use cases.)

> I'm setting the mnemonic bar at "once you
> know what a Python qname is, is the name suggestive enough to let you
> remember it?", not at the much higher "if you encounter the Python
> qname term without knowing what it means, are you going to be able to
> guess correctly without looking it up?".

I would say that all of __qname__, __q_name__, __qualname__,
__qualifiedname__, etc meet that bar, as do __extendedname__ or
__impl_name__.

> I'm also considering the
> reverse problem of  "if you encounter the Python qname term without
> knowing what it means, are you going to *think* you know what it means
> without looking it up and be incorrect?".

I wouldn't, but I'm probably a bad judge on that.

-jJ


From greg.ewing at canterbury.ac.nz  Thu Nov 10 04:11:23 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 10 Nov 2011 16:11:23 +1300
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <CA+OGgf44f-3Us+dpDpdSpD_fReCzf6qTFh4mt+=gS5vEgN+tPA@mail.gmail.com>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<j9emgm$hvg$1@dough.gmane.org>
	<F8D98BE5-F3E5-47DA-87D7-D56C3ECE5496@masklinn.net>
	<j9eo0l$u3r$1@dough.gmane.org>
	<DAA27B99-430F-4845-A2C3-50AD7FB41BA0@masklinn.net>
	<j9eq7s$h0b$1@dough.gmane.org>
	<73289562-327E-4771-9D10-4D6ED251477E@masklinn.net>
	<CA+OGgf44f-3Us+dpDpdSpD_fReCzf6qTFh4mt+=gS5vEgN+tPA@mail.gmail.com>
Message-ID: <4EBB40DB.5070700@canterbury.ac.nz>

On 10/11/11 15:17, Jim Jewett wrote:

> And frankly, "import funkymathtype.math as math" probably is the right
> answer, in the short run, but that is still pretty fragile against
> other modules having imported math themselves.

The monkeypatching idea doesn't entirely avoid that problem either,
since you would have to make sure the module doing the monkeypatching
was imported before any other module imported stuff from math.

-- 
Greg


From ncoghlan at gmail.com  Thu Nov 10 04:22:27 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 10 Nov 2011 13:22:27 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <1320889009.3370.8.camel@localhost.localdomain>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<CADiSq7eK85LkhVoLJBcJ0=OO9riENtYOEc_QuLsGkK3GyUBxoA@mail.gmail.com>
	<20111108105540.6ca5d579@resist.wooz.org>
	<CA+OGgf53Wiz3TGDuB7g+fi7-n2L7Rp+ppWUib+Cs5TJg=qa1Wg@mail.gmail.com>
	<20111109170438.0626d40b@limelight.wooz.org>
	<CADiSq7d2TkAQm=TPqwFBddfFbwfwgMO2Rp+KYTMhibKaANGg8A@mail.gmail.com>
	<20111110020042.20580ba3@pitrou.net>
	<CADiSq7eCaxRJ32y5yvpKJ7Z6NDZtTbHxocatXA4XYBi8DkLYGQ@mail.gmail.com>
	<1320889009.3370.8.camel@localhost.localdomain>
Message-ID: <CADiSq7cc3MTMx89Uq7M3tk+MSJCOuZkGwWuAHv+6R7BADiNnig@mail.gmail.com>

On Thu, Nov 10, 2011 at 11:36 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> "q name" is easy to say, "qual name" is relatively hard to say - if
>> we're going to abbreviate, it should be to something pronounceable.
>
> This must depend where you come from. I have no problem pronouncing
> "qualname" with a gross French accent, and I'm sure Victor would
> understand me :-)

It isn't so much a matter of finding qualname hard to say, as finding
qname really easy to say, so I don't actually mind if you do decide to
change it in the PEP :)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From anacrolix at gmail.com  Thu Nov 10 09:10:04 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Thu, 10 Nov 2011 19:10:04 +1100
Subject: [Python-ideas] try-else without except or finally
Message-ID: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>

Hi Ideas,

I frequently find myself in the following situations:

1) I wish to do something if no exception is thrown, for instance:

        try:
            logger.debug('Fiber starting: %s', self)
            try:
                self._result = self._routine()
            finally:
                logger.debug('Fiber finished: %s', self)
        except:
            raise
        else:
            raise FiberExit(self)
        finally:
            self._finished.set()
            unregister_fiber(self)

Here it's sufficient that that if an exception is already present, I
don't need to raise another. The except clause is clearly pointless.
2) I'm experimenting with catching various exceptions and remove the
last except clause. I need to put a finally: pass; to avoid having to
restructure all my code, since this is currently the only way to
maintain the try-except-else-finally statement without catching dummy
exceptions.

I propose that the except clause be optional.

Cheers,
Matt


From rob.cliffe at btinternet.com  Thu Nov 10 09:38:14 2011
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Thu, 10 Nov 2011 08:38:14 +0000
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
Message-ID: <4EBB8D76.1080903@btinternet.com>

Omitting the except clause would make the code more obscure.
'else' means the second of two alternatives.  Omitting the first 
alternative altogether obfuscates the code and makes it look as if the 
first alternative is the try block, i.e. it suggests if anything that 
the code following 'else:' is executed when there is an exception rather 
than when there isn't.
-1 from me
Rob Cliffe

On 10/11/2011 08:10, Matt Joiner wrote:
> Hi Ideas,
>
> I frequently find myself in the following situations:
>
> 1) I wish to do something if no exception is thrown, for instance:
>
>          try:
>              logger.debug('Fiber starting: %s', self)
>              try:
>                  self._result = self._routine()
>              finally:
>                  logger.debug('Fiber finished: %s', self)
>          except:
>              raise
>          else:
>              raise FiberExit(self)
>          finally:
>              self._finished.set()
>              unregister_fiber(self)
>
> Here it's sufficient that that if an exception is already present, I
> don't need to raise another. The except clause is clearly pointless.
> 2) I'm experimenting with catching various exceptions and remove the
> last except clause. I need to put a finally: pass; to avoid having to
> restructure all my code, since this is currently the only way to
> maintain the try-except-else-finally statement without catching dummy
> exceptions.
>
> I propose that the except clause be optional.
>
> Cheers,
> Matt
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From dennis.kaarsemaker at booking.com  Thu Nov 10 10:12:59 2011
From: dennis.kaarsemaker at booking.com (Dennis Kaarsemaker)
Date: Thu, 10 Nov 2011 10:12:59 +0100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
Message-ID: <1320916379.10012.69.camel@seahawk>

On Thu, 2011-11-10 at 19:10 +1100, Matt Joiner wrote:
> Hi Ideas,
> 
> I frequently find myself in the following situations:
> 
> 1) I wish to do something if no exception is thrown, for instance:

How is this...

>         try:
>             logger.debug('Fiber starting: %s', self)
>             try:
>                 self._result = self._routine()
>             finally:
>                 logger.debug('Fiber finished: %s', self)
>         except:
>             raise
>         else:
>             raise FiberExit(self)
>         finally:
>             self._finished.set()
>             unregister_fiber(self)

different from this...

        try:
            logger.debug('Fiber starting: %s', self)
            try:
                self._result = self._routine()
            finally:
                logger.debug('Fiber finished: %s', self)
            raise FiberExit(self)
        finally:
            self._finished.set()
            unregister_fiber(self)

That should do exactly what you want, shouldn't it?
-- 
Dennis Kaarsemaker         Senior Unix System Administrator
Amsterdam, The Netherlands   dennis.kaarsemaker at booking.com
http://www.booking.com           tel external +3120715 3409
                                    tel internal (7207)3409




From p.f.moore at gmail.com  Thu Nov 10 12:23:01 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 10 Nov 2011 11:23:01 +0000
Subject: [Python-ideas] Add additional special method lookups to math
	module
In-Reply-To: <4EBB0685.4030601@pearwood.info>
References: <CANerV6ntrdq2WPFzaCOAaWy1U3SN7J_8qSB1eWNsGnjksd0pdw@mail.gmail.com>
	<4EBB0685.4030601@pearwood.info>
Message-ID: <CACac1F8LR0C1tHaMGahd4LLh-dqoGEb3-_HcdGfaet=QQsuorw@mail.gmail.com>

On 9 November 2011 23:02, Steven D'Aprano <steve at pearwood.info> wrote:
> * Instead of having to decide what operations should be supported ahead of
> time, perhaps there is a way for types to register themselves with the math
> module, e.g. say "I support sin, but not sinh". Somewhat akin to the way
> ABCs work, at least conceptually. One advantage of that may be that numeric
> classes won't have to use dunder methods for supporting the math module,
> e.g. MyNumber might register sin rather than __sin__.

I haven't checked the details, but isn't this *exactly* how ABCs work?
So math could first check for exact floats (for performance), then
check for whether the object is an instance of the
"TranscendentalMaths" ABC, and finally fall back to converting to
float.

All of this is ignoring the question of whether it's acceptable to
change the documented contract of math to only work on actual floats,
of course...

Paul.


From steve at pearwood.info  Thu Nov 10 14:04:34 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 11 Nov 2011 00:04:34 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
Message-ID: <4EBBCBE2.7090409@pearwood.info>

Matt Joiner wrote:
> Hi Ideas,
> 
> I frequently find myself in the following situations:
> 
> 1) I wish to do something if no exception is thrown, for instance:
[...]
> I propose that the except clause be optional.


I too have been in the situation of wanting an else clause and a finally 
clause, but no except clause.

+1 on allowing else without an except.



-- 
Steven



From ubershmekel at gmail.com  Thu Nov 10 14:13:53 2011
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Thu, 10 Nov 2011 15:13:53 +0200
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBCBE2.7090409@pearwood.info>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
Message-ID: <CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>

On Nov 10, 2011 3:05 PM, "Steven D&apos;Aprano" <steve at pearwood.info> wrote:
>
> Matt Joiner wrote:
>>
>> Hi Ideas,
>>
>> I frequently find myself in the following situations:
>>
>> 1) I wish to do something if no exception is thrown, for instance:
>
> [...]
>
>> I propose that the except clause be optional.
>
>
>
> I too have been in the situation of wanting an else clause and a finally
clause, but no except clause.
>
>

"Try...else..." doesn't make any sense without the except jn my eyes. The
"else" has to conjugate something.

-1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111110/00ab8030/attachment.html>

From steve at pearwood.info  Thu Nov 10 14:28:50 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 11 Nov 2011 00:28:50 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
Message-ID: <4EBBD192.3020901@pearwood.info>

Yuval Greenfield wrote:

> "Try...else..." doesn't make any sense without the except jn my eyes. The
> "else" has to conjugate something.


We already have while...else and for...else blocks in Python. The 
principle is the same:


while condition():
     ...
else:
     # Run the else block unless we exit the while block with a break.
     ...


try:
     ...
else:
     # Run the else block unless we exit the try block with an exception.
     ...
finally:
     ...


This currently doesn't work without a pointless "except: raise" clause 
to satisfy the compiler. I try to avoid writing pointless code.



-- 
Steven



From rob.cliffe at btinternet.com  Thu Nov 10 14:35:57 2011
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Thu, 10 Nov 2011 13:35:57 +0000
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBD192.3020901@pearwood.info>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info>
Message-ID: <4EBBD33D.7080908@btinternet.com>



On 10/11/2011 13:28, Steven D'Aprano wrote:
> Yuval Greenfield wrote:
>
>> "Try...else..." doesn't make any sense without the except jn my eyes. 
>> The
>> "else" has to conjugate something.
>
>
> We already have while...else and for...else blocks in Python. The 
> principle is the same:
>
>
> while condition():
>     ...
> else:
>     # Run the else block unless we exit the while block with a break.
>     ...
>
>
> try:
>     ...
> else:
>     # Run the else block unless we exit the try block with an exception.
>     ...
> finally:
>     ...
>
>
> This currently doesn't work without a pointless "except: raise" clause 
> to satisfy the compiler. I try to avoid writing pointless code.
>
>
>
"except: raise" makes it explicit that an exception is to be 
propogated.  Far from being pointless it makes the code much more 
readable.  "Explicit is better than implicit."
Rob Cliffe


From steve at pearwood.info  Thu Nov 10 15:24:57 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 11 Nov 2011 01:24:57 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBD33D.7080908@btinternet.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>	<4EBBD192.3020901@pearwood.info>
	<4EBBD33D.7080908@btinternet.com>
Message-ID: <4EBBDEB9.2010904@pearwood.info>

Rob Cliffe wrote:

> "except: raise" makes it explicit that an exception is to be 
> propogated.  Far from being pointless it makes the code much more 
> readable.  "Explicit is better than implicit."

Look, I love the Zen as much as the next guy, but sometimes they get 
used as thought-terminating cliches. Catching an exception only to 
unconditionally raise it again unchanged is a waste of everybody's time: 
don't catch exceptions you have no intention of dealing with.

Python has allowed try...finally without an except clause since version 
1.5, if not earlier.

Virtually every line of Python code may implicitly raise an exception. 
Despite the Zen, we don't go around writing

try:
     n = len(x)
except:
     raise

just to satisfy the urge to make it explicit that any exception that 
occurs will be propagated. We don't need an explicit reminder that any 
exceptions will be propagated because, absent an except clause to 
explicitly silence it, that's what exceptions do: they propagate.

Errors should never pass silently.
Unless explicitly silenced.

(see, I can quote the Zen too).


-- 
Steven


From rob.cliffe at btinternet.com  Thu Nov 10 15:31:04 2011
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Thu, 10 Nov 2011 14:31:04 +0000
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBDEB9.2010904@pearwood.info>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>	<4EBBD192.3020901@pearwood.info>	<4EBBD33D.7080908@btinternet.com>
	<4EBBDEB9.2010904@pearwood.info>
Message-ID: <4EBBE028.3070304@btinternet.com>

Bear in mind new and inexperienced users, or perhaps someone who doesn't 
know Python at all but is trying to figure out what a Python program does.

     try:
         ...
     else:
         ...

is not at all intuitive.
If we want to allow that syntax, it would be better if 'else' were 
replaced by something more meaningful like 'ifnoexception'.  Maybe 
someone can think of something meaningful but more concise.


On 10/11/2011 14:24, Steven D'Aprano wrote:
> Rob Cliffe wrote:
>
>> "except: raise" makes it explicit that an exception is to be 
>> propogated.  Far from being pointless it makes the code much more 
>> readable.  "Explicit is better than implicit."
>
> Look, I love the Zen as much as the next guy, but sometimes they get 
> used as thought-terminating cliches. Catching an exception only to 
> unconditionally raise it again unchanged is a waste of everybody's 
> time: don't catch exceptions you have no intention of dealing with.
>
> Python has allowed try...finally without an except clause since 
> version 1.5, if not earlier.
>
> Virtually every line of Python code may implicitly raise an exception. 
> Despite the Zen, we don't go around writing
>
> try:
>     n = len(x)
> except:
>     raise
>
> just to satisfy the urge to make it explicit that any exception that 
> occurs will be propagated. We don't need an explicit reminder that any 
> exceptions will be propagated because, absent an except clause to 
> explicitly silence it, that's what exceptions do: they propagate.
>
> Errors should never pass silently.
> Unless explicitly silenced.
>
> (see, I can quote the Zen too).
>
>


From arnodel at gmail.com  Thu Nov 10 15:43:18 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Thu, 10 Nov 2011 14:43:18 +0000
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBE028.3070304@btinternet.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<4EBBDEB9.2010904@pearwood.info> <4EBBE028.3070304@btinternet.com>
Message-ID: <CAJ6cK1bPmsNTBZ6Qsp53naLfHL7EKs0_Q5mtojo-WbNQ495UMw@mail.gmail.com>

On 10 November 2011 14:31, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
> Bear in mind new and inexperienced users, or perhaps someone who doesn't
> know Python at all but is trying to figure out what a Python program does.
>
> ? ?try:
> ? ? ? ?...
> ? ?else:
> ? ? ? ?...
>
> is not at all intuitive.
> If we want to allow that syntax, it would be better if 'else' were replaced
> by something more meaningful like 'ifnoexception'. ?Maybe someone can think
> of something meaningful but more concise.

"noexcept".  However, I think "else" is fine.  It's similar to how
it's used in loops.  We have:

- if / else
- for / break / else
- while / break / else
- try / except / else

It makes sense to me.  Also note that "else" is allowed after loops
even if they have no "break".  So why not allow "else" after a try
without an except?

-- 
Arnaud


From guido at python.org  Thu Nov 10 16:53:25 2011
From: guido at python.org (Guido van Rossum)
Date: Thu, 10 Nov 2011 07:53:25 -0800
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBD33D.7080908@btinternet.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
Message-ID: <CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>

On Thu, Nov 10, 2011 at 5:35 AM, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
> "except: raise" makes it explicit that an exception is to be propogated.
> ?Far from being pointless it makes the code much more readable. ?"Explicit
> is better than implicit."

+1. Nobody's going to understand try...else... on first reading.

-- 
--Guido van Rossum (python.org/~guido)


From python at mrabarnett.plus.com  Thu Nov 10 19:00:12 2011
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 10 Nov 2011 18:00:12 +0000
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAJ6cK1bPmsNTBZ6Qsp53naLfHL7EKs0_Q5mtojo-WbNQ495UMw@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<4EBBDEB9.2010904@pearwood.info> <4EBBE028.3070304@btinternet.com>
	<CAJ6cK1bPmsNTBZ6Qsp53naLfHL7EKs0_Q5mtojo-WbNQ495UMw@mail.gmail.com>
Message-ID: <4EBC112C.1030604@mrabarnett.plus.com>

On 10/11/2011 14:43, Arnaud Delobelle wrote:
> On 10 November 2011 14:31, Rob Cliffe<rob.cliffe at btinternet.com>  wrote:
>> Bear in mind new and inexperienced users, or perhaps someone who doesn't
>> know Python at all but is trying to figure out what a Python program does.
>>
>>     try:
>>         ...
>>     else:
>>         ...
>>
>> is not at all intuitive.
>> If we want to allow that syntax, it would be better if 'else' were replaced
>> by something more meaningful like 'ifnoexception'.  Maybe someone can think
>> of something meaningful but more concise.
>
> "noexcept".  However, I think "else" is fine.  It's similar to how
> it's used in loops.  We have:
>
> - if / else
> - for / break / else
> - while / break / else
> - try / except / else
>
> It makes sense to me.  Also note that "else" is allowed after loops
> even if they have no "break".  So why not allow "else" after a try
> without an except?
>
The case of 'else' with the for-loops and while-loops is slightly
different because the body of the loop is repeated and you don't want
the body of the 'else' suite to be repeated.

With the 'try' statement that's not a problem.

-1


From ubershmekel at gmail.com  Thu Nov 10 20:00:54 2011
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Thu, 10 Nov 2011 21:00:54 +0200
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBD192.3020901@pearwood.info>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info>
Message-ID: <CANSw7Kxh_1cCf0E584V7OebwbuFKXF-=fRouqKYovx+P6VOXVw@mail.gmail.com>

On Thu, Nov 10, 2011 at 3:28 PM, Steven D'Aprano <steve at pearwood.info>wrote:

> Yuval Greenfield wrote:
>
>  "Try...else..." doesn't make any sense without the except jn my eyes. The
>> "else" has to conjugate something.
>>
>
>
> We already have while...else and for...else blocks in Python. The
> principle is the same:
>
>
> while condition():
>    ...
> else:
>    # Run the else block unless we exit the while block with a break.
>    ...
>
>
I'd rename or remove for/else and while/else from the language. Guido has
said "we probably should not do more of these".

http://mail.python.org/pipermail/python-ideas/2009-October/006083.html

Though these constructs are technically useful, even experts who read the
documentation misunderstand or misuse them. People who are new to the
language will be even worse.

--Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111110/a6fc4386/attachment.html>

From ethan at stoneleaf.us  Thu Nov 10 20:05:09 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 10 Nov 2011 11:05:09 -0800
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBD192.3020901@pearwood.info>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info>
Message-ID: <4EBC2065.9040900@stoneleaf.us>

Steven D'Aprano wrote:
> Yuval Greenfield wrote:
> 
>> "Try...else..." doesn't make any sense without the except jn my eyes. The
>> "else" has to conjugate something.
> 
> 
> We already have while...else and for...else blocks in Python. The 
> principle is the same:
> 
> 
> while condition():
>     ...
> else:
>     # Run the else block unless we exit the while block with a break.
>     ...
> 
> 
> try:
>     ...
> else:
>     # Run the else block unless we exit the try block with an exception.
>     ...
> finally:
>     ...

-1 to skipping the except

while/else and for/else are more along the lines of 'when condition is 
False, run the else suite -- how would you have a false condition for 
the try?  You don't, you have a false condition for the except.

~Ethan~


From tjreedy at udel.edu  Thu Nov 10 20:55:34 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 10 Nov 2011 14:55:34 -0500
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBD192.3020901@pearwood.info>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info>
Message-ID: <j9ha7q$s02$1@dough.gmane.org>

On 11/10/2011 8:28 AM, Steven D'Aprano wrote:

> try:
> ...
> else:
> # Run the else block unless we exit the try block with an exception.
> ...

As Dennis Kaarsemaker already pointed out in his response to the OP, the 
'else:' is not needed and this is the same (unless we are missing 
something) as

try:
   ...
   ...

In other words, *every* statement in a try block after the first is 
preceded by an implicit 'run this unless we already exited with an 
exception'.

-- 
Terry Jan Reedy



From p.f.moore at gmail.com  Thu Nov 10 21:13:31 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 10 Nov 2011 20:13:31 +0000
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <j9ha7q$s02$1@dough.gmane.org>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <j9ha7q$s02$1@dough.gmane.org>
Message-ID: <CACac1F84MxDm+DYFLQA65aQZy6_N+9yRDjZh3ERs1S7B5CE54Q@mail.gmail.com>

On 10 November 2011 19:55, Terry Reedy <tjreedy at udel.edu> wrote:
> On 11/10/2011 8:28 AM, Steven D'Aprano wrote:
>
>> try:
>> ...
>> else:
>> # Run the else block unless we exit the try block with an exception.
>> ...
>
> As Dennis Kaarsemaker already pointed out in his response to the OP, the
> 'else:' is not needed and this is the same (unless we are missing something)
> as
>
> try:
> ?...
> ?...
>
> In other words, *every* statement in a try block after the first is preceded
> by an implicit 'run this unless we already exited with an exception'.

Ah! I noted the post by Dennis Kaarsemaker pointing out that the
original code could be rewritten without needing the dummy except or
the else, but hadn't spotted that the argument generalised to *all*
uses of an else without an except.

Paul.


From digitalxero at gmail.com  Thu Nov 10 21:14:05 2011
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Thu, 10 Nov 2011 15:14:05 -0500
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
Message-ID: <CAMPUAFO+SmGOzxs=fzwoTfbqs8Ut0=OJesAePiBk7t26-80vmA@mail.gmail.com>

On Thu, Nov 10, 2011 at 10:53 AM, Guido van Rossum <guido at python.org> wrote:
> On Thu, Nov 10, 2011 at 5:35 AM, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
>> "except: raise" makes it explicit that an exception is to be propogated.
>> ?Far from being pointless it makes the code much more readable. ?"Explicit
>> is better than implicit."
>
> +1. Nobody's going to understand try...else... on first reading.
>
> --
> --Guido van Rossum (python.org/~guido)


I agree, try...else... when reading it would be try this block of
code, else if it fails do this block of code


From anacrolix at gmail.com  Thu Nov 10 22:38:46 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Fri, 11 Nov 2011 08:38:46 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBD33D.7080908@btinternet.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
Message-ID: <CAB4yi1OJO-BzvEvpZnCx+pGm+OfKHLYvVw-1V9u4Hx4qAqQ7PQ@mail.gmail.com>

"Explicit is better than implicit." does not apply here at all. It's
already known that an uncaught exception will not be caught and the
rule was not applied there.

except:
   raise

is as redundant as:

for stuff in things:
    do_the_things()
else:
    pass

On Fri, Nov 11, 2011 at 12:35 AM, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
>
>
> On 10/11/2011 13:28, Steven D'Aprano wrote:
>>
>> Yuval Greenfield wrote:
>>
>>> "Try...else..." doesn't make any sense without the except jn my eyes. The
>>> "else" has to conjugate something.
>>
>>
>> We already have while...else and for...else blocks in Python. The
>> principle is the same:
>>
>>
>> while condition():
>> ? ?...
>> else:
>> ? ?# Run the else block unless we exit the while block with a break.
>> ? ?...
>>
>>
>> try:
>> ? ?...
>> else:
>> ? ?# Run the else block unless we exit the try block with an exception.
>> ? ?...
>> finally:
>> ? ?...
>>
>>
>> This currently doesn't work without a pointless "except: raise" clause to
>> satisfy the compiler. I try to avoid writing pointless code.
>>
>>
>>
> "except: raise" makes it explicit that an exception is to be propogated.
> ?Far from being pointless it makes the code much more readable. ?"Explicit
> is better than implicit."
> Rob Cliffe
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From tjreedy at udel.edu  Thu Nov 10 23:10:24 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 10 Nov 2011 17:10:24 -0500
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAMPUAFO+SmGOzxs=fzwoTfbqs8Ut0=OJesAePiBk7t26-80vmA@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<CAMPUAFO+SmGOzxs=fzwoTfbqs8Ut0=OJesAePiBk7t26-80vmA@mail.gmail.com>
Message-ID: <j9hi4k$q0v$1@dough.gmane.org>

> On Thu, Nov 10, 2011 at 10:53 AM, Guido van Rossum<guido at python.org>  wrote:
>> On Thu, Nov 10, 2011 at 5:35 AM, Rob Cliffe<rob.cliffe at btinternet.com>  wrote:
>>> "except: raise" makes it explicit that an exception is to be propogated.
>>>   Far from being pointless it makes the code much more readable.  "Explicit
>>> is better than implicit."
>>
>> +1. Nobody's going to understand try...else... on first reading.

Perhaps because an else: in this context, without being subordinate to 
anything, would be irrelevant, equivalent to 'pass' or, more exactly, 
'if True:'.

Else is always subordinate to some conditional clause$. The 'if' clause 
for try statements is the first except clause, not the try header: 
'except x:' means something like 'if raised(x):' (since the 
corresponding try setup statement). The else clause is actually an elif 
clause: 'else:' means 'elif nothing_raised():'.

To illustrate, the C equivalent of

try:
   a = b/c
except ZeroDivisionError:
   print('bah')
else:
   print('hooray')

would be (minus the punctuation that I have forgotten) something like

errno = 0  /* setup error detection mechanism, just as try: does */
a = b/c
if errno == EZERODIV  /* or whatever it is */
   printf('bah')
elif errno == 0  /* ie, no error since setup */
   printf('hooray')

  I suspect Python *could* have been defined to require 'except 
NOEXCEPTION:' instead of 'else:' by initializing an internal exception 
variable (equivalent to C's errno) with NOEXCEPTION. Howver, 'else:' is 
definitely nicer (just as normal 'else:' is equivalent to and nicer than 
'elif True:').


$ A for loop is equivalent to a while loop with condition z"iterator 
(initially iter(iterable)) is not empty". Break and continue have the 
same meaning as in while loops.

Any while loop, such as

while c:
   x
else:
   y

is equivalent to an if statement with goto. In 
'Python-with-label-and-goto' the above would be the same as

label start_loop
if c:
   x
   goto start_loop
else:
   y

So: in Python (as elsewhere), 'else' clauses always follow one or more 
headers that specify conditional execution and only trigger when the 
previous condition or conditions are false. Their use in various 
statement types is consistent and logically correct.

-- 
Terry Jan Reedy



From anacrolix at gmail.com  Thu Nov 10 23:21:45 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Fri, 11 Nov 2011 09:21:45 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <j9hi4k$q0v$1@dough.gmane.org>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<CAMPUAFO+SmGOzxs=fzwoTfbqs8Ut0=OJesAePiBk7t26-80vmA@mail.gmail.com>
	<j9hi4k$q0v$1@dough.gmane.org>
Message-ID: <CAB4yi1OSBr69n_gU5TfbrOPG-XgYf0wVK0D+SyPfwmdQRQMnHQ@mail.gmail.com>

Your examples of else with loops are missing the extra behaviour on break?
This is where the real value is in this usage.
On Nov 11, 2011 9:11 AM, "Terry Reedy" <tjreedy at udel.edu> wrote:

> On Thu, Nov 10, 2011 at 10:53 AM, Guido van Rossum<guido at python.org>
>>  wrote:
>>
>>> On Thu, Nov 10, 2011 at 5:35 AM, Rob Cliffe<rob.cliffe at btinternet.**com<rob.cliffe at btinternet.com>>
>>>  wrote:
>>>
>>>> "except: raise" makes it explicit that an exception is to be propogated.
>>>>  Far from being pointless it makes the code much more readable.
>>>>  "Explicit
>>>> is better than implicit."
>>>>
>>>
>>> +1. Nobody's going to understand try...else... on first reading.
>>>
>>
> Perhaps because an else: in this context, without being subordinate to
> anything, would be irrelevant, equivalent to 'pass' or, more exactly, 'if
> True:'.
>
> Else is always subordinate to some conditional clause$. The 'if' clause
> for try statements is the first except clause, not the try header: 'except
> x:' means something like 'if raised(x):' (since the corresponding try setup
> statement). The else clause is actually an elif clause: 'else:' means 'elif
> nothing_raised():'.
>
> To illustrate, the C equivalent of
>
> try:
>  a = b/c
> except ZeroDivisionError:
>  print('bah')
> else:
>  print('hooray')
>
> would be (minus the punctuation that I have forgotten) something like
>
> errno = 0  /* setup error detection mechanism, just as try: does */
> a = b/c
> if errno == EZERODIV  /* or whatever it is */
>  printf('bah')
> elif errno == 0  /* ie, no error since setup */
>  printf('hooray')
>
>  I suspect Python *could* have been defined to require 'except
> NOEXCEPTION:' instead of 'else:' by initializing an internal exception
> variable (equivalent to C's errno) with NOEXCEPTION. Howver, 'else:' is
> definitely nicer (just as normal 'else:' is equivalent to and nicer than
> 'elif True:').
>
>
> $ A for loop is equivalent to a while loop with condition z"iterator
> (initially iter(iterable)) is not empty". Break and continue have the same
> meaning as in while loops.
>
> Any while loop, such as
>
> while c:
>  x
> else:
>  y
>
> is equivalent to an if statement with goto. In
> 'Python-with-label-and-goto' the above would be the same as
>
> label start_loop
> if c:
>  x
>  goto start_loop
> else:
>  y
>
> So: in Python (as elsewhere), 'else' clauses always follow one or more
> headers that specify conditional execution and only trigger when the
> previous condition or conditions are false. Their use in various statement
> types is consistent and logically correct.
>
> --
> Terry Jan Reedy
>
> ______________________________**_________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111111/0f3e48f8/attachment.html>

From tjreedy at udel.edu  Fri Nov 11 00:26:47 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 10 Nov 2011 18:26:47 -0500
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAB4yi1OSBr69n_gU5TfbrOPG-XgYf0wVK0D+SyPfwmdQRQMnHQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<CAMPUAFO+SmGOzxs=fzwoTfbqs8Ut0=OJesAePiBk7t26-80vmA@mail.gmail.com>
	<j9hi4k$q0v$1@dough.gmane.org>
	<CAB4yi1OSBr69n_gU5TfbrOPG-XgYf0wVK0D+SyPfwmdQRQMnHQ@mail.gmail.com>
Message-ID: <j9hmjr$p16$1@dough.gmane.org>

On 11/10/2011 5:21 PM, Matt Joiner wrote:
> Your examples of else with loops are missing the extra behaviour on
> break?

I omitted 'break' as not relevant to the particular points I was aiming 
at. To add it in:

while c:
   x
   if d: break
else:
   y

in PythonGTL could be

label start_loop
if c:
   x
   if d: goto after_loop
   goto start_loop
else:
   y
label after_loop

> This is where the real value is in this usage.

Yes, 'break' is the main motivation for 'while...else'.
Return and raise (explicit or otherwise) also break the loop. But since 
they raise stylistic preference issues, I won't go into them here.

-- 
Terry Jan Reedy



From steve at pearwood.info  Fri Nov 11 00:43:12 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 11 Nov 2011 10:43:12 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>	<4EBBD192.3020901@pearwood.info>
	<4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
Message-ID: <4EBC6190.6060104@pearwood.info>

Guido van Rossum wrote:
> On Thu, Nov 10, 2011 at 5:35 AM, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
>> "except: raise" makes it explicit that an exception is to be propogated.
>>  Far from being pointless it makes the code much more readable.  "Explicit
>> is better than implicit."
> 
> +1. Nobody's going to understand try...else... on first reading.


Nobody's going to understand try...finally on first reading either, or 
for...else. And I can tell you, the first time I saw Python code, I had 
no idea how to translate "for x in something" into idioms I understood. 
The only for loops I had ever seen before then had been "for i = 1 to 
100" Pascal/C style loops, and I was completely mystified.

It seems somewhat bizarre to me that in another thread we are talking 
about about adding "cofunctions" to a language that already has threads, 
generators, coroutines and metaclasses, and then on the other hand we're 
worried that users won't be able to generalise try...else...finally from 
try...except...else...finally.

However, Terry's explanation of Dennis' observation that any 
hypothetical try...else...finally block can be re-written as 
try...finally satisfies me. That's a good enough reason to keep the 
status quo.

Downgrading my vote from +1 to +0.



-- 
Steven


From anacrolix at gmail.com  Fri Nov 11 03:48:07 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Fri, 11 Nov 2011 13:48:07 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <j9hmjr$p16$1@dough.gmane.org>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<CAMPUAFO+SmGOzxs=fzwoTfbqs8Ut0=OJesAePiBk7t26-80vmA@mail.gmail.com>
	<j9hi4k$q0v$1@dough.gmane.org>
	<CAB4yi1OSBr69n_gU5TfbrOPG-XgYf0wVK0D+SyPfwmdQRQMnHQ@mail.gmail.com>
	<j9hmjr$p16$1@dough.gmane.org>
Message-ID: <CAB4yi1ORxvAcVarDU6vxE++y5G0HEu62hDWwweFZaaoiqH0L_g@mail.gmail.com>

Thanks for clarifying this. If anything, as mentioned earlier, the
meaning of else isn't entirely obvious on the loops, nor is it obvious
on try/except/else/finally clauses. The first time I encountered
for/else I was dumbfounded, then I realised how handy it was. I
wouldn't want to see these constructs removed. They remove the need
for really nasty patterns, I'm sure this was discussed when they were
added.

On Fri, Nov 11, 2011 at 10:26 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 11/10/2011 5:21 PM, Matt Joiner wrote:
>>
>> Your examples of else with loops are missing the extra behaviour on
>> break?
>
> I omitted 'break' as not relevant to the particular points I was aiming at.
> To add it in:
>
> while c:
> ?x
> ?if d: break
> else:
> ?y
>
> in PythonGTL could be
>
> label start_loop
> if c:
> ?x
> ?if d: goto after_loop
> ?goto start_loop
> else:
> ?y
> label after_loop
>
>> This is where the real value is in this usage.
>
> Yes, 'break' is the main motivation for 'while...else'.
> Return and raise (explicit or otherwise) also break the loop. But since they
> raise stylistic preference issues, I won't go into them here.
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From ncoghlan at gmail.com  Fri Nov 11 04:41:29 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 11 Nov 2011 13:41:29 +1000
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBC6190.6060104@pearwood.info>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<4EBC6190.6060104@pearwood.info>
Message-ID: <CADiSq7dVDCmUzdwEJfz1WqOtu=DnpRY1EBfs=xO-8kgmR_jwnA@mail.gmail.com>

On Fri, Nov 11, 2011 at 9:43 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> However, Terry's explanation of Dennis' observation that any hypothetical
> try...else...finally block can be re-written as try...finally satisfies me.
> That's a good enough reason to keep the status quo.
>
> Downgrading my vote from +1 to +0.

A full -1 from me, for the reason already given: it's completely
redundant. The "else:" clause on try statements exists specifically to
cover the following situation:

1. You have code in a try block that may trigger associated exception handlers
2. One or more of those exception handlers may suppress the exception,
allowing execution to resume after the try statement
3. You have some code that you want to run *only* if the try block
doesn't throw an exception (i.e. if an exception is thrown and
suppressed, you don't want to run this section of the code)

If none of your exception handlers can suppress the exception and
there's either no "finally:" close or it's OK if the additional code
runs after the cleanup code, then "else:" is not really needed - you
can just write the additional code after the whole try statement.

If there are no exception handlers at all, then either don't use a try
statement at all, or use a simple try-finally statement if you need
some guaranteed cleanup.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From guido at python.org  Fri Nov 11 04:50:45 2011
From: guido at python.org (Guido van Rossum)
Date: Thu, 10 Nov 2011 19:50:45 -0800
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CADiSq7dVDCmUzdwEJfz1WqOtu=DnpRY1EBfs=xO-8kgmR_jwnA@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<4EBC6190.6060104@pearwood.info>
	<CADiSq7dVDCmUzdwEJfz1WqOtu=DnpRY1EBfs=xO-8kgmR_jwnA@mail.gmail.com>
Message-ID: <CAP7+vJ+WijVYrzF215JAwQ5sSfMmLONfu1h9f4_SGQpjoxwyuQ@mail.gmail.com>

On Thu, Nov 10, 2011 at 7:41 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Fri, Nov 11, 2011 at 9:43 AM, Steven D'Aprano <steve at pearwood.info> wrote:
>> However, Terry's explanation of Dennis' observation that any hypothetical
>> try...else...finally block can be re-written as try...finally satisfies me.
>> That's a good enough reason to keep the status quo.
>>
>> Downgrading my vote from +1 to +0.
>
> A full -1 from me, for the reason already given: it's completely
> redundant. The "else:" clause on try statements exists specifically to
> cover the following situation:
>
> 1. You have code in a try block that may trigger associated exception handlers
> 2. One or more of those exception handlers may suppress the exception,
> allowing execution to resume after the try statement
> 3. You have some code that you want to run *only* if the try block
> doesn't throw an exception (i.e. if an exception is thrown and
> suppressed, you don't want to run this section of the code)

4. You don't want exceptions is the else-block to be handled by any of
the handlers. (Otherwise you could just put the else-block's code at
the end of the try-block.)

> If none of your exception handlers can suppress the exception and
> there's either no "finally:" close or it's OK if the additional code
> runs after the cleanup code, then "else:" is not really needed - you
> can just write the additional code after the whole try statement.
>
> If there are no exception handlers at all, then either don't use a try
> statement at all, or use a simple try-finally statement if you need
> some guaranteed cleanup.

-- 
--Guido van Rossum (python.org/~guido)


From ron3200 at gmail.com  Fri Nov 11 23:38:52 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Fri, 11 Nov 2011 16:38:52 -0600
Subject: [Python-ideas] Throwing exceptions from inside coroutines.
Message-ID: <1321051132.21755.36.camel@Gutsy>


Idea...  Exceptions should be thrown out of coroutines rather than
raised inside them and allowed to propagate out.

There are several benefits.

* It doesn't stop or leave the coroutine in a state that can't be
continued.

* It maintains the boundary between coroutines and other code in such a
way that they may be replaced or wrapped in threads or processes easier.

* It creates a new way to communicate to outside framework which
simplifies creating schedulers.

* It would work with yield-from expressions to almost completely
eliminate any support framework like trampolines runners. (and the
cothread.py module here.)


My fist thoughts on this was to allow a resumable exception type, but
after thinking about it more, I don't think it would be a good idea.

Rather than try to explain this in words, here is a version of Nicks
example cothread.py module with a throws() function that can be used to
throw out an exception in a clean way.  This isn't meant to be included
in the library, but is a way to experiment with the API used to write
coroutines as Nick suggested.

Also here are two modules from Gregs examples that shows some context in
which these features are used.

Cheers,
   Ron

-------------- next part --------------
A non-text attachment was scrubbed...
Name: cothread.py
Type: text/x-python
Size: 3046 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111111/9db6714a/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sched_test.py
Type: text/x-python
Size: 1774 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111111/9db6714a/attachment-0001.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: philosophers.py
Type: text/x-python
Size: 3426 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111111/9db6714a/attachment-0002.py>

From wuwei23 at gmail.com  Mon Nov 14 03:36:59 2011
From: wuwei23 at gmail.com (alex23)
Date: Sun, 13 Nov 2011 18:36:59 -0800 (PST)
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
Message-ID: <54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>

On Nov 11, 1:53?am, Guido van Rossum <gu... at python.org> wrote:
> +1. Nobody's going to understand try...else... on first reading.

On the other hand, no one properly understands for...else... when
first encountering it either.


From g.rodola at gmail.com  Mon Nov 14 15:25:05 2011
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Mon, 14 Nov 2011 15:25:05 +0100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
Message-ID: <CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>

2011/11/14 alex23 <wuwei23 at gmail.com>:
> On Nov 11, 1:53?am, Guido van Rossum <gu... at python.org> wrote:
>> +1. Nobody's going to understand try...else... on first reading.
>
> On the other hand, no one properly understands for...else... when
> first encountering it either.

I see no value in adding another hardly understandable construct.
Plus, in my mind try/else appears to be logically different than
for/else (which I find very useful).
-1

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/


From anacrolix at gmail.com  Mon Nov 14 20:34:23 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Tue, 15 Nov 2011 06:34:23 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
	<CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>
Message-ID: <CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>

To be fair it's not a new construct, it's loosening the constraints on
an existing one.

else has the same use as in the for and while compound statements:
It's executed if the first block completes "successfully". An
exception being thrown in a try block, cancels execution of the "else"
block, the same way as "break" in the for and while statements.

I have a problem with the idea now in that:

try:
    a
else:
    b

is the same as:

try:
    a
    b
else:
    pass

and even:

try:
    pass
else:
    a
    b

in every possible interpretation. This isn't possible with if, for,
while, try/except, and try/finally compound statements, all of which
alter the meaning in *some* way.

So I maintain that the else block could be made allowable even without
an except block, but it would be purely for convenience. I don't think
it's worth it anymore.

Cheers all for consideration.

On Tue, Nov 15, 2011 at 1:25 AM, Giampaolo Rodol? <g.rodola at gmail.com> wrote:
> 2011/11/14 alex23 <wuwei23 at gmail.com>:
>> On Nov 11, 1:53?am, Guido van Rossum <gu... at python.org> wrote:
>>> +1. Nobody's going to understand try...else... on first reading.
>>
>> On the other hand, no one properly understands for...else... when
>> first encountering it either.
>
> I see no value in adding another hardly understandable construct.
> Plus, in my mind try/else appears to be logically different than
> for/else (which I find very useful).
> -1
>
> --- Giampaolo
> http://code.google.com/p/pyftpdlib/
> http://code.google.com/p/psutil/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From ckaynor at zindagigames.com  Mon Nov 14 21:04:30 2011
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Mon, 14 Nov 2011 12:04:30 -0800
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
	<CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>
	<CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>
Message-ID: <CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>

On Mon, Nov 14, 2011 at 11:34 AM, Matt Joiner <anacrolix at gmail.com> wrote:
> To be fair it's not a new construct, it's loosening the constraints on
> an existing one.
>
> else has the same use as in the for and while compound statements:
> It's executed if the first block completes "successfully". An
> exception being thrown in a try block, cancels execution of the "else"
> block, the same way as "break" in the for and while statements.
>
> I have a problem with the idea now in that:
>
> try:
> ? ?a
> else:
> ? ?b
>
> is the same as:
>
> try:
> ? ?a
> ? ?b
> else:
> ? ?pass
>
> and even:
>
> try:
> ? ?pass
> else:
> ? ?a
> ? ?b
>
> in every possible interpretation. This isn't possible with if, for,
> while, try/except, and try/finally compound statements, all of which
> alter the meaning in *some* way.

If you add finally, the else clause still maintains the same semantics
in a try statement - only except statements modify the equivalence of
else:
try:
 a
else:
 b
finally:
 c

is the same as:
try:
 a
 b
else:
 pass
finally:
 c

and

try:
 pass
else:
 a
 b
finally:
 c

What changes with the finally is that you cannot move the "b"
statement out of the try completely. Without the finally clause, the
following are the same:

try:
 a
else:
 b

try:
 a
b

a
b




Similar behavior can be shown with for: else and while: else, if there
is no break (I'm only showing a while statement below, however it
converts exactly to a for statement):
while a:
 b
else:
 c

is the same as:
while a:
 b
c

so long as "b" does not include a break statement.



>
> So I maintain that the else block could be made allowable even without
> an except block, but it would be purely for convenience. I don't think
> it's worth it anymore.

While it is redundant to allow else without except, I would agree that
it should be allowed for the same reason that else is allowed on for
and while without a break. The alternative would be to make the later
illegal. Either seems reasonable, however the former maintains
backwards compatibility, while the later breaks it.

>
> Cheers all for consideration.


From ethan at stoneleaf.us  Mon Nov 14 21:23:02 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 14 Nov 2011 12:23:02 -0800
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>	<4EBBD192.3020901@pearwood.info>
	<4EBBD33D.7080908@btinternet.com>	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>	<CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>	<CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>
	<CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
Message-ID: <4EC178A6.4020002@stoneleaf.us>

Chris Kaynor wrote:
> While it is redundant to allow else without except, I would agree that
> it should be allowed for the same reason that else is allowed on for
> and while without a break. The alternative would be to make the later
> illegal. Either seems reasonable, however the former maintains
> backwards compatibility, while the later breaks it.

if / else: if the 'if' condition is False, run the 'else'

for / else: when the 'for' condition is False, run the 'else'

while / else: when the 'while' condition is False, run the 'else'

try /except / else: when the 'except' condition is False, run the 'else'

try / else: when the 'try' condition is False, run the 'else' ??

To be consistent, the 'else' in a try/else would basically be an 'except 
Exception' -- it completely reverses the meaning of the 'else' in a 
try/except/else block.

~Ethan~


From anacrolix at gmail.com  Mon Nov 14 22:21:23 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Tue, 15 Nov 2011 08:21:23 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
	<CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>
	<CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>
	<CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
Message-ID: <CAB4yi1NHi-Jcdq3UVt6Tc6TG85Cp-9dcNtf6a=nq7AZq4XzkFw@mail.gmail.com>

> While it is redundant to allow else without except, I would agree that

yeah that nails it.

> it should be allowed for the same reason that else is allowed on for> and while without a break. The alternative would be to make the later

that's a really good point, i hadn't considered this. similar things
also occur when you "drop" out of some construct because you can
predict the control flow. for example when certain blocks are
guaranteed to throw exceptions, it becomes unnecessary to put
alternative behaviour in an "else" block.

try:
    a
except:
    b
    raise
c

In the above, c does not get put in an else because it's redundant.

The same applies here:

if a:
    raise b
c

> illegal. Either seems reasonable, however the former maintains> backwards compatibility, while the later breaks it.

Yes for clarification you're saying:

while a:
    b
else:
    c

Is redundant if b does not break.

I think this puts the idea of else without except into the "why not?"
camp for me.
On Tue, Nov 15, 2011 at 7:04 AM, Chris Kaynor <ckaynor at zindagigames.com> wrote:
> On Mon, Nov 14, 2011 at 11:34 AM, Matt Joiner <anacrolix at gmail.com> wrote:
>> To be fair it's not a new construct, it's loosening the constraints on
>> an existing one.
>>
>> else has the same use as in the for and while compound statements:
>> It's executed if the first block completes "successfully". An
>> exception being thrown in a try block, cancels execution of the "else"
>> block, the same way as "break" in the for and while statements.
>>
>> I have a problem with the idea now in that:
>>
>> try:
>> ? ?a
>> else:
>> ? ?b
>>
>> is the same as:
>>
>> try:
>> ? ?a
>> ? ?b
>> else:
>> ? ?pass
>>
>> and even:
>>
>> try:
>> ? ?pass
>> else:
>> ? ?a
>> ? ?b
>>
>> in every possible interpretation. This isn't possible with if, for,
>> while, try/except, and try/finally compound statements, all of which
>> alter the meaning in *some* way.
>
> If you add finally, the else clause still maintains the same semantics
> in a try statement - only except statements modify the equivalence of
> else:
> try:
> ?a
> else:
> ?b
> finally:
> ?c
>
> is the same as:
> try:
> ?a
> ?b
> else:
> ?pass
> finally:
> ?c
>
> and
>
> try:
> ?pass
> else:
> ?a
> ?b
> finally:
> ?c
>
> What changes with the finally is that you cannot move the "b"
> statement out of the try completely. Without the finally clause, the
> following are the same:
>
> try:
> ?a
> else:
> ?b
>
> try:
> ?a
> b
>
> a
> b
>
>
>
>
> Similar behavior can be shown with for: else and while: else, if there
> is no break (I'm only showing a while statement below, however it
> converts exactly to a for statement):
> while a:
> ?b
> else:
> ?c
>
> is the same as:
> while a:
> ?b
> c
>
> so long as "b" does not include a break statement.
>
>
>
>>
>> So I maintain that the else block could be made allowable even without
>> an except block, but it would be purely for convenience. I don't think
>> it's worth it anymore.
>
> While it is redundant to allow else without except, I would agree that
> it should be allowed for the same reason that else is allowed on for
> and while without a break. The alternative would be to make the later
> illegal. Either seems reasonable, however the former maintains
> backwards compatibility, while the later breaks it.
>
>>
>> Cheers all for consideration.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From anacrolix at gmail.com  Mon Nov 14 22:24:06 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Tue, 15 Nov 2011 08:24:06 +1100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EC178A6.4020002@stoneleaf.us>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
	<CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>
	<CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>
	<CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
	<4EC178A6.4020002@stoneleaf.us>
Message-ID: <CAB4yi1PNqY_8ry4-2BgDzKQH7-79QOD8R0PqYEqkwoTG8rRZtg@mail.gmail.com>

well for all cases bar the if/else, "else" is used more like a "then".

for/then, while/then, try/except/then/finally

On Tue, Nov 15, 2011 at 7:23 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> Chris Kaynor wrote:
>>
>> While it is redundant to allow else without except, I would agree that
>> it should be allowed for the same reason that else is allowed on for
>> and while without a break. The alternative would be to make the later
>> illegal. Either seems reasonable, however the former maintains
>> backwards compatibility, while the later breaks it.
>
> if / else: if the 'if' condition is False, run the 'else'
>
> for / else: when the 'for' condition is False, run the 'else'
>
> while / else: when the 'while' condition is False, run the 'else'
>
> try /except / else: when the 'except' condition is False, run the 'else'
>
> try / else: when the 'try' condition is False, run the 'else' ??
>
> To be consistent, the 'else' in a try/else would basically be an 'except
> Exception' -- it completely reverses the meaning of the 'else' in a
> try/except/else block.
>
> ~Ethan~
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From grosser.meister.morti at gmx.net  Mon Nov 14 22:37:47 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Mon, 14 Nov 2011 22:37:47 +0100
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <4EBBD192.3020901@pearwood.info>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info>
Message-ID: <4EC18A2B.4010700@gmx.net>

On 11/10/2011 02:28 PM, Steven D'Aprano wrote:
>
> try:
> ...
> else:
> # Run the else block unless we exit the try block with an exception.
> ...
> finally:
> ...
>

To me this would intuitively read as:
Do the try block if you can, else (if you can't) do the else block.
And you can't if an exception is raised.

This is exactly the opposite of the intended meaning. The except is needed because the else refers 
to it. Without the except the else would refer to the try and one would expect behaviour as with an 
"except:" block.

Also when ever would you need an else block without an except block? Just append it to the try block 
and you get the same behaviour.

As for the not immediately intuitive meaning of the finally block: Yes, maybe "ensure" would have 
been better. But it's to late for that to change.

-1 from me

	-panzi


From ncoghlan at gmail.com  Mon Nov 14 23:38:40 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 15 Nov 2011 08:38:40 +1000
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
	<CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>
	<CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>
	<CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
Message-ID: <CADiSq7dZTSWZcX+X-Qb8hT9newVYR9g8J_q+iGveKyikqwLZ_A@mail.gmail.com>

On 11/15/11, Chris Kaynor <ckaynor at zindagigames.com> wrote:
> On Mon, Nov 14, 2011 at 11:34 AM, Matt Joiner <anacrolix at gmail.com> wrote:
>> So I maintain that the else block could be made allowable even without
>> an except block, but it would be purely for convenience. I don't think
>> it's worth it anymore.
>
> While it is redundant to allow else without except, I would agree that
> it should be allowed for the same reason that else is allowed on for
> and while without a break. The alternative would be to make the later
> illegal. Either seems reasonable, however the former maintains
> backwards compatibility, while the later breaks it.

This change is *not going to happen*. Guido said drop it. Every core
developer in the conversation said drop it. Please let the thread die
already.

The fact that 'else:' is allowed on loops without a break statement
(despite such a construct being pointless) is just a way to simplify
the language compilation process. While it obviously could be detected
and prevented if we really wanted to (since we basically do that to
allow 'yield' to alter the meaning of a function definition), reaching
inside a suite to detect that kind of thing is a pain and simply not
worth the hassle in this specific case. If it really bothers you,
propose a patch to pyflakes (etc) that issues a warning for this kind
of error (assuming they don't detect it already).

In no way, shape or form does the fact we allow a particular redundant
construct because detecting it is a pain to implement justify relaxing
the parsing rules to allow an equally redundant construct that all
compliant Python implementations *already* reject (and is much easier
to detect in the first place, since the required precursor is a peer
suite rather than a statement embedded inside such a suite).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


From solipsis at pitrou.net  Tue Nov 15 00:18:18 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 15 Nov 2011 00:18:18 +0100
Subject: [Python-ideas] try-else without except or finally
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
	<CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>
	<CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>
	<CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
	<CADiSq7dZTSWZcX+X-Qb8hT9newVYR9g8J_q+iGveKyikqwLZ_A@mail.gmail.com>
Message-ID: <20111115001818.639be7bd@pitrou.net>

On Tue, 15 Nov 2011 08:38:40 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 11/15/11, Chris Kaynor <ckaynor at zindagigames.com> wrote:
> > On Mon, Nov 14, 2011 at 11:34 AM, Matt Joiner <anacrolix at gmail.com> wrote:
> >> So I maintain that the else block could be made allowable even without
> >> an except block, but it would be purely for convenience. I don't think
> >> it's worth it anymore.
> >
> > While it is redundant to allow else without except, I would agree that
> > it should be allowed for the same reason that else is allowed on for
> > and while without a break. The alternative would be to make the later
> > illegal. Either seems reasonable, however the former maintains
> > backwards compatibility, while the later breaks it.
> 
> This change is *not going to happen*. Guido said drop it. Every core
> developer in the conversation said drop it. Please let the thread die
> already.

What Nick said!

Regards

Antoine.




From tjreedy at udel.edu  Tue Nov 15 01:53:05 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 14 Nov 2011 19:53:05 -0500
Subject: [Python-ideas] try-else without except or finally
In-Reply-To: <CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
References: <CAB4yi1Md8C-6KKA+DL3eX7OfvxG=VhHpXFQjz=33KhsgY65neQ@mail.gmail.com>
	<4EBBCBE2.7090409@pearwood.info>
	<CANSw7KzwJ3hpxmsUpR_z0nRo_oVnv3_zcRBe38prdkKL3JFxXw@mail.gmail.com>
	<4EBBD192.3020901@pearwood.info> <4EBBD33D.7080908@btinternet.com>
	<CAP7+vJ+u8CSdJ71inLSac9B_cF2QsoD1QH7h-_uS+jA_aGyQgQ@mail.gmail.com>
	<54a37430-5629-47ba-bd67-991e17b46cc2@w20g2000prc.googlegroups.com>
	<CAFYqXL8rJf9XXbCCBuZ+wQHUR61mEzEbYK1XToB+wuHA7BytbQ@mail.gmail.com>
	<CAB4yi1M8vo80h1vnFRrAOh-_T4ZJA=+YqVZZ1DZSmUy5Pz0x5w@mail.gmail.com>
	<CALvWhxt1Dj9nq-sdt_8BygtKN_wxJCb9pgxhQAP8sza2maW8aQ@mail.gmail.com>
Message-ID: <j9sd5k$og4$1@dough.gmane.org>

On 11/14/2011 3:04 PM, Chris Kaynor wrote:

> While it is redundant to allow else without except,

It is as redundant as allowing an 'else' without 'if', which is also not 
allowed. If fact, the exactly meaning of 'else' depends on what it is 
subordinate to -- 'if', 'while', 'for', or 'except'. So a bare 'else' 
would be ambiguous as to exact meaning, though the effect of 'pass' 
would be the same.

The construct analogous to if...elif...else is except...except...else. 
After the first 'except', 'except' is an abbreviation for 'else-except', 
just as 'elif' is an abbreviation for 'else-if'.

Try-else is *not* analogous to if-else. 'Try' is not a conditional. The 
next line *always* executes (successfully or not). 'Try' only means 
'prepare a context for special treatment of exceptions'. One special 
treatment is to execute code only if there is an exception (after it is 
caught). The other is to execute code even if there is an exception.

> I would agree that it should be allowed for the same reason
> that else is allowed on for and while without a break.

One is also allowed to write dubious code in numerous other ways:

if <some condition that can never be true>: body
if <some condition that must always be true>: body
while <condition>: break; body
while <condition>: body; break
a=1; a=2
...

So what? The fact that a language spec, especially one limited to a 
context-free LL(1) grammar, allows one to write dead or irrelevant code 
in several ways is *not* a reason to add another, especially one that is 
logically incoherent and *always* useless, not just sometimes.

The CPython  parser and compiler are written to run reasonably fast. 
Third-party code checkers raise warnings for things they can catch with 
more extended analysis.

Anyway, there is no analogy between 'try' and 'while' or 'for'. Both of 
the latter are 'if' statements augmented with a hidden goto to produce 
looping. The 'else' is subordinate to the 'if' part.  Both can be 
terminated early by 'break', which is only allowed in loops. None of 
this has anything to do with 'try'.

> The alternative would be to make the later illegal.

Try writing a Python-legal grammar that does that;-).

-- 
Terry Jan Reedy



From mikegraham at gmail.com  Wed Nov 16 14:30:50 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Wed, 16 Nov 2011 08:30:50 -0500
Subject: [Python-ideas] Renaming the "test" package
In-Reply-To: <CANSw7KxjTv6CX2rrY1Jc+s3GQRSfnLG0AgWNoM9kXLnQCA_3Cw@mail.gmail.com>
References: <CANSw7KxjTv6CX2rrY1Jc+s3GQRSfnLG0AgWNoM9kXLnQCA_3Cw@mail.gmail.com>
Message-ID: <CAEBZo3P4C-8nsPSLvd6ZQ=mAQAcLM6urqSRQ-JYPnMUqZBGW=A@mail.gmail.com>

On Sun, Nov 6, 2011 at 3:56 AM, Yuval Greenfield <ubershmekel at gmail.com> wrote:
> For python4, the "test" package should actually be named
> __pythontest__ or something similar. There are 2 million results for
> ""test.py"" on google.
>
> What do you think?
>
> --Yuval

The test package isn't really meant for normal users anyhow, so
shadowing "test" isn't a big deal. I see your point and don't disagree
with the concept, but don't think it's a big enough deal to do
anything about.

Mike


From ubershmekel at gmail.com  Wed Nov 16 16:55:21 2011
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 16 Nov 2011 17:55:21 +0200
Subject: [Python-ideas] Renaming the "test" package
In-Reply-To: <CAEBZo3P4C-8nsPSLvd6ZQ=mAQAcLM6urqSRQ-JYPnMUqZBGW=A@mail.gmail.com>
References: <CANSw7KxjTv6CX2rrY1Jc+s3GQRSfnLG0AgWNoM9kXLnQCA_3Cw@mail.gmail.com>
	<CAEBZo3P4C-8nsPSLvd6ZQ=mAQAcLM6urqSRQ-JYPnMUqZBGW=A@mail.gmail.com>
Message-ID: <CANSw7KzOL6OaLvyCBFMOC+Lxjd8Tj31m3qH0+X1_wOMb7kCn7Q@mail.gmail.com>

On Wed, Nov 16, 2011 at 3:30 PM, Mike Graham <mikegraham at gmail.com> wrote:

> On Sun, Nov 6, 2011 at 3:56 AM, Yuval Greenfield <ubershmekel at gmail.com>
> wrote:
> ...
> Mike
>


I should have explained.


A file of mine was named "test.py" and one day I got
some bizarre "AttributeError: 'module' object has no attribute..."
because my test.py was moved elsewhere. So instead of a failed import which
would have been more obvious I got that.

I agree there's no direct damage done by this "shadowing". Maybe the better
word is "ghosting" because it appears to give something though it doesn't.


--Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111116/afe228d1/attachment.html>

From solipsis at pitrou.net  Fri Nov 18 21:01:46 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 18 Nov 2011 21:01:46 +0100
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
Message-ID: <20111118210146.435b13f8@pitrou.net>

On Mon, 7 Nov 2011 10:34:24 -0500
Barry Warsaw <barry at python.org> wrote:
> On Oct 30, 2011, at 12:18 AM, Antoine Pitrou wrote:
> 
> >I would like to propose the following PEP for discussion and, if
> >possible, acceptance. I think the proposal shouldn't be too
> >controversial (I find it quite simple and straightforward myself :-)).
> 
> Nice PEP, and +1 for the concept (with Guido's preferred format).  However,
> "qname" is pretty obscure and I only guessed what the "q" stood for by reading
> the title of the PEP.
> 
> It seems to me that this attribute represents the dotted path from module
> globals to the object.  You have to be careful not to confuse this with a file
> system path, so something like __dotted_name__, __dotted_path__, or
> __full_name__ perhaps.  I don't much care, but I do think cryptic
> abbreviations should be avoided.

Based on the feedback received, I've finally changed the attribute name
to __qualname__.  Pronunciation issues notwithstanding ;-), it is a
reasonable compromise between explicitness and shortness/easy-of-typing.

Regards

Antoine.




From ncoghlan at gmail.com  Fri Nov 18 23:49:21 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 19 Nov 2011 08:49:21 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <20111118210146.435b13f8@pitrou.net>
References: <20111030001801.2f52ceb2@pitrou.net>
	<20111107103424.4c2e2ef1@resist.wooz.org>
	<20111118210146.435b13f8@pitrou.net>
Message-ID: <CADiSq7fweyDe3XwLod4xzSnZd05i_yeMx4Ac6jjsn_6kpkL5Vw@mail.gmail.com>

On Sat, Nov 19, 2011 at 6:01 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Based on the feedback received, I've finally changed the attribute name
> to __qualname__. ?Pronunciation issues notwithstanding ;-), it is a
> reasonable compromise between explicitness and shortness/easy-of-typing.

OK, I'll update PEP 395 accordingly (and given some discussions on
import-sig this week, the title of that PEP is probably going to
change to "Qualified names for modules")

I can't promise not to shorten it back to q-name if I ever have
occasion to say it out loud, though :)

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ron3200 at gmail.com  Sat Nov 19 07:24:39 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 19 Nov 2011 00:24:39 -0600
Subject: [Python-ideas] Exceptions thrown from generators.. patch.
Message-ID: <1321683879.22715.88.camel@Gutsy>


I was able to create a patch for testing this idea.  The hard part was
in getting to know cpython well enough to do it.  :-)


To get it to work, I made the following change in ceval.c so that the
main loop will accept a pointer rather than an int for the throwflag.
That allows an opcode to set it before it yields an exception.  ie... a
reverse throw.

PyObject *
PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) {
    return PyEval_EvalFrame_Ex(f, &throwflag);
}

PyObject *
PyEval_EvalFrame_Ex(PyFrameObject *f, int *throwflag)
{
   ...
        TARGET(YIELD_EXCEPT)        
            *throwflag = 1;
            retval = POP();
            f->f_stacktop = stack_pointer;
            why = WHY_YIELD;
            goto fast_yield;
   ...


The genobject gen_send_ex() function checks the throwflag value after a
send to see if it got a thrown out exception back.  (Rather than one
yielded out.)

A new keyword 'throws' was needed to go with the YIELD_EXCEPT opcode. I
didn't see anyway to do it with a function or method.  It should be
possible to set the exception in the ceval loop rather than yielding it
out, but I think that would be more complex. 

Because the exception isn't raised inside the generator code object, but
is yielded out first, the generator can be continued as if it was a
regular yielded value.  No magic required, and no fiddling with the
exception stack was needed. It doesn't effect unexpected exceptions, or
exceptions raised with raise.  Those will terminate a generator as
always.

Python 3.3.0a0 (qbase qtip tip yield_except:92ac2848438f+, Nov 18 2011,
21:59:57) 
[GCC 4.6.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def G():
...     yield 1
...     throws ValueError
...     yield 2
... 
>>> g = G()
>>> next(g)
1
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError
>>> next(g)
2
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration


The three main benefits of being able to do this are...

* To use switch like exception structures for flow control in schedulers
and coroutines.

* For consumer type coroutines to be able reject and re-request data in
a nice way without terminating.   ie.. the reverse of throwing in an
exception in, in order to change what a generator does.

* It creates alternative channels for data input and output by being
able to both throw exceptions in and out of generators.  Those signals
can carry objects in and out and not burden the fast yield data path
with testing for special wrapper objects.


Here's an example of it being used in a simple scheduler.
-----------

class Suspend(Exception): pass

def Person(name, count, mode):
    n = 0
    while n < count:
        if mode == 0:
            # The normal data path.
            yield name, count
        else:
            # Use an exception as an alternative data path.
            throws Suspend(name, count)
        n += 1
    # return
    raise StopIteration(name, n)

def main(data, mode):
    stack = [Person(*(args + (mode,))) for args in data]
    results = []
    while stack:
        done = []
        for ct in stack:
            try:
                print('yield', *next(ct))     # from yield
            except Suspend as exc:
                print('throws', *exc.args)    # from throws
            except StopIteration as exc:
                results.append(exc.args)
                continue
            done.append(ct)
        stack = done
    print(results)
    return results

if __name__ == "__main__":
    data = [("John", 2), ("Micheal", 3), ("Terry", 4)]
    results1 = main(data, 0)
    results2 = main(data, 1)
    assert(results1 == results2 == data)

-------------
The output looks like...

yield John 2
yield Micheal 3
yield Terry 4
yield John 2
yield Micheal 3
yield Terry 4
yield Micheal 3
yield Terry 4
yield Terry 4
[('John', 2), ('Micheal', 3), ('Terry', 4)]
throws John 2
throws Micheal 3
throws Terry 4
throws John 2
throws Micheal 3
throws Terry 4
throws Micheal 3
throws Terry 4
throws Terry 4
[('John', 2), ('Micheal', 3), ('Terry', 4)]


This shows that 'throws' works a lot like 'yield'.


Open issues:

* A better name than 'throws' might be good.

* Should it get the object sent in.

   <object> = throws <exception>

Or should it be ...

   throws <exception>

* What would be the best argument form.. Should it take the same
arguments as raise or just a single expression.


Python's test suite passes as this doesn't change anything that already
works.

I haven't tested it with the yield-from patch yet, but I think if it can
throw out exceptions in the same way yield-from yields out, that it will
make some things easier and nicer to do.

If anyone is interested, I can create a tracker item and put the patch
there where it can be improved further.

Cheers,
   Ron




From greg at krypto.org  Sat Nov 19 08:55:18 2011
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 18 Nov 2011 23:55:18 -0800
Subject: [Python-ideas] Exceptions thrown from generators.. patch.
In-Reply-To: <1321683879.22715.88.camel@Gutsy>
References: <1321683879.22715.88.camel@Gutsy>
Message-ID: <CAGE7PNJXjzg4Y4G8oH_xfzaC-Zyqcq-=qMFKKVarfzxOKqhp4g@mail.gmail.com>

On Fri, Nov 18, 2011 at 10:24 PM, Ron Adam <ron3200 at gmail.com> wrote:

>
> I was able to create a patch for testing this idea.  The hard part was
> in getting to know cpython well enough to do it.  :-)
>
>
> To get it to work, I made the following change in ceval.c so that the
> main loop will accept a pointer rather than an int for the throwflag.
> That allows an opcode to set it before it yields an exception.  ie... a
> reverse throw.
>
> PyObject *
> PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) {
>    return PyEval_EvalFrame_Ex(f, &throwflag);
> }
>
> PyObject *
> PyEval_EvalFrame_Ex(PyFrameObject *f, int *throwflag)
> {
>   ...
>        TARGET(YIELD_EXCEPT)
>            *throwflag = 1;
>            retval = POP();
>            f->f_stacktop = stack_pointer;
>            why = WHY_YIELD;
>            goto fast_yield;
>   ...
>
>
> The genobject gen_send_ex() function checks the throwflag value after a
> send to see if it got a thrown out exception back.  (Rather than one
> yielded out.)
>
> A new keyword 'throws' was needed to go with the YIELD_EXCEPT opcode. I
> didn't see anyway to do it with a function or method.  It should be
> possible to set the exception in the ceval loop rather than yielding it
> out, but I think that would be more complex.
>
> Because the exception isn't raised inside the generator code object, but
> is yielded out first, the generator can be continued as if it was a
> regular yielded value.  No magic required, and no fiddling with the
> exception stack was needed. It doesn't effect unexpected exceptions, or
> exceptions raised with raise.  Those will terminate a generator as
> always.
>
> Python 3.3.0a0 (qbase qtip tip yield_except:92ac2848438f+, Nov 18 2011,
> 21:59:57)
> [GCC 4.6.1] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> def G():
> ...     yield 1
> ...     throws ValueError
> ...     yield 2
> ...
> >>> g = G()
> >>> next(g)
> 1
> >>> next(g)
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> ValueError
> >>> next(g)
> 2
> >>> next(g)
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> StopIteration
>
>
> The three main benefits of being able to do this are...
>
> * To use switch like exception structures for flow control in schedulers
> and coroutines.
>
> * For consumer type coroutines to be able reject and re-request data in
> a nice way without terminating.   ie.. the reverse of throwing in an
> exception in, in order to change what a generator does.
>
> * It creates alternative channels for data input and output by being
> able to both throw exceptions in and out of generators.  Those signals
> can carry objects in and out and not burden the fast yield data path
> with testing for special wrapper objects.
>
>
> Here's an example of it being used in a simple scheduler.
> -----------
>
> class Suspend(Exception): pass
>
> def Person(name, count, mode):
>    n = 0
>    while n < count:
>        if mode == 0:
>            # The normal data path.
>            yield name, count
>        else:
>            # Use an exception as an alternative data path.
>            throws Suspend(name, count)
>        n += 1
>    # return
>    raise StopIteration(name, n)
>
> def main(data, mode):
>    stack = [Person(*(args + (mode,))) for args in data]
>    results = []
>    while stack:
>        done = []
>        for ct in stack:
>            try:
>                print('yield', *next(ct))     # from yield
>            except Suspend as exc:
>                print('throws', *exc.args)    # from throws
>            except StopIteration as exc:
>                results.append(exc.args)
>                continue
>            done.append(ct)
>        stack = done
>    print(results)
>    return results
>
> if __name__ == "__main__":
>    data = [("John", 2), ("Micheal", 3), ("Terry", 4)]
>    results1 = main(data, 0)
>    results2 = main(data, 1)
>    assert(results1 == results2 == data)
>
> -------------
> The output looks like...
>
> yield John 2
> yield Micheal 3
> yield Terry 4
> yield John 2
> yield Micheal 3
> yield Terry 4
> yield Micheal 3
> yield Terry 4
> yield Terry 4
> [('John', 2), ('Micheal', 3), ('Terry', 4)]
> throws John 2
> throws Micheal 3
> throws Terry 4
> throws John 2
> throws Micheal 3
> throws Terry 4
> throws Micheal 3
> throws Terry 4
> throws Terry 4
> [('John', 2), ('Micheal', 3), ('Terry', 4)]
>
>
> This shows that 'throws' works a lot like 'yield'.
>
>
neat!


>
> Open issues:
>
> * A better name than 'throws' might be good.
>

I don't like adding another keyword or confusing things by adding the
"throw" verb to a language that already firmly uses the verb "raise" when
speaking of exceptions.

A double word syntax might make sense here.  It'd be good to keep 'yeild'
in it to make it clear that this is offering the exception out in a
non-terminal manner.

  yield raise MyException(x,y,z)

or if you've caught an exception from something and want to pass that on to
the caller without killing the iteration the natural base form of that
would be:

  yield raise

to unset the existing exception and yeild it out instead, mirroring what a
bare raise within an except: clause does.


> * Should it get the object sent in.
>
>   <object> = throws <exception>
>
> Or should it be ...
>
>   throws <exception>
>
> * What would be the best argument form.. Should it take the same
> arguments as raise or just a single expression.
>
>
> Python's test suite passes as this doesn't change anything that already
> works.
>
> I haven't tested it with the yield-from patch yet, but I think if it can
> throw out exceptions in the same way yield-from yields out, that it will
> make some things easier and nicer to do.
>
> If anyone is interested, I can create a tracker item and put the patch
> there where it can be improved further.
>
> Cheers,
>   Ron
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111118/0c975a1d/attachment.html>

From ncoghlan at gmail.com  Sat Nov 19 09:25:17 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 19 Nov 2011 18:25:17 +1000
Subject: [Python-ideas] Exceptions thrown from generators.. patch.
In-Reply-To: <CAGE7PNJXjzg4Y4G8oH_xfzaC-Zyqcq-=qMFKKVarfzxOKqhp4g@mail.gmail.com>
References: <1321683879.22715.88.camel@Gutsy>
	<CAGE7PNJXjzg4Y4G8oH_xfzaC-Zyqcq-=qMFKKVarfzxOKqhp4g@mail.gmail.com>
Message-ID: <CADiSq7f55KgseYj-Xs1nT7RE5vOtqHc2pz-fvgB0igM4XMDyPg@mail.gmail.com>

On Sat, Nov 19, 2011 at 5:55 PM, Gregory P. Smith <greg at krypto.org> wrote:
> A double word syntax might make sense here. ?It'd be good to keep 'yeild' in
> it to make it clear that this is offering the exception out in a
> non-terminal manner.
> ? yield raise MyException(x,y,z)
> or if you've caught an exception from something and want to pass that on to
> the caller without killing the iteration the natural base form of that would
> be:
> ? yield raise
> to unset the existing exception and yeild it out instead, mirroring what a
> bare raise within an except: clause does.

Indeed, "yield raise" would be quite appropriate terminology for this
new channel of communication.

However, there's still a potential stack unwinding problem here. For
asymmetric coroutines, we need to be able to communicate an arbitrary
distance up the stack. This idea, as it stands, doesn't provide that -
like an ordinary yielded value, it can only yield control one level
out. So even if the innermost generator is left in a resumable state,
any *external* iterators are still going to be terminated.

It potentially becomes more useful in combination with 'yield from',
since yielded exceptions would be passed up the stack, the same as
yielded values.

So, while it's a neat trick and very cool that it could be implemented
with such a small change, I still don't see how it helps substantially
with the challenge of allowing *any* Python frame on a coroutine
stack, not just generator frames.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ron3200 at gmail.com  Sat Nov 19 18:48:59 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 19 Nov 2011 11:48:59 -0600
Subject: [Python-ideas] Exceptions thrown from generators.. patch.
In-Reply-To: <CADiSq7f55KgseYj-Xs1nT7RE5vOtqHc2pz-fvgB0igM4XMDyPg@mail.gmail.com>
References: <1321683879.22715.88.camel@Gutsy>
	<CAGE7PNJXjzg4Y4G8oH_xfzaC-Zyqcq-=qMFKKVarfzxOKqhp4g@mail.gmail.com>
	<CADiSq7f55KgseYj-Xs1nT7RE5vOtqHc2pz-fvgB0igM4XMDyPg@mail.gmail.com>
Message-ID: <1321724939.24665.56.camel@Gutsy>

On Sat, 2011-11-19 at 18:25 +1000, Nick Coghlan wrote:
> On Sat, Nov 19, 2011 at 5:55 PM, Gregory P. Smith <greg at krypto.org> wrote:
> > A double word syntax might make sense here.  It'd be good to keep 'yeild' in
> > it to make it clear that this is offering the exception out in a
> > non-terminal manner.
> >   yield raise MyException(x,y,z)
> > or if you've caught an exception from something and want to pass that on to
> > the caller without killing the iteration the natural base form of that would
> > be:
> >   yield raise
> > to unset the existing exception and yeild it out instead, mirroring what a
> > bare raise within an except: clause does.
> 
> Indeed, "yield raise" would be quite appropriate terminology for this
> new channel of communication.

I'll see if I can get that to work.


> However, there's still a potential stack unwinding problem here. For
> asymmetric coroutines, we need to be able to communicate an arbitrary
> distance up the stack. This idea, as it stands, doesn't provide that -
> like an ordinary yielded value, it can only yield control one level
> out. So even if the innermost generator is left in a resumable state,
> any *external* iterators are still going to be terminated.

It's really no different than gen.throw.() only going one deep.  It
doesn't go all the way to the bottom "implicitly" and then unwinds back
out.


> It potentially becomes more useful in combination with 'yield from',
> since yielded exceptions would be passed up the stack, the same as
> yielded values.

Yes, I think it would work nicely with yield-from.


> So, while it's a neat trick and very cool that it could be implemented
> with such a small change,

> I still don't see how it helps substantially
> with the challenge of allowing *any* Python frame on a coroutine
> stack, not just generator frames.

I think *any* needs to be qualified in some way.  The frame inside need
to have some mechanism to be suspended or to pass control to someplace
else. (or through it)  That mechanism needs to be explicitly spelled in
some way.

We really need a good example of what that would/should look like.  What
exactly is the behavior you are looking for?

Cheers,
   Ron


























From ron3200 at gmail.com  Sat Nov 19 23:57:51 2011
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 19 Nov 2011 16:57:51 -0600
Subject: [Python-ideas] Exceptions thrown from generators.. patch.
In-Reply-To: <CADiSq7f55KgseYj-Xs1nT7RE5vOtqHc2pz-fvgB0igM4XMDyPg@mail.gmail.com>
References: <1321683879.22715.88.camel@Gutsy>
	<CAGE7PNJXjzg4Y4G8oH_xfzaC-Zyqcq-=qMFKKVarfzxOKqhp4g@mail.gmail.com>
	<CADiSq7f55KgseYj-Xs1nT7RE5vOtqHc2pz-fvgB0igM4XMDyPg@mail.gmail.com>
Message-ID: <1321743471.25369.35.camel@Gutsy>

On Sat, 2011-11-19 at 18:25 +1000, Nick Coghlan wrote:
> So, while it's a neat trick and very cool that it could be implemented
> with such a small change, I still don't see how it helps substantially
> with the challenge of allowing *any* Python frame on a coroutine
> stack, not just generator frames. 

I was thinking about this today and a few things occurred to me.

You don't want to suspend a regular function.  What would happen if
another function tried to call it in it's suspended state?

To make that work, each invocation would need it's own frame.  That is
why generators return a generator object instance.  To do that with
functions would require making a function call into a
called-function-instance.  To be able to suspend and resume that
instance, you will need a reference to the instance rather than the
function.  So then you get something that is very much like a generator.
The only difference is it doesn't have a yield.  But in order for that
to suspend, it would need some mechanism to suspend itself.  Like a
yield.  And then we are right back to generators.

The other approach is to use thread objects.  Which create instances
that can be suspended.  The advantage of using threads, is that the
thread manager can suspend and resume threads independently of what the
thread is doing.  But that requires more resources to do that and they
may not be as efficient in tight loops.

The two different approaches are completely separate.  So I don't see
how this effects either yield-from or a possible yield-raise feature.

I'm probably missing something, but I can't put my finger on it.

Cheers,
   Ron




From cmjohnson.mailinglist at gmail.com  Sun Nov 20 00:43:16 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sat, 19 Nov 2011 13:43:16 -1000
Subject: [Python-ideas] Except block for with-statement
Message-ID: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>

I was looking through the "What's New for Python 3.3" file and I saw this example code:

try:
    with open("document.txt") as f:
        content = f.read()
except FileNotFoundError:
    print("document.txt file is missing")
except PermissionError:
    print("You are not allowed to read document.txt")

and I thought it would be more elegant if you could drop the try.

with open("document.txt") as f:
    content = f.read()
except FileNotFoundError:
    print("document.txt file is missing")
except PermissionError:
    print("You are not allowed to read document.txt")

I assume that this has already been proposed and rejected. Does anyone have a good explanation of why? ISTM that the with-statement is mostly just a way of abstracting out try-except blocks, so it might be nice to have a way of handling errors that pop up during the initialization phase instead of just the code-block phase.

I guess one obvious counter argument is that naive programmers might think that the "except" applies to things in the block instead of just things in the initializer. But naive programmers think lots of wrong things. ;-)

From anacrolix at gmail.com  Sun Nov 20 03:40:41 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Sun, 20 Nov 2011 13:40:41 +1100
Subject: [Python-ideas] Except block for with-statement
In-Reply-To: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
References: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
Message-ID: <CAB4yi1Nu+eS=-JGQ8bWdvZSDCSPm6z2FMSTJUoPHeBJ6JmB88w@mail.gmail.com>

That's a fantastic idea. As it stands context objects need to be done
manually if exception handling is desired during enter. The except here
could apply to the with statement only, since it's not possible to
differentiate at present. Either way +1 to some consideration of Carl's
proposal.
On Nov 20, 2011 10:43 AM, "Carl M. Johnson" <cmjohnson.mailinglist at gmail.com>
wrote:

> I was looking through the "What's New for Python 3.3" file and I saw this
> example code:
>
> try:
>    with open("document.txt") as f:
>        content = f.read()
> except FileNotFoundError:
>    print("document.txt file is missing")
> except PermissionError:
>    print("You are not allowed to read document.txt")
>
> and I thought it would be more elegant if you could drop the try.
>
> with open("document.txt") as f:
>    content = f.read()
> except FileNotFoundError:
>    print("document.txt file is missing")
> except PermissionError:
>    print("You are not allowed to read document.txt")
>
> I assume that this has already been proposed and rejected. Does anyone
> have a good explanation of why? ISTM that the with-statement is mostly just
> a way of abstracting out try-except blocks, so it might be nice to have a
> way of handling errors that pop up during the initialization phase instead
> of just the code-block phase.
>
> I guess one obvious counter argument is that naive programmers might think
> that the "except" applies to things in the block instead of just things in
> the initializer. But naive programmers think lots of wrong things. ;-)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111120/e4336848/attachment.html>

From anacrolix at gmail.com  Sun Nov 20 03:43:05 2011
From: anacrolix at gmail.com (Matt Joiner)
Date: Sun, 20 Nov 2011 13:43:05 +1100
Subject: [Python-ideas] Except block for with-statement
In-Reply-To: <CAB4yi1Nu+eS=-JGQ8bWdvZSDCSPm6z2FMSTJUoPHeBJ6JmB88w@mail.gmail.com>
References: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
	<CAB4yi1Nu+eS=-JGQ8bWdvZSDCSPm6z2FMSTJUoPHeBJ6JmB88w@mail.gmail.com>
Message-ID: <CAB4yi1PggeD58-c-QvtEbo6Bb5DYR4qoQeZnv4tB19jG9gOu9Q@mail.gmail.com>

Oops, just read the rest of the proposal which said exactly this.
On Nov 20, 2011 1:40 PM, "Matt Joiner" <anacrolix at gmail.com> wrote:

> That's a fantastic idea. As it stands context objects need to be done
> manually if exception handling is desired during enter. The except here
> could apply to the with statement only, since it's not possible to
> differentiate at present. Either way +1 to some consideration of Carl's
> proposal.
> On Nov 20, 2011 10:43 AM, "Carl M. Johnson" <
> cmjohnson.mailinglist at gmail.com> wrote:
>
>> I was looking through the "What's New for Python 3.3" file and I saw this
>> example code:
>>
>> try:
>>    with open("document.txt") as f:
>>        content = f.read()
>> except FileNotFoundError:
>>    print("document.txt file is missing")
>> except PermissionError:
>>    print("You are not allowed to read document.txt")
>>
>> and I thought it would be more elegant if you could drop the try.
>>
>> with open("document.txt") as f:
>>    content = f.read()
>> except FileNotFoundError:
>>    print("document.txt file is missing")
>> except PermissionError:
>>    print("You are not allowed to read document.txt")
>>
>> I assume that this has already been proposed and rejected. Does anyone
>> have a good explanation of why? ISTM that the with-statement is mostly just
>> a way of abstracting out try-except blocks, so it might be nice to have a
>> way of handling errors that pop up during the initialization phase instead
>> of just the code-block phase.
>>
>> I guess one obvious counter argument is that naive programmers might
>> think that the "except" applies to things in the block instead of just
>> things in the initializer. But naive programmers think lots of wrong
>> things. ;-)
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111120/e7d71cad/attachment.html>

From guido at python.org  Sun Nov 20 05:09:12 2011
From: guido at python.org (Guido van Rossum)
Date: Sat, 19 Nov 2011 20:09:12 -0800
Subject: [Python-ideas] Except block for with-statement
In-Reply-To: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
References: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
Message-ID: <CAP7+vJJCN8umoE-Q63Up+U4Kx8gtum3=Gka5_zRyY3gyspS+Gg@mail.gmail.com>

On Sat, Nov 19, 2011 at 3:43 PM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> I was looking through the "What's New for Python 3.3" file and I saw this example code:
>
> try:
> ? ?with open("document.txt") as f:
> ? ? ? ?content = f.read()
> except FileNotFoundError:
> ? ?print("document.txt file is missing")
> except PermissionError:
> ? ?print("You are not allowed to read document.txt")
>
> and I thought it would be more elegant if you could drop the try.
>
> with open("document.txt") as f:
> ? ?content = f.read()
> except FileNotFoundError:
> ? ?print("document.txt file is missing")
> except PermissionError:
> ? ?print("You are not allowed to read document.txt")

Eew. No please. There seems to be a trend in proposing more random
variants of the existing compound statements. Please don't do this.
The existing complement of compound statements is quite sufficient and
the new proposals do nothing but complicate the parser, the
documentation, and the learning process for occasional users.

> I assume that this has already been proposed and rejected. Does anyone have a good explanation of why?

I suppose "because I say so" is not a good explanation? :-)

> ISTM that the with-statement is mostly just a way of abstracting out try-except blocks, so it might be nice to have a way of handling errors that pop up during the initialization phase instead of just the code-block phase.

Wrong. The with-statement abstracts out a try-*finally* block. A
try-except cannot be replaced by a with-statement (or it would have to
be a very crooked context manager).

> I guess one obvious counter argument is that naive programmers might think that the "except" applies to things in the block instead of just things in the initializer. But naive programmers think lots of wrong things. ;-)

That's actually a deal killer right there. (If it wasn't born dead,
that is. :-) Your proposal above, through it "this could be equivalent
to that" example, makes it seem like the except block applies to the
entire with-statement. But here you appear to say that an error in the
body of the with-block would not be covered? And your example use of
exceptions that are only raised by open() strengthens this.

If a proposal manages to confuse even its proposer, perhaps that is
enough to reject it?

-- 
--Guido van Rossum (python.org/~guido)


From tjreedy at udel.edu  Sun Nov 20 09:05:24 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 20 Nov 2011 03:05:24 -0500
Subject: [Python-ideas] Except block for with-statement
In-Reply-To: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
References: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
Message-ID: <jaacce$186$1@dough.gmane.org>

On 11/19/2011 6:43 PM, Carl M. Johnson wrote:
> I was looking through the "What's New for Python 3.3" file and I saw this example code:
>
> try:
>      with open("document.txt") as f:
>          content = f.read()
> except FileNotFoundError:
>      print("document.txt file is missing")
> except PermissionError:
>      print("You are not allowed to read document.txt")
>
> and I thought it would be more elegant if you could drop the try.
>
> with open("document.txt") as f:
>      content = f.read()
> except FileNotFoundError:
>      print("document.txt file is missing")
> except PermissionError:
>      print("You are not allowed to read document.txt")

This saves one short line. That is insufficient reason for a puzzling 
construct.

-- 
Terry Jan Reedy



From ncoghlan at gmail.com  Sun Nov 20 13:07:39 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 20 Nov 2011 22:07:39 +1000
Subject: [Python-ideas] Exceptions thrown from generators.. patch.
In-Reply-To: <1321743471.25369.35.camel@Gutsy>
References: <1321683879.22715.88.camel@Gutsy>
	<CAGE7PNJXjzg4Y4G8oH_xfzaC-Zyqcq-=qMFKKVarfzxOKqhp4g@mail.gmail.com>
	<CADiSq7f55KgseYj-Xs1nT7RE5vOtqHc2pz-fvgB0igM4XMDyPg@mail.gmail.com>
	<1321743471.25369.35.camel@Gutsy>
Message-ID: <CADiSq7dP62b4DUF6W8L1FA3pPEET5ZNCZ2p-x5A+ZLqFoQ5WRQ@mail.gmail.com>

On Sun, Nov 20, 2011 at 8:57 AM, Ron Adam <ron3200 at gmail.com> wrote:
> I'm probably missing something, but I can't put my finger on it.

Re-read the last discussion of Greg's coroutine PEP in the list
archives. If every frame on the stack has to be a generator frame,
you're effectively bifurcating Python into two languages - "normal
Python" (which uses both functions and generators) and "coroutine
Python" (which uses generators for everything). That's a bad idea, and
the reason so many people prefer the thread-style model of greenlet
based programming to the Twisted-style "inside out" model of event
driven programming.

This is a post where I highlight some of the issues with bifurcating
the language, as well as the fact that "generators-all-the-way-down"
*does* lead to bifurcation:
http://mail.python.org/pipermail/python-ideas/2011-October/012570.html

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Sun Nov 20 13:22:00 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 20 Nov 2011 22:22:00 +1000
Subject: [Python-ideas] Except block for with-statement
In-Reply-To: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
References: <424342F7-00E7-4C6D-AA06-3B612B37E563@gmail.com>
Message-ID: <CADiSq7e15oEwgq+bB0=ZM6TTtyMo823zFTJ2jNmW0A6mw-Lh=w@mail.gmail.com>

On Sun, Nov 20, 2011 at 9:43 AM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> I was looking through the "What's New for Python 3.3" file and I saw this example code:
>
> try:
> ? ?with open("document.txt") as f:
> ? ? ? ?content = f.read()
> except FileNotFoundError:
> ? ?print("document.txt file is missing")
> except PermissionError:
> ? ?print("You are not allowed to read document.txt")

Hmm, this code is dubious, as the scope of the try block is too broad.
It should really be written so it only covers the creation of the file
object:

try:
   f = open("document.txt")
except FileNotFoundError:
   print("document.txt file is missing")
except PermissionError:
   print("You are not allowed to read document.txt")
else:
   with f:
       content = f.read()

At that point, it's hopefully clearer why this proposal doesn't make
sense. (A tracker issue suggesting that the What's New example be
updated wouldn't hurt, though).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From mwm at mired.org  Wed Nov 23 00:09:34 2011
From: mwm at mired.org (Mike Meyer)
Date: Tue, 22 Nov 2011 15:09:34 -0800
Subject: [Python-ideas] proto-pep capturing the concurrency discusion
Message-ID: <CAD=7U2BSiRiVuuBPZmzTMUJR4Vn+ZL4ffyWucSGKBR6CuVZPgA@mail.gmail.com>

I tried to capture all the ideas generated during the concurrency
discussion early this month in the form of an informational PEP. I
believe that information is worth preserving where others can find it,
and a PEP seems like a logical place. Since it doesn't make a specific
proposal, it's informational PEP. Possibly this is an abuse of the PEP
mechanism, in which case I'll find another forum for it.

The point this time is not to debate the ideas proposed, but to check
for errors and omissions. If there's an alternative I missed, let me
know. If there's a problem with some alternative I missed, let me
know. If I think it's already in the document, I'll probably point it
out and ask for suggestions on how to make it more obvious.

     Thanks,
     <mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pep-concurrent.rst
Type: application/octet-stream
Size: 11661 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111122/06c18b93/attachment.obj>

From ncoghlan at gmail.com  Wed Nov 23 01:30:07 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 23 Nov 2011 10:30:07 +1000
Subject: [Python-ideas] proto-pep capturing the concurrency discusion
In-Reply-To: <CAD=7U2BSiRiVuuBPZmzTMUJR4Vn+ZL4ffyWucSGKBR6CuVZPgA@mail.gmail.com>
References: <CAD=7U2BSiRiVuuBPZmzTMUJR4Vn+ZL4ffyWucSGKBR6CuVZPgA@mail.gmail.com>
Message-ID: <CADiSq7cJoSyfWVcg9fnuJkW9HGXcX--51nQEH8Dr72_Rm4baog@mail.gmail.com>

On Wed, Nov 23, 2011 at 9:09 AM, Mike Meyer <mwm at mired.org> wrote:
> I tried to capture all the ideas generated during the concurrency
> discussion early this month in the form of an informational PEP. I
> believe that information is worth preserving where others can find it,
> and a PEP seems like a logical place. Since it doesn't make a specific
> proposal, it's informational PEP. Possibly this is an abuse of the PEP
> mechanism, in which case I'll find another forum for it.

The Python wiki is a fairly common location for pre-PEPs that haven't
really made it to the point of proposing specific changes:
http://wiki.python.org/moin/PythonEnhancementProposals

It also uses ReST for formatting, so you can just drop your proto-PEP
into a new page and go from there.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From someuniquename at gmail.com  Sun Nov 27 22:33:48 2011
From: someuniquename at gmail.com (Roman Evstifeev)
Date: Mon, 28 Nov 2011 00:33:48 +0300
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <1320516118.14236.11.camel@Gutsy>
References: <20111030001801.2f52ceb2@pitrou.net> <4EB5653A.4000302@netwok.org>
	<CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>
	<1320516118.14236.11.camel@Gutsy>
Message-ID: <CAMX-vmK6UKMN=4fYmjjiq8sBMC5kQiWMqqUjAViqC=JucKj9TQ@mail.gmail.com>

On Sat, Nov 5, 2011 at 9:01 PM, Ron Adam <ron3200 at gmail.com> wrote:
> On Sat, 2011-11-05 at 11:04 -0600, Eric Snow wrote:
>> On Sat, Nov 5, 2011 at 10:32 AM, ?ric Araujo <merwok at netwok.org> wrote:
>> > Hi,
>> >
>> >> I would like to propose the following PEP for discussion and, if
>> >> possible, acceptance. I think the proposal shouldn't be too
>> >> controversial (I find it quite simple and straightforward myself :-)).
>> >> [snip PEP]
>> >
>> > +1.
>> >
>> > For nested functions, I too think that 'f.<locals>.g' has too many dots;
>> > I like '<local in f>.g' or '<f locals>.g'.
>>
>> I like it too but don't think it's too many dots.
>>
>> The function from which the locals came _could_ be rolled into the
>> brackets. ?However, in the context of some object (like the class X to
>> which f belongs), 'X.f.<locals of f>.g' makes more sense in that case
>> than 'X.<locals of f>.g', since the locals is related to f and not X.
>> But, then the f is sort of redundant, so you go back to
>> 'X.f.<locals>.g', and '<locals>' is still sort of unambiguous.
>>
>> The disconnect is that <locals> is an externally anonymous namespace
>> resulting from a call, rather than bound to any external namespace
>> (like an object). ?Perhaps it would be appropriate to use
>> 'X.f().<locals>.g' to make that clear.
>
> I think if you consider locals in f as an implementation detail of f's
> name space rather than a sub item of f, it's not as confusing. ?It's
> better to think of locals as being part of f, rather than in f. ?That is
> why <f locals> makes more sense than f.<locals>. ?For example locals is
> in f's frame object, so if you follow that reasoning you get.
> f.<frame>.<locals>, ?but I don't think we need all that.
>
> Hmmm... ?I think it actually should be spelled...
>
> ? ? ?f.<local g>
>
> Following a pattern of...
>
> ? ? ?x ? ? ? ? ? ? ? ? object x
> ? ? ?x.f ? ? ? ? ? ? ? f in object x
> ? ? ?x.f.<local g> ? ? local g in f in x
>
> That's both clear and concise.
>
> Cheers,
> ? Ron
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>

May i bikeshed a bit more please?

+1 to Ron

1)   C.f.<locals>.g
2)   C.<f locals>.g
3)   C.f.<local g>
I think less verbose and confusing is variant 3


From ncoghlan at gmail.com  Sun Nov 27 23:05:49 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 28 Nov 2011 08:05:49 +1000
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <CAMX-vmK6UKMN=4fYmjjiq8sBMC5kQiWMqqUjAViqC=JucKj9TQ@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net> <4EB5653A.4000302@netwok.org>
	<CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>
	<1320516118.14236.11.camel@Gutsy>
	<CAMX-vmK6UKMN=4fYmjjiq8sBMC5kQiWMqqUjAViqC=JucKj9TQ@mail.gmail.com>
Message-ID: <CADiSq7d1EFHNV1CRpdVrUd16TiTVdaHsD-kOO0u-X5uX=CvMVw@mail.gmail.com>

No, Guido already accepted the PEP with the existing spelling.

--
Nick Coghlan (via Gmail on Android, so likely to be more terse than usual)
On Nov 28, 2011 7:34 AM, "Roman Evstifeev" <someuniquename at gmail.com> wrote:

> On Sat, Nov 5, 2011 at 9:01 PM, Ron Adam <ron3200 at gmail.com> wrote:
> > On Sat, 2011-11-05 at 11:04 -0600, Eric Snow wrote:
> >> On Sat, Nov 5, 2011 at 10:32 AM, ?ric Araujo <merwok at netwok.org> wrote:
> >> > Hi,
> >> >
> >> >> I would like to propose the following PEP for discussion and, if
> >> >> possible, acceptance. I think the proposal shouldn't be too
> >> >> controversial (I find it quite simple and straightforward myself
> :-)).
> >> >> [snip PEP]
> >> >
> >> > +1.
> >> >
> >> > For nested functions, I too think that 'f.<locals>.g' has too many
> dots;
> >> > I like '<local in f>.g' or '<f locals>.g'.
> >>
> >> I like it too but don't think it's too many dots.
> >>
> >> The function from which the locals came _could_ be rolled into the
> >> brackets.  However, in the context of some object (like the class X to
> >> which f belongs), 'X.f.<locals of f>.g' makes more sense in that case
> >> than 'X.<locals of f>.g', since the locals is related to f and not X.
> >> But, then the f is sort of redundant, so you go back to
> >> 'X.f.<locals>.g', and '<locals>' is still sort of unambiguous.
> >>
> >> The disconnect is that <locals> is an externally anonymous namespace
> >> resulting from a call, rather than bound to any external namespace
> >> (like an object).  Perhaps it would be appropriate to use
> >> 'X.f().<locals>.g' to make that clear.
> >
> > I think if you consider locals in f as an implementation detail of f's
> > name space rather than a sub item of f, it's not as confusing.  It's
> > better to think of locals as being part of f, rather than in f.  That is
> > why <f locals> makes more sense than f.<locals>.  For example locals is
> > in f's frame object, so if you follow that reasoning you get.
> > f.<frame>.<locals>,  but I don't think we need all that.
> >
> > Hmmm...  I think it actually should be spelled...
> >
> >      f.<local g>
> >
> > Following a pattern of...
> >
> >      x                 object x
> >      x.f               f in object x
> >      x.f.<local g>     local g in f in x
> >
> > That's both clear and concise.
> >
> > Cheers,
> >   Ron
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > http://mail.python.org/mailman/listinfo/python-ideas
> >
>
> May i bikeshed a bit more please?
>
> +1 to Ron
>
> 1)   C.f.<locals>.g
> 2)   C.<f locals>.g
> 3)   C.f.<local g>
> I think less verbose and confusing is variant 3
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111128/6a9e0da2/attachment.html>

From guido at python.org  Mon Nov 28 00:35:36 2011
From: guido at python.org (Guido van Rossum)
Date: Sun, 27 Nov 2011 15:35:36 -0800
Subject: [Python-ideas] PEP 3155 - Qualified name for classes and
	functions
In-Reply-To: <CADiSq7d1EFHNV1CRpdVrUd16TiTVdaHsD-kOO0u-X5uX=CvMVw@mail.gmail.com>
References: <20111030001801.2f52ceb2@pitrou.net> <4EB5653A.4000302@netwok.org>
	<CALFfu7B69hYd=35aS7FJTyWvz__DTF+Wxr3gmrAJz_9HtW9Tjw@mail.gmail.com>
	<1320516118.14236.11.camel@Gutsy>
	<CAMX-vmK6UKMN=4fYmjjiq8sBMC5kQiWMqqUjAViqC=JucKj9TQ@mail.gmail.com>
	<CADiSq7d1EFHNV1CRpdVrUd16TiTVdaHsD-kOO0u-X5uX=CvMVw@mail.gmail.com>
Message-ID: <CAP7+vJ+XxbP_nO3515K761PDtXnJ6zm3qk9pekw6TSCrUaonjA@mail.gmail.com>

And, lest this becomes proof by authority, let me point out that the
<f locals> and <local g> notations are inconsistent, since at other
positions in tha actual name is between the dots (or at least has a
dot or the end of the string on each side).

--Guido

On Sun, Nov 27, 2011 at 2:05 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> No, Guido already accepted the PEP with the existing spelling.
>
> --
> Nick Coghlan (via Gmail on Android, so likely to be more terse than usual)
>
> On Nov 28, 2011 7:34 AM, "Roman Evstifeev" <someuniquename at gmail.com> wrote:
>>
>> On Sat, Nov 5, 2011 at 9:01 PM, Ron Adam <ron3200 at gmail.com> wrote:
>> > On Sat, 2011-11-05 at 11:04 -0600, Eric Snow wrote:
>> >> On Sat, Nov 5, 2011 at 10:32 AM, ?ric Araujo <merwok at netwok.org> wrote:
>> >> > Hi,
>> >> >
>> >> >> I would like to propose the following PEP for discussion and, if
>> >> >> possible, acceptance. I think the proposal shouldn't be too
>> >> >> controversial (I find it quite simple and straightforward myself
>> >> >> :-)).
>> >> >> [snip PEP]
>> >> >
>> >> > +1.
>> >> >
>> >> > For nested functions, I too think that 'f.<locals>.g' has too many
>> >> > dots;
>> >> > I like '<local in f>.g' or '<f locals>.g'.
>> >>
>> >> I like it too but don't think it's too many dots.
>> >>
>> >> The function from which the locals came _could_ be rolled into the
>> >> brackets. ?However, in the context of some object (like the class X to
>> >> which f belongs), 'X.f.<locals of f>.g' makes more sense in that case
>> >> than 'X.<locals of f>.g', since the locals is related to f and not X.
>> >> But, then the f is sort of redundant, so you go back to
>> >> 'X.f.<locals>.g', and '<locals>' is still sort of unambiguous.
>> >>
>> >> The disconnect is that <locals> is an externally anonymous namespace
>> >> resulting from a call, rather than bound to any external namespace
>> >> (like an object). ?Perhaps it would be appropriate to use
>> >> 'X.f().<locals>.g' to make that clear.
>> >
>> > I think if you consider locals in f as an implementation detail of f's
>> > name space rather than a sub item of f, it's not as confusing. ?It's
>> > better to think of locals as being part of f, rather than in f. ?That is
>> > why <f locals> makes more sense than f.<locals>. ?For example locals is
>> > in f's frame object, so if you follow that reasoning you get.
>> > f.<frame>.<locals>, ?but I don't think we need all that.
>> >
>> > Hmmm... ?I think it actually should be spelled...
>> >
>> > ? ? ?f.<local g>
>> >
>> > Following a pattern of...
>> >
>> > ? ? ?x ? ? ? ? ? ? ? ? object x
>> > ? ? ?x.f ? ? ? ? ? ? ? f in object x
>> > ? ? ?x.f.<local g> ? ? local g in f in x
>> >
>> > That's both clear and concise.
>> >
>> > Cheers,
>> > ? Ron
>> > _______________________________________________
>> > Python-ideas mailing list
>> > Python-ideas at python.org
>> > http://mail.python.org/mailman/listinfo/python-ideas
>> >
>>
>> May i bikeshed a bit more please?
>>
>> +1 to Ron
>>
>> 1) ? C.f.<locals>.g
>> 2) ? C.<f locals>.g
>> 3) ? C.f.<local g>
>> I think less verbose and confusing is variant 3
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
>



-- 
--Guido van Rossum (python.org/~guido)


From g.rodola at gmail.com  Wed Nov 30 14:32:24 2011
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Wed, 30 Nov 2011 14:32:24 +0100
Subject: [Python-ideas] Is there a reason why file.readlines() doesn't/can't
	return an iterator?
Message-ID: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>

This is problably too late and I'm probably missing something but
given amount of generators/iterators introduced in python 3.X
(http://docs.python.org/release/3.0.1/whatsnew/3.0.html#views-and-iterators-instead-of-lists)
file.readlines() seems a good case where an iterator can be more
appropriate than a list.
I realized it while writing this recipe:
http://code.activestate.com/recipes/577968-log-watcher-tail-f-log/
In this specific case, having readlines() yield a single line at a
time would save a lot of memory.
Maybe we can introduce a new parameter to do this?

Regards,

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/


From eliben at gmail.com  Wed Nov 30 14:52:52 2011
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 30 Nov 2011 15:52:52 +0200
Subject: [Python-ideas] Is there a reason why file.readlines()
 doesn't/can't return an iterator?
In-Reply-To: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
Message-ID: <CAF-Rda__rXutu0AXoqQ8nRiNrwPFC9_=M-THH2DyhK5jAmToGQ@mail.gmail.com>

On Wed, Nov 30, 2011 at 15:32, Giampaolo Rodol? <g.rodola at gmail.com> wrote:

> This is problably too late and I'm probably missing something but
> given amount of generators/iterators introduced in python 3.X
> (
> http://docs.python.org/release/3.0.1/whatsnew/3.0.html#views-and-iterators-instead-of-lists
> )
> file.readlines() seems a good case where an iterator can be more
> appropriate than a list.
> I realized it while writing this recipe:
> http://code.activestate.com/recipes/577968-log-watcher-tail-f-log/
> In this specific case, having readlines() yield a single line at a
> time would save a lot of memory.
> Maybe we can introduce a new parameter to do this?
>

Giampaolo, perhaps I'm missing something, but what's bad about having a
generator instead:

linesgen = (line for line in file)

You'd basically now do with linesgen whatever you wanted to do with an
iterator-returning file.readlines()

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/9d8a4f2c/attachment.html>

From amauryfa at gmail.com  Wed Nov 30 14:54:30 2011
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 30 Nov 2011 14:54:30 +0100
Subject: [Python-ideas] Is there a reason why file.readlines()
 doesn't/can't return an iterator?
In-Reply-To: <CAF-Rda__rXutu0AXoqQ8nRiNrwPFC9_=M-THH2DyhK5jAmToGQ@mail.gmail.com>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<CAF-Rda__rXutu0AXoqQ8nRiNrwPFC9_=M-THH2DyhK5jAmToGQ@mail.gmail.com>
Message-ID: <CAGmFidawaf=DZwL-DJ7Wyqjwvr4fEf=vnjCSr9kF1Xsc3Z19YQ@mail.gmail.com>

2011/11/30 Eli Bendersky <eliben at gmail.com>

> linesgen = (line for line in file)
>

Shorter::
    iter(file)

-- 
Amaury Forgeot d'Arc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/1084e6d3/attachment.html>

From masklinn at masklinn.net  Wed Nov 30 14:55:46 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 30 Nov 2011 14:55:46 +0100
Subject: [Python-ideas] Is there a reason why file.readlines()
	doesn't/can't return an iterator?
In-Reply-To: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
Message-ID: <A0A74343-6816-481D-B772-8111223DB36C@masklinn.net>

On 2011-11-30, at 14:32 , Giampaolo Rodol? wrote:
> This is problably too late and I'm probably missing something but
> given amount of generators/iterators introduced in python 3.X
> (http://docs.python.org/release/3.0.1/whatsnew/3.0.html#views-and-iterators-instead-of-lists)
> file.readlines() seems a good case where an iterator can be more
> appropriate than a list.
Iterating on the file itself already iterates by line, what would be the value of doing it *again* for readlines()? The sizehint?

Note how file objects *used* to have a `xreadlines()` method which was deprecated (and removed in Python3) in favor of iterating the file object directly.



From masklinn at masklinn.net  Wed Nov 30 14:57:16 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 30 Nov 2011 14:57:16 +0100
Subject: [Python-ideas] Is there a reason why file.readlines()
	doesn't/can't return an iterator?
In-Reply-To: <CAGmFidawaf=DZwL-DJ7Wyqjwvr4fEf=vnjCSr9kF1Xsc3Z19YQ@mail.gmail.com>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<CAF-Rda__rXutu0AXoqQ8nRiNrwPFC9_=M-THH2DyhK5jAmToGQ@mail.gmail.com>
	<CAGmFidawaf=DZwL-DJ7Wyqjwvr4fEf=vnjCSr9kF1Xsc3Z19YQ@mail.gmail.com>
Message-ID: <39647018-84A4-4190-883C-9E73A1E4A414@masklinn.net>

On 2011-11-30, at 14:54 , Amaury Forgeot d'Arc wrote:
> 2011/11/30 Eli Bendersky <eliben at gmail.com>
>> linesgen = (line for line in file)
>> 
> 
> Shorter::
>    iter(file)
Even shorter::
    file

(iter(file) just returns the file object in Python2, and the BufferedReader in Python3, so it's a useless indirection)

From __peter__ at web.de  Wed Nov 30 14:59:30 2011
From: __peter__ at web.de (Peter Otten)
Date: Wed, 30 Nov 2011 14:59:30 +0100
Subject: [Python-ideas] Is there a reason why file.readlines()
	doesn't/can't return an iterator?
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
Message-ID: <jb5crq$j0m$1@dough.gmane.org>

Giampaolo Rodol? wrote:

> This is problably too late and I'm probably missing something but
> given amount of generators/iterators introduced in python 3.X
> (http://docs.python.org/release/3.0.1/whatsnew/3.0.html#views-and-
iterators-instead-of-lists)
> file.readlines() seems a good case where an iterator can be more
> appropriate than a list.
> I realized it while writing this recipe:
> http://code.activestate.com/recipes/577968-log-watcher-tail-f-log/
> In this specific case, having readlines() yield a single line at a
> time would save a lot of memory.
> Maybe we can introduce a new parameter to do this?

You already have an iterator -- the file itself. Why would you need another 
one?

My observation on the Tutor mailing list is that there are no valid uses of 
readlines(). It's just easier to discover the readlines() method than to 
find out that you can iterate over the file directly.

So my counter-proposal is to remove the readlines method but to leave the 
corresponding entry in the documentation with a a description of the "right 
way" to iterate over the lines of a file. Backwards compatibility be damned 
;)



From ubershmekel at gmail.com  Wed Nov 30 15:31:14 2011
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 30 Nov 2011 16:31:14 +0200
Subject: [Python-ideas] Is there a reason why file.readlines()
 doesn't/can't return an iterator?
In-Reply-To: <jb5crq$j0m$1@dough.gmane.org>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org>
Message-ID: <CANSw7Kwjmdyb=cdDH+YCE034FMLKzMbxAGd3GdJCQBq4h-BATQ@mail.gmail.com>

Since we marginally brushed the subject of backwards compatibility
damning...

I'm kind of surprised that files do that. Now that I check I'm surprised
that StringIO does that.

text = 'asdf\nword'
for item in StringIO(text):    !=    for item in text:

I'm not sure if I like that.

--Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/46c8673f/attachment.html>

From g.rodola at gmail.com  Wed Nov 30 16:42:16 2011
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Wed, 30 Nov 2011 16:42:16 +0100
Subject: [Python-ideas] Is there a reason why file.readlines()
 doesn't/can't return an iterator?
In-Reply-To: <jb5crq$j0m$1@dough.gmane.org>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org>
Message-ID: <CAFYqXL9WcWoXeo9-bPooM+4tNhQZy+tpCdRYjffkgvQPtjtjMw@mail.gmail.com>

Il 30 novembre 2011 14:59, Peter Otten <__peter__ at web.de> ha scritto:
> Giampaolo Rodol? wrote:
>
>> This is problably too late and I'm probably missing something but
>> given amount of generators/iterators introduced in python 3.X
>> (http://docs.python.org/release/3.0.1/whatsnew/3.0.html#views-and-
> iterators-instead-of-lists)
>> file.readlines() seems a good case where an iterator can be more
>> appropriate than a list.
>> I realized it while writing this recipe:
>> http://code.activestate.com/recipes/577968-log-watcher-tail-f-log/
>> In this specific case, having readlines() yield a single line at a
>> time would save a lot of memory.
>> Maybe we can introduce a new parameter to do this?
>
> You already have an iterator -- the file itself. Why would you need another
> one?
>
> My observation on the Tutor mailing list is that there are no valid uses of
> readlines(). It's just easier to discover the readlines() method than to
> find out that you can iterate over the file directly.
>
> So my counter-proposal is to remove the readlines method but to leave the
> corresponding entry in the documentation with a a description of the "right
> way" to iterate over the lines of a file. Backwards compatibility be damned
> ;)

Yes, you're right.
Now that you pointed this out I realize that my request isn't actually
necessary.
To be fair, readlines() is different than an iterated file object in
that it provides the sizehint parameter, so it does cover a use case.
Maybe the doc should just mention that unless sizehint parameter is
used the recommended way to get file lines is to iterate against the
file object itself.

Regards,

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/


From merwok at netwok.org  Wed Nov 30 16:51:27 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Wed, 30 Nov 2011 16:51:27 +0100
Subject: [Python-ideas] Is there a reason why
 file.readlines()	doesn't/can't return an iterator?
In-Reply-To: <jb5crq$j0m$1@dough.gmane.org>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org>
Message-ID: <4ED650FF.9020705@netwok.org>

Le 30/11/2011 14:59, Peter Otten a ?crit :
> My observation on the Tutor mailing list is that there are no valid
> uses of readlines().
I disagree :)  You don?t always read big files, or you may prefer to
close the file handle early at the cost of memory, or you need a list to
do some processing that works with lists but not iterators.  FWIW, I use
readlines a lot to get list of lines for unit tests, for example (in
case of failure, lists make better diffs than strings).

> So my counter-proposal is to remove the readlines method but to leave the 
> corresponding entry in the documentation with a a description of the "right 
> way" to iterate over the lines of a file.
A doc entry for a non-existing function would be very confusing.

> Backwards compatibility be damned ;)
Not sure what the smiley means; compatibility is taken seriously, and
removing things has a very high bar.

Cheers


From guido at python.org  Wed Nov 30 16:54:11 2011
From: guido at python.org (Guido van Rossum)
Date: Wed, 30 Nov 2011 07:54:11 -0800
Subject: [Python-ideas] Is there a reason why file.readlines()
 doesn't/can't return an iterator?
In-Reply-To: <CANSw7Kwjmdyb=cdDH+YCE034FMLKzMbxAGd3GdJCQBq4h-BATQ@mail.gmail.com>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org>
	<CANSw7Kwjmdyb=cdDH+YCE034FMLKzMbxAGd3GdJCQBq4h-BATQ@mail.gmail.com>
Message-ID: <CAP7+vJJZQccwEFBSXMkhcK2v02FUjpNer3zz14A1949s4iZiew@mail.gmail.com>

On Wed, Nov 30, 2011 at 6:31 AM, Yuval Greenfield <ubershmekel at gmail.com>wrote:

> Since we marginally brushed the subject of backwards compatibility
> damning...
>
> I'm kind of surprised that files do that. Now that I check I'm surprised
> that StringIO does that.
>
> text = 'asdf\nword'
> for item in StringIO(text):    !=    for item in text:
>
> I'm not sure if I like that.
>

We thought about that. Really. We concluded that while iterating over the
characters of a file would be more "pure" but that it would rarely be
useful, while iterating over the files would be incredibly useful. That was
probably about 10 years ago, and I still think we made the right decision.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/6f194658/attachment.html>

From masklinn at masklinn.net  Wed Nov 30 17:18:30 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 30 Nov 2011 17:18:30 +0100
Subject: [Python-ideas] Is there a reason why
	file.readlines()	doesn't/can't return an iterator?
In-Reply-To: <4ED650FF.9020705@netwok.org>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org> <4ED650FF.9020705@netwok.org>
Message-ID: <16CD1DD8-82FC-4DB5-A162-05FC6292E543@masklinn.net>

On 2011-11-30, at 16:51 , ?ric Araujo wrote:
> Le 30/11/2011 14:59, Peter Otten a ?crit :
>> My observation on the Tutor mailing list is that there are no valid
>> uses of readlines().
> I disagree :)  You don?t always read big files, or you may prefer to
> close the file handle early at the cost of memory, or you need a list to
> do some processing that works with lists but not iterators.  FWIW, I use
> readlines a lot to get list of lines for unit tests, for example (in
> case of failure, lists make better diffs than strings).
Then again, the same process should be used here than the one recommended
for map, filter, dict.(keys|values|items) in Python 3: `list(file)` will
yield a list of lines.

From merwok at netwok.org  Wed Nov 30 17:23:16 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Wed, 30 Nov 2011 17:23:16 +0100
Subject: [Python-ideas] Is there a reason why
 file.readlines()	doesn't/can't return an iterator?
In-Reply-To: <16CD1DD8-82FC-4DB5-A162-05FC6292E543@masklinn.net>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org> <4ED650FF.9020705@netwok.org>
	<16CD1DD8-82FC-4DB5-A162-05FC6292E543@masklinn.net>
Message-ID: <4ED65874.2040201@netwok.org>

Le 30/11/2011 17:18, Masklinn a ?crit :
> On 2011-11-30, at 16:51 , ?ric Araujo wrote:
>> You don?t always read big files, or you may prefer to
>> close the file handle early at the cost of memory, or you need a list to
>> do some processing that works with lists but not iterators.  FWIW, I use
>> readlines a lot to get list of lines for unit tests, for example (in
>> case of failure, lists make better diffs than strings).
> Then again, the same process should be used here than the one recommended
> for map, filter, dict.(keys|values|items) in Python 3: `list(file)` will
> yield a list of lines.

Good point.  readlines strips the end-of-line characters though, so to
use list(fp) I?d have to change all my expected outputs to add \n.

Cheers


From __peter__ at web.de  Wed Nov 30 17:38:29 2011
From: __peter__ at web.de (Peter Otten)
Date: Wed, 30 Nov 2011 17:38:29 +0100
Subject: [Python-ideas] Is there a reason why
	file.readlines()	doesn't/can't return an iterator?
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org> <4ED650FF.9020705@netwok.org>
Message-ID: <jb5m5r$vrl$1@dough.gmane.org>

?ric Araujo wrote:

> Le 30/11/2011 14:59, Peter Otten a ?crit :
>> My observation on the Tutor mailing list is that there are no valid
>> uses of readlines().
> I disagree :)  

Not really, I'm relaying observations made on a list mostly frequented by 
newbies rather than giving my opinion.

> You don?t always read big files, or you may prefer to
> close the file handle early at the cost of memory, or you need a list to
> do some processing that works with lists but not iterators.  FWIW, I use
> readlines a lot to get list of lines for unit tests, for example (in
> case of failure, lists make better diffs than strings).

If there were no readlines() method, would you lobby its introduction or go 
with list(file)?

>> So my counter-proposal is to remove the readlines method but to leave the
>> corresponding entry in the documentation with a a description of the
>> "right way" to iterate over the lines of a file.
> A doc entry for a non-existing function would be very confusing.
> 
>> Backwards compatibility be damned ;)
> Not sure what the smiley means; compatibility is taken seriously, and
> removing things has a very high bar.

I wanted to express that the change would be good for newbies, but that I 
don't expect it to happen.



From merwok at netwok.org  Wed Nov 30 17:46:46 2011
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Wed, 30 Nov 2011 17:46:46 +0100
Subject: [Python-ideas] Is there a reason why
 file.readlines()	doesn't/can't return an iterator?
In-Reply-To: <jb5m5r$vrl$1@dough.gmane.org>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>	<jb5crq$j0m$1@dough.gmane.org>
	<4ED650FF.9020705@netwok.org> <jb5m5r$vrl$1@dough.gmane.org>
Message-ID: <4ED65DF6.3060900@netwok.org>

Hi Peter,

Le 30/11/2011 17:38, Peter Otten a ?crit :
> ?ric Araujo wrote:
>> Le 30/11/2011 14:59, Peter Otten a ?crit :
>>> My observation on the Tutor mailing list is that there are no valid
>>> uses of readlines().
>> I disagree :)
> Not really, I'm relaying observations made on a list mostly frequented by 
> newbies rather than giving my opinion.
Ah, I misunderstood.

>> You don?t always read big files, or you may prefer to
>> close the file handle early at the cost of memory, or you need a list to
>> do some processing that works with lists but not iterators.  FWIW, I use
>> readlines a lot to get list of lines for unit tests, for example (in
>> case of failure, lists make better diffs than strings).
> If there were no readlines() method, would you lobby its introduction or go 
> with list(file)?
I?d use list, if only because I need to support old Python versions <wink>.

>>> Backwards compatibility be damned ;)
>> Not sure what the smiley means; compatibility is taken seriously, and
>> removing things has a very high bar.
> I wanted to express that the change would be good for newbies, but that I 
> don't expect it to happen.
Okay.  You can open a report on bugs.python.org to ask that the doc for
readlines mention list(fp) as an alternative.

Regards


From __peter__ at web.de  Wed Nov 30 18:08:56 2011
From: __peter__ at web.de (Peter Otten)
Date: Wed, 30 Nov 2011 18:08:56 +0100
Subject: [Python-ideas] Is there a reason why
	file.readlines()	doesn't/can't return an iterator?
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org> <4ED650FF.9020705@netwok.org>
	<16CD1DD8-82FC-4DB5-A162-05FC6292E543@masklinn.net>
	<4ED65874.2040201@netwok.org>
Message-ID: <jb5nut$a5e$1@dough.gmane.org>

?ric Araujo wrote:

> Good point.  readlines strips the end-of-line characters though, so to
> use list(fp) I?d have to change all my expected outputs to add \n.

No, you seem to be confusing what_used_to_be_file.readlines() with 
str.splitlines().

>>> open("tmp.txt").readlines()
['alpha\n', 'beta\n', 'gamma\n']
>>> open("tmp.txt").read().splitlines()
['alpha', 'beta', 'gamma']




From solipsis at pitrou.net  Wed Nov 30 18:10:10 2011
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 30 Nov 2011 18:10:10 +0100
Subject: [Python-ideas] Is there a reason why file.readlines()
 doesn't/can't return an iterator?
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org>
	<CAFYqXL9WcWoXeo9-bPooM+4tNhQZy+tpCdRYjffkgvQPtjtjMw@mail.gmail.com>
Message-ID: <20111130181010.005c8eb2@pitrou.net>

On Wed, 30 Nov 2011 16:42:16 +0100
Giampaolo Rodol? <g.rodola at gmail.com>
wrote:
> To be fair, readlines() is different than an iterated file object in
> that it provides the sizehint parameter, so it does cover a use case.
> Maybe the doc should just mention that unless sizehint parameter is
> used the recommended way to get file lines is to iterate against the
> file object itself.

For the record, I don't think I have ever used the sizehint parameter.
It probably serves a purpose but the use cases must be quite
specialized.

Regards

Antoine.




From jkbbwr at gmail.com  Wed Nov 30 18:30:43 2011
From: jkbbwr at gmail.com (Jakob Bowyer)
Date: Wed, 30 Nov 2011 17:30:43 +0000
Subject: [Python-ideas] Making min and max behave more like any and all
Message-ID: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>

I just had something pointed out to me in #python that min and max accept
*args, I know for a fact that any and all only use a single iterable
password.
Shouldn't we allow either one of the two ideas?

Either max/min take only iterable arguments.
OR
Allow any/all to use *args

In the second case this becomes legal

any(1 in a, 2 in b, 3 in c)

In the first case this becomes illegal
max(1,2,3,4,5)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/3f8eac62/attachment.html>

From __peter__ at web.de  Wed Nov 30 18:53:56 2011
From: __peter__ at web.de (Peter Otten)
Date: Wed, 30 Nov 2011 18:53:56 +0100
Subject: [Python-ideas] Is there a reason why
	file.readlines()	doesn't/can't return an iterator?
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org> <4ED650FF.9020705@netwok.org>
	<jb5m5r$vrl$1@dough.gmane.org> <4ED65DF6.3060900@netwok.org>
Message-ID: <jb5qja$3k1$1@dough.gmane.org>

?ric Araujo wrote:

> Okay.  You can open a report on bugs.python.org to ask that the doc for
> readlines mention list(fp) as an alternative.

That would make sense to me only if readlines() were deprecated. I did 
something slightly different, propose a hint that the user should avoid 
readlines() in a for-loop, see

http://bugs.python.org/issue13510



From guido at python.org  Wed Nov 30 19:04:01 2011
From: guido at python.org (Guido van Rossum)
Date: Wed, 30 Nov 2011 10:04:01 -0800
Subject: [Python-ideas] Is there a reason why file.readlines()
 doesn't/can't return an iterator?
In-Reply-To: <4ED65874.2040201@netwok.org>
References: <CAFYqXL9bQxV7c3j9yecRTK-eRKpq2-B2f2_rEqnT8rxKZWOktQ@mail.gmail.com>
	<jb5crq$j0m$1@dough.gmane.org> <4ED650FF.9020705@netwok.org>
	<16CD1DD8-82FC-4DB5-A162-05FC6292E543@masklinn.net>
	<4ED65874.2040201@netwok.org>
Message-ID: <CAP7+vJJxg8vzkO32--4BzOvJ6FQ1OPTNxw523g8wLvQYSA9F1g@mail.gmail.com>

On Wed, Nov 30, 2011 at 8:23 AM, ?ric Araujo <merwok at netwok.org> wrote:

> Good point.  readlines strips the end-of-line characters though,
>

No it doesn't.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/55ad6449/attachment.html>

From ubershmekel at gmail.com  Wed Nov 30 19:06:39 2011
From: ubershmekel at gmail.com (Yuval Greenfield)
Date: Wed, 30 Nov 2011 20:06:39 +0200
Subject: [Python-ideas] Making min and max behave more like any and all
In-Reply-To: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>
References: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>
Message-ID: <CANSw7Kz91unbR5TfSkGAu4ZqYN8NYvbvCc+73TE-o1dTTywSvA@mail.gmail.com>

On Wed, Nov 30, 2011 at 7:30 PM, Jakob Bowyer <jkbbwr at gmail.com> wrote:

> I just had something pointed out to me in #python that min and max accept
> *args, I know for a fact that any and all only use a single iterable
> password.
> Shouldn't we allow either one of the two ideas?
>
>
any(True, False, True)
all(False, True, True)
max(1,2,3,4)
min(1,2,3,4)
sum(1,2,3,4)
list(1,2,3,4)
set(1,2,3,4)

TOOWTDI would mean to allow only one argument in all of the above.
Currently only max/min break the rule.

I'm +1 for making python consistent. And specifically would be +0 for
allowing it all.

--Yuval

Btw even enumerate could have been a candidate though it already has a
second argument.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/d871228c/attachment.html>

From arnodel at gmail.com  Wed Nov 30 19:11:50 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Wed, 30 Nov 2011 18:11:50 +0000
Subject: [Python-ideas] Making min and max behave more like any and all
In-Reply-To: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>
References: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>
Message-ID: <CAJ6cK1bGAECTSDAdCy9RYuvgToEG1JvtmiNffGzyVaZDoqFXrA@mail.gmail.com>

On Nov 30, 2011 5:31 PM, "Jakob Bowyer" <jkbbwr at gmail.com> wrote:
>
> I just had something pointed out to me in #python that min and max accept
*args, I know for a fact that any and all only use a single iterable
password.
> Shouldn't we allow either one of the two ideas?
>
> Either max/min take only iterable arguments.
> OR
> Allow any/all to use *args
>
> In the second case this becomes legal
>
> any(1 in a, 2 in b, 3 in c)

I'd use 'or' here.  For all, I'd use 'and'.

-- 
Arnaud
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/6391bd9c/attachment.html>

From guido at python.org  Wed Nov 30 19:15:14 2011
From: guido at python.org (Guido van Rossum)
Date: Wed, 30 Nov 2011 10:15:14 -0800
Subject: [Python-ideas] Making min and max behave more like any and all
In-Reply-To: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>
References: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>
Message-ID: <CAP7+vJJy8L=VbyEEY7h__QXRRh7tg8fTcyjVcz_jkpgM8ZkeLA@mail.gmail.com>

On Wed, Nov 30, 2011 at 9:30 AM, Jakob Bowyer <jkbbwr at gmail.com> wrote:

> I just had something pointed out to me in #python that min and max accept
> *args, I know for a fact that any and all only use a single iterable
> password.
> Shouldn't we allow either one of the two ideas?
>
> Either max/min take only iterable arguments.
> OR
> Allow any/all to use *args
>

This seems consistency for consistency's sake, not for any particularly
good reason. min/max are quite different from any/all, e.g. min() or max()
without arguments makes no sense, while any() / all() on an empty list has
a perfectly fine meaning (False / True, respectively).

In the second case this becomes legal
>
> any(1 in a, 2 in b, 3 in c)
>

But that's just alternative syntax for

  1 in a or 2 in b or 3 in c

except that it doesn't have shortcut semantics, so it would be an
attractive nuisance. Note that any() / all() *do* have shortcut semantics,
in the sense that they stop iterating over the argument iterable as soon as
they have an answer that forces the outcome (True in the case of any(),
False in the case of all()). This doesn't apply to min() or max() -- you
always have to look at all values before you can decide on which is the
largest. (It would be different if we had a general infinity, but we don't
-- only floating point numbers support infinity).


> In the first case this becomes illegal
> max(1,2,3,4,5)
>

Which I presume is rarely needed but if necessary can easily be written as
max((1, 2, 3, 4, 5)).

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/25de239e/attachment.html>

From masklinn at masklinn.net  Wed Nov 30 19:22:19 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 30 Nov 2011 19:22:19 +0100
Subject: [Python-ideas] Making min and max behave more like any and all
In-Reply-To: <CAP7+vJJy8L=VbyEEY7h__QXRRh7tg8fTcyjVcz_jkpgM8ZkeLA@mail.gmail.com>
References: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>
	<CAP7+vJJy8L=VbyEEY7h__QXRRh7tg8fTcyjVcz_jkpgM8ZkeLA@mail.gmail.com>
Message-ID: <6DA42FE6-124E-44FC-A3F0-2A56EF0E69F2@masklinn.net>

On 2011-11-30, at 19:15 , Guido van Rossum wrote:
>> In the first case this becomes illegal
>> max(1,2,3,4,5)
>> 
> 
> Which I presume is rarely needed but if necessary can easily be written as
> max((1, 2, 3, 4, 5)).
The *args form is rather nice for the (pretty common I'd think) task of min/maxing a pair of values, one know (default) and one unknown for instance.

It is also a common interface for min/max functions.

From guido at python.org  Wed Nov 30 19:26:21 2011
From: guido at python.org (Guido van Rossum)
Date: Wed, 30 Nov 2011 10:26:21 -0800
Subject: [Python-ideas] Making min and max behave more like any and all
In-Reply-To: <6DA42FE6-124E-44FC-A3F0-2A56EF0E69F2@masklinn.net>
References: <CAA+RL7HZ_kh-RRwda8wWruFXQKfSgp+fmmykoKLqVdCP68S7Nw@mail.gmail.com>
	<CAP7+vJJy8L=VbyEEY7h__QXRRh7tg8fTcyjVcz_jkpgM8ZkeLA@mail.gmail.com>
	<6DA42FE6-124E-44FC-A3F0-2A56EF0E69F2@masklinn.net>
Message-ID: <CAP7+vJKyarXjU_a-j9id1sidrem91-mMjwHr4-63ZDe5vYGSqw@mail.gmail.com>

On Wed, Nov 30, 2011 at 10:22 AM, Masklinn <masklinn at masklinn.net> wrote:

> On 2011-11-30, at 19:15 , Guido van Rossum wrote:
> >> In the first case this becomes illegal
> >> max(1,2,3,4,5)
> >>
> >
> > Which I presume is rarely needed but if necessary can easily be written
> as
> > max((1, 2, 3, 4, 5)).
> The *args form is rather nice for the (pretty common I'd think) task of
> min/maxing a pair of values, one know (default) and one unknown for
> instance.
>
> It is also a common interface for min/max functions.


Of course. IMO the status quo is optimal -- max(a, b) is very useful. Not
so for any(a, b) though.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20111130/614ed30c/attachment.html>