[Python-ideas] Simpler thread synchronization using "Sticky Condition"

Richard Whitehead richard.whitehead at ieee.org
Tue Mar 26 07:14:57 EDT 2019


Antoine,

Thanks for a couple of very useful comments.

The usefulness of wait_for(), especially in avoiding the problem of 
spurious-waits, is something I'd forgotten - this is really just syntactic 
sugar, but I agree it is very useful - as illustrated by the fact that I 
didn't remember to put my wait() call in its own loop in my example!

On the other hand, the lack of locking makes this new thing more like an 
Event than a Condition. Also, since it would only make sense for one thread 
to wait on this object, rather than many, that again makes it more like an 
Event.  And Event only has wait(), not wait_for() - it makes no sense to 
"loop" on an object that is "sticky", you would just get livelock. So in 
summary, this new class (or behaviour) would still only have wait() and not 
wait_for(). Does that devalue it, do you think?

I know that C# has AutoResetEvent, but I didn't know it was native to the 
Windows API. I'm pretty sure it is not native to Linux (there are a couple 
of other useful features, such as "Wait on multiple events", that are in 
Windows but not Linux, and therefore are also not in Python). Does this 
affect implementation - would you expect an AutoResetEvent to be implemented 
natively on Windows, rather than being composed out of pure Python using a 
boolean and a Condition?

Regarding the link you sent, I don't entirely agree with the opinion 
expressed: if you try to use a Semaphore for this purpose you will soon find 
that it is "the wrong way round", it is intended to protect resources from 
multiple accesses, not to synchronize those multiple accesses.

Thanks,

Richard


-----Original Message----- 
From: python-ideas-request at python.org
Sent: Tuesday, March 26, 2019 10:49 AM
To: python-ideas at python.org
Subject: Python-ideas Digest, Vol 148, Issue 143

Send Python-ideas mailing list submissions to
python-ideas at python.org

To subscribe or unsubscribe via the World Wide Web, visit
https://mail.python.org/mailman/listinfo/python-ideas
or, via email, send a message with subject or body 'help' to
python-ideas-request at python.org

You can reach the person managing the list at
python-ideas-owner at python.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Python-ideas digest..."


Today's Topics:

   1. The Mailing List Digest Project (Abdur-Rahmaan Janhangeer)
   2. Simpler thread synchronization using "Sticky Condition"
      (Richard Whitehead)
   3. Re: Simpler thread synchronization using "Sticky Condition"
      (Antoine Pitrou)
   4. Re: A directive for indentation type, stricter indentation
      parsing. (Mikhail V)


----------------------------------------------------------------------

Message: 1
Date: Tue, 26 Mar 2019 09:07:15 +0400
From: Abdur-Rahmaan Janhangeer <arj.python at gmail.com>
To: Python <python-list at python.org>, python-ideas
<python-ideas at python.org>
Subject: [Python-ideas] The Mailing List Digest Project
Message-ID:
<CADrxXXmP3-1eyYPNRgaQb1MOcXA8yYVHxxDk6orX85gExR4wRg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

As proposed on python-ideas, i setup a repo to turn mail threads into
articles.

here is the repo

https://github.com/Abdur-rahmaanJ/py-mailing-list-summary

i included a script to build .md to .html (with syntax highlighting) here
is the index

https://abdur-rahmaanj.github.io/py-mailing-list-summary/

included 3 articles as a start

if you want to contribute an article, just follow existing .md format and
put it in the .md folder

planning to go across ideas, list and dev

i can tell you, it's a really enjoyable experience.

psst. we can enhance some html later

-- 
Abdur-Rahmaan Janhangeer
Mauritius
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
<http://mail.python.org/pipermail/python-ideas/attachments/20190326/ab8ee7e8/attachment-0001.html>

------------------------------

Message: 2
Date: Tue, 26 Mar 2019 09:27:18 -0000
From: "Richard Whitehead" <richard.whitehead at ieee.org>
To: <python-ideas at python.org>
Subject: [Python-ideas] Simpler thread synchronization using "Sticky
Condition"
Message-ID: <005301d4e3b6$1c1099b0$5431cd10$@ieee.org>
Content-Type: text/plain; charset="us-ascii"

Problem:

Using Python's Condition class is error-prone and difficult. For example,
the following pseudo-code would be quite typical of its use:



condition = threading.Condition()



def sender():

     while alive():

          wait_for_my_data_from_hardware()

           with condition:

                send_data_to_receiver()

                condition.raise()



def receiver():

     while alive():

          with condition:

                 condition.wait()

                 receive_all_data_from_sender()

                 process_data()



Main code will then run sender() and receiver() in separate threads.



(I know that in a simple case like this, I could just use a Queue. I've been
working with code that needs to service several events, where polling queues
would introduce overhead and latency so a Condition must be used, but I've
simplified the above as much as possible to illustrate the issues even in
this simple case).



There are two issues even with the above case:

1. Raising a condition only has any effect if the condition is already being
waited upon; a condition does not "remember" that it has been raised. So in
the example above, if the receiver starts after the sender, it will wait on
the condition even though data is already available for it.  This issue can
be solved by rearranging the code, waiting at the end of the loop rather
than the start; but this is counter-intuitive and lots of examples online
look like the above.

2. In the receiver, the condition has to be kept locked (be inside the
"with" statement") all the time. The lock is automatically released while
the condition is being waited upon, but otherwise it must be locked,
otherwise we risk missing the condition being raised. The result is that
process_data() is called with the lock held - and so this will prevent the
sender going round its loop. The sending thread becomes a slave to the
processing thread. This may have a huge performance penalty, losing the
advantage of loose coupling that threading should provide.



You might think that using an Event, rather than a Condition, would solve
things, since an Event does "remember" that is has been set. But there is no
atomic way to wait on an event and clear it afterwards.



Solution:

The solution is very simple: to create something that might be called a
StickyCondition, or an AutoResetEvent. This is a Condition that does
remember it has been raised; or if you prefer, it is an Event that resets
itself after it has been waited upon.  The example then becomes:



auto_event = threading.AutoResetEvent()



def sender():

     while alive():

           wait_for_my_data_from_hardware()

           send_data_to_receiver()

           auto_event.set()



def receiver():

     while alive():

           auto_event.wait()

           receive_all_data_from_sender()

           process_data()



This solves both of the issues described: the receiver will not wait if data
is already available, and the sender is not blocked by a lock that is being
held by the receiver. The event doesn't need to be locked at all, because of
its internal memory.



Implementation is trivial, involving a boolean and a Condition in the
cross-thread case. I presume it would be more involved for the cross-process
case, but I can see no reason it would be impossible.



Please let me know if you think this would be useful.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
<http://mail.python.org/pipermail/python-ideas/attachments/20190326/e1d826d2/attachment-0001.html>

------------------------------

Message: 3
Date: Tue, 26 Mar 2019 11:38:14 +0100
From: Antoine Pitrou <solipsis at pitrou.net>
To: python-ideas at python.org
Subject: Re: [Python-ideas] Simpler thread synchronization using
"Sticky Condition"
Message-ID: <20190326113814.4abe0332 at fsol>
Content-Type: text/plain; charset=US-ASCII

On Tue, 26 Mar 2019 09:27:18 -0000
"Richard Whitehead" <richard.whitehead at ieee.org>
wrote:
> Problem:
>
> Using Python's Condition class is error-prone and difficult. For example,
> the following pseudo-code would be quite typical of its use:
[...]

Nowadays, I would recommend to always use `Condition.wait_for()` rather
than `Condition.wait()`.

A Condition, unlike what the name suggests, is just a means of
synchronization.  It doesn't have a boolean state per se.  You have to
manage your boolean state (or any other kind of state) separately, hence
the usefulness of `wait_for()`.

As for auto-reset events, the Windows API has them, here's an
insightful writeup about them:
https://devblogs.microsoft.com/oldnewthing/?p=30773

But, yes, perhaps auto-reset events would be a good addition
regardless.  Either through a flag to Event, or as a separate class.

Regards

Antoine.




------------------------------

Message: 4
Date: Tue, 26 Mar 2019 13:48:48 +0300
From: Mikhail V <mikhailwas at gmail.com>
Cc: python-ideas <python-ideas at python.org>
Subject: Re: [Python-ideas] A directive for indentation type, stricter
indentation parsing.
Message-ID:
<CAE6NSRBnNNiN3BiZd8Zo9DCjae_8ar91YjXX6nktGrbBVskeSw at mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"

On Tue, Mar 26, 2019 at 7:04 AM fhsxfhsx <fhsxfhsx at 126.com> wrote:
>
> Just as to your example, you can try `textwrap.dedent`
>
Procedural removal is not cool, because it does not know the parent 
indentation
of the statement that contains the text block, thus it can't resolve
automatically
string indents  that were intentionally indented to include extra space.

E.g. this, where "s=" line is already inside an indented block:
    s = """
                Hello
            world"""

Say I use 1 tab for 1 indent - what I want here is to remove 1 tab from
each string line AND 1 tab that belongs to code formatting (current indent
of the "s=" line). And if you try to guess it from the string  itself -
it is impossible to resolve all cases, because you still need some criteria
- for example you could use criteria "minimal indent inside the string
is the indent" but
this will again fail if you want extra shift on same level inside the 
string.
E.g. this:
    s = """
                Hello
                world"""

Here I do not want to remove all indents but only as in above - only 1
level inside string
and 1 from parent line.
Do you get it?

So in other words if I want my string blocks aligned within containing
blocks, it  becomes
impossible to resolve all un-indenting cases automatically.


Mikhail


------------------------------

Subject: Digest Footer

_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
https://mail.python.org/mailman/listinfo/python-ideas


------------------------------

End of Python-ideas Digest, Vol 148, Issue 143
********************************************** 



More information about the Python-ideas mailing list