[ python-Feature Requests-1453341 ] sys.setatomicexecution - for critical sections

Sat Apr 8 20:34:13 CEST 2006

Feature Requests item #1453341, was opened at 2006-03-19 05:52
Message generated for change (Comment added) made by ncoghlan
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1453341&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: None
Status: Closed
Resolution: Rejected
Priority: 5
Submitted By: kxroberto (kxroberto)
Assigned to: Nobody/Anonymous (nobody)
Summary: sys.setatomicexecution -  for critical sections

Initial Comment:
In order to maintain threaded code uncomplicated (VHL)
and in order to avoid massive use of locks, when
introducing few critical non-atomic sections, it should
be possible to put out that practical hammer ..

    try:
        last_ci=sys.setcheckinterval(sys.maxint)
        critical_function()  # now runs atomically
    finally:
        sys.setcheckinterval(last_ci)
    #(sys.setcheckinterval assumed to return the last
value)

..by an official function/class (in sys or thread):

Maybe:
======

atomic = sys.setatomicexecution(mode)
try:
    print "Executing critical section"
finally:
    sys.setatomicexecution(atomic)

there should be different modes/levels for blocking :

* 0=None
* 1=block only python execution in other threads
* 2=block signals
* 4/5/7/15=block threads at OS level (if OS supports)
* 8=(reserved: block future sub-/stackless switching
inside current thread..)

see:
http://groups.google.de/group/comp.lang.python/browse_frm/thread/bf5461507803975e/3bd7dfa9422a1200

compare:
http://www.stackless.com/wiki/Tasklets#critical-sections

---

Also Python could define officially its time atoms
beyond CPU atoms in the docs (That also defines the VHL
usability of the language).

Thus thinks like single-element changes ...

obj.var=x  , d[key]=x , l.append(x) .pop()

should be guaranteed to work atomic/correct FOR EVER. 

l[slice],  d.keys() .items(), .. questionable? 
If not guaranteed for the future, offer duplicates for
speed critical key building blocks like: 
l.slice_atomic(slice), d.keys_atomic() ,...  in order
to make code compatible for the future. 

---

Extra fun for blowing up python libs for poeple who
don't want to learn that try..finally all the time:

copy/deepcopy/dump maybe could be enriched by
copy_atomic , deepcopy_atomic, dump_atomic - or just
RuntimeError tolerant versions,   deepcopy_save  (no
use of .iterxxx)

----------------------------------------------------------------------

>Comment By: Nick Coghlan (ncoghlan)
Date: 2006-04-09 04:34

Message:
Logged In: YES 
user_id=1038590

On the other changes you suggest (which Raymond didn't bring
up on python-dev):

Python can't formally define as atomic any operations that
may execute arbitrary Python code, as the interpreter cannot
control what that code may do. All of the examples you give
are in that category.

Slowing down the common cases (unthreaded code and
Queue-based threaded code) by adding internal locking to
every data structure is also considered highly undesirable.

----------------------------------------------------------------------

Comment By: Nick Coghlan (ncoghlan)
Date: 2006-04-09 04:17

Message:
Logged In: YES 
user_id=1038590

Raymond brought this idea up on python-dev at the time of
the c.l.p discussion - it was rejected on the basis that
thread synchronisation tools (Queues and Locks) are provided
for a reason. Python level access to the Global Interpreter
Lock is neither necessary nor desirable.

Avoiding the tools provided to permit threading to work
correctly and then finding that threaded code doesn't work
as desired really shouldn't be surprising.

FWIW, Python 2.5 aims to make normal locking easier to use
by permitting:

from __future__ import with_statement
from threading import Lock

sync_lock = Lock()

def my_func(*args, **kwds):
    with sync_lock:
        # Only one thread at a time can enter this section
        # regardless of IO or anything else
    # This section, on the other hand, is a free-for-all

If you genuinely have to avoid normal thread synchronisation
primitives, you can abuse (and I really do mean abuse) the
interpreter's import lock for this purpose:

imp.acquire_lock()
try: 
    print 'critical section'
finally:
    imp.release_lock()

Or even:

@contextlib.contextmanager
def CriticalSection()
    imp.acquire_lock()
    try:
        yield
    finally:
        imp.release_lock()

with CriticalSection():
    print 'critical section'

----------------------------------------------------------------------

Comment By: kxroberto (kxroberto)
Date: 2006-03-21 01:28

Message:
Logged In: YES 
user_id=972995

... only PyEval_RestoreThread with the harder execution
level in its tstate

----------------------------------------------------------------------

Comment By: kxroberto (kxroberto)
Date: 2006-03-21 01:24

Message:
Logged In: YES 
user_id=972995

thus the GIL could simply have a harder state 2 : "locked
hard for PyEval_AcquireThread/PyEval_AcquireLock/.." ?

Only PyEval_RestoreThread gets the lock again after
PyEval_SaveThread.

Robert

----------------------------------------------------------------------

Comment By: Martin Gfeller (gfe)
Date: 2006-03-20 22:17

Message:
Logged In: YES 
user_id=884167

- sys.setcheckinterval(sys.maxint) does not prevent 
  thread switching when doing IO, does it? There is
  no way that I know of to prevent thread switching
  in this situation.

- When calling back into Python from C Code, there is
  no way to tell Python not to relinquish the GIL; such
  an option could be useful for some C code, because
  the GIL provides a "global critical section". 

Martin

----------------------------------------------------------------------

Comment By: kxroberto (kxroberto)
Date: 2006-03-19 07:54

Message:
Logged In: YES 
user_id=972995

or:

>>> atomic = sys.acquireatomicexecution(mode)
>>> try: 
... 	print 'critical section'
... finally:
... 	atomic.release()

>>> with sys.acquireatomicexecution(mode):
...     print 'critical section'

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1453341&group_id=5470