Let's imagine I have an algorithm which depends on a context variable. I write an algorithm (elided below for space) which depends on it. Then I realize that I can improve the performance of my algorithm by using concurrent.futures. But my algorithm will change its behaviour because it does not inherit the context. Simply trying to parallelize an "embarrassingly parallel" algorithm changes its behaviour. What I really want is a stack-scoped variable: a variable which retains its value for all child scopes whether in the same thread or not, unless it is overwritten in a child scope (whether in the same thread or not). from decimal import getcontext import concurrent.futures def my_algorithm(input): # some real algorithm here, which relies on decimal precision return getcontext().prec getcontext().prec = 8 vals = map(my_algorithm, range(0, 10)) print(list(vals)) with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: # Start the load operations and mark each future with its URL results = executor.map(my_algorithm, range(0, 10)) print(list(results)) Results: [8, 8, 8, 8, 8, 8, 8, 8, 8, 8] [28, 28, 28, 28, 28, 28, 28, 28, 28, 28]
On 18Aug2021 16:30, Paul Prescod <prescod@gmail.com> wrote:
Do you mean a facility for setting an attribute on an object and reversing that when you exit the scope where you set it? I've got a stackattrs context manager I use for that. Example use: # this is on PyPI from cs.context import stackattrs with stackattrs(getcontext(), prec=8): # .prec=8 vals = map(my_algorithm, range(0, 10)) # .prec is whatever it used to be, including absent Otherwise it is not clear to me what you are after. If this isn't what you want, can you describe what's different? Cheers, Cameron Simpson <cs@cskk.id.au>
There are certain values that are managed independent of any specific code-accessible object. The decimal precision is a good example: https://docs.python.org/3/library/decimal.html We can call these context variables. As your code shows, the true "home" of the "prec" value is not some object you manage in your code, but rather some background object called the Context. The Context is not inherited by sub-threads. Your code is fine, because you did not use threads. Replace the map with concurrent.futures code (as in my example). Your code will not work properly anymore. This is the issue.
Perhaps we need a library for creating/managing threads that inherits all current context values? On Thu, Aug 19, 2021 at 7:48 AM Paul Prescod <prescod@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
asyncio.to_thread creates threads that inherit the current context, according to https://www.python.org/dev/peps/pep-0567/#rationale the decimal module should use contextvars for this too
On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum <guido@python.org> wrote:
Perhaps we need a library for creating/managing threads that inherits all current context values?
Or is it a "kind of context variable that is shared among threads?" That was more the direction my mind was going. Context variables that hold explicitly or implicitly immutable values are safe to share. Context variables that hold values intended to be mutated: seldom. What if it were some kind of ContextVar subclass or flag where you would say you did or didn't want something shared? And for legacy modules like decimal, you could turn on the flag somehow to get the sharing behaviour. Sometimes threads are made by third-party libraries and those libraries don't know anything about the context. In my case, the pattern was: 1. my code set context variable 2. my code configured a third party library with a callback 3. the third party library used threading for perf reasons 4. it called my callback, which failed due to the context variable failing to be inherited I believe (after only thinking about it for a day or so) that one can generally determine the right "sharing rule" at the point of definition of a ContextVar, based on the semantics and type of the Var. On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum <guido@python.org> wrote:
Oh, I like that. It does feel like a property of the variable. (But can you efficiently enumerate all context vars when creating a thread?) I imagine that thread pools add some complication, because you don’t want to inherit these accidentally between tasks run by the same worker. On Thu, Aug 19, 2021 at 14:28 Paul Prescod <prescod@gmail.com> wrote:
On 18Aug2021 16:30, Paul Prescod <prescod@gmail.com> wrote:
Do you mean a facility for setting an attribute on an object and reversing that when you exit the scope where you set it? I've got a stackattrs context manager I use for that. Example use: # this is on PyPI from cs.context import stackattrs with stackattrs(getcontext(), prec=8): # .prec=8 vals = map(my_algorithm, range(0, 10)) # .prec is whatever it used to be, including absent Otherwise it is not clear to me what you are after. If this isn't what you want, can you describe what's different? Cheers, Cameron Simpson <cs@cskk.id.au>
There are certain values that are managed independent of any specific code-accessible object. The decimal precision is a good example: https://docs.python.org/3/library/decimal.html We can call these context variables. As your code shows, the true "home" of the "prec" value is not some object you manage in your code, but rather some background object called the Context. The Context is not inherited by sub-threads. Your code is fine, because you did not use threads. Replace the map with concurrent.futures code (as in my example). Your code will not work properly anymore. This is the issue.
Perhaps we need a library for creating/managing threads that inherits all current context values? On Thu, Aug 19, 2021 at 7:48 AM Paul Prescod <prescod@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
asyncio.to_thread creates threads that inherit the current context, according to https://www.python.org/dev/peps/pep-0567/#rationale the decimal module should use contextvars for this too
On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum <guido@python.org> wrote:
Perhaps we need a library for creating/managing threads that inherits all current context values?
Or is it a "kind of context variable that is shared among threads?" That was more the direction my mind was going. Context variables that hold explicitly or implicitly immutable values are safe to share. Context variables that hold values intended to be mutated: seldom. What if it were some kind of ContextVar subclass or flag where you would say you did or didn't want something shared? And for legacy modules like decimal, you could turn on the flag somehow to get the sharing behaviour. Sometimes threads are made by third-party libraries and those libraries don't know anything about the context. In my case, the pattern was: 1. my code set context variable 2. my code configured a third party library with a callback 3. the third party library used threading for perf reasons 4. it called my callback, which failed due to the context variable failing to be inherited I believe (after only thinking about it for a day or so) that one can generally determine the right "sharing rule" at the point of definition of a ContextVar, based on the semantics and type of the Var. On Thu, Aug 19, 2021 at 8:43 AM Guido van Rossum <guido@python.org> wrote:
Oh, I like that. It does feel like a property of the variable. (But can you efficiently enumerate all context vars when creating a thread?) I imagine that thread pools add some complication, because you don’t want to inherit these accidentally between tasks run by the same worker. On Thu, Aug 19, 2021 at 14:28 Paul Prescod <prescod@gmail.com> wrote:
participants (4)
-
Cameron Simpson -
Guido van Rossum -
Paul Prescod -
Thomas Grainger