Does anybody really use frame->f_tstate ?
Hi colleagues, this is my second attempt to get rid of the f_tstate field in frames. I need to find every user of this field. What am I talking about? Well, Python always has a thread state variable which is a unique handle to the current thread. This variable is accessed in many places, and there exists a fast macro to get at it. Every executing Python frame also gets a copy on creation. In some cases, this frame->f_tstate field is used, in other cases the current tstate variable is used. If this sounds foreign to you, please stop reading here. ------------------------------------------------------------- I would like to get rid of the frame->f_tstate, and I'm trying to find out if there is a need for it. I don't need it, for Stackless, it is the opposite, it disturbs. There was a small thread about this in June this year, where Guido convinced me that it is possible to create a traceback on a frame that comes from a different thread. http://mail.python.org/pipermail/python-dev/2003-June/036254.html Ok, this is in fact possible, although I don't think anybody has a need for this. My question to all extension writers is this: If you use frame->f_tstate at all, do you use it just because it is handy, or do you want to use it for some other purpose? One purpose could be that you really want to create a traceback on a different than the current thread. I have never seen this, but who knows, so that's why I'm asking the Python world. In most cases, a traceback will be created on a frame that is currently processd or just has been processed. Accessing a frame of a different thread that is being processed might make sense for special debugger cases. My proposal is -------------- a) change semantics of PytraceBack_Here to use the current tstate. b) if such a special purpose exists, create a new function for it. c) if urgent, different needs exist to keep f_tstate, then let's forget about this proposal. Especially for Stackless, I'd be keen of getting rid of this. thanks for input -- chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Dear Python community, since I didn't get *any* reply to this request, either the request was bad or there is really nobody using f_tstate in a way that makes it urgent to keep. I will wait a few hours and then make the change to Stackless, and I'd like to propose to do the same to the Python core. Christian Tismer wrote:
Hi colleagues,
this is my second attempt to get rid of the f_tstate field in frames. I need to find every user of this field.
What am I talking about? Well, Python always has a thread state variable which is a unique handle to the current thread. This variable is accessed in many places, and there exists a fast macro to get at it. Every executing Python frame also gets a copy on creation. In some cases, this frame->f_tstate field is used, in other cases the current tstate variable is used.
If this sounds foreign to you, please stop reading here.
-------------------------------------------------------------
I would like to get rid of the frame->f_tstate, and I'm trying to find out if there is a need for it. I don't need it, for Stackless, it is the opposite, it disturbs.
There was a small thread about this in June this year, where Guido convinced me that it is possible to create a traceback on a frame that comes from a different thread.
http://mail.python.org/pipermail/python-dev/2003-June/036254.html
Ok, this is in fact possible, although I don't think anybody has a need for this.
My question to all extension writers is this: If you use frame->f_tstate at all, do you use it just because it is handy, or do you want to use it for some other purpose?
One purpose could be that you really want to create a traceback on a different than the current thread. I have never seen this, but who knows, so that's why I'm asking the Python world.
In most cases, a traceback will be created on a frame that is currently processd or just has been processed. Accessing a frame of a different thread that is being processed might make sense for special debugger cases.
My proposal is --------------
a) change semantics of PytraceBack_Here to use the current tstate.
b) if such a special purpose exists, create a new function for it.
c) if urgent, different needs exist to keep f_tstate, then let's forget about this proposal.
Especially for Stackless, I'd be keen of getting rid of this.
thanks for input -- chris
-- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
since I didn't get *any* reply to this request, either the request was bad or there is really nobody using f_tstate in a way that makes it urgent to keep. I will wait a few hours and then make the change to Stackless, and I'd like to propose to do the same to the Python core.
I saved the message, but haven't had the time yet to think things through. I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote: ...
I saved the message, but haven't had the time yet to think things through.
I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated.
Right, f_tstate is never updated. I think there is another inconsistent situation, which can be created easily. If a generator is run by a different thread than it's creator, then the frame is run in that other thread. eval_fame correctly uses tstate, but if tracing is activated, call_trace uses frame->f_tstate for no obvious reason, which will probably mess up the tracing flags of the wrong thread. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated.
Right, f_tstate is never updated. I think there is another inconsistent situation, which can be created easily. If a generator is run by a different thread than it's creator, then the frame is run in that other thread. eval_fame correctly uses tstate, but if tracing is activated, call_trace uses frame->f_tstate for no obvious reason, which will probably mess up the tracing flags of the wrong thread.
Right. Could you dig through CVS logs to find out when f_tstate was first introduced? Maybe there's a clue about why there. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote: ...
Could you dig through CVS logs to find out when f_tstate was first introduced? Maybe there's a clue about why there.
Yeehaa! Quite some ride... Well, f_tstate was introduced here, in frameobject.c: ----------------------------------------------------- Revision : 2.29 Date : 1997/5/5 20:55:52 Author : 'guido' State : 'Exp' Lines : +13 -2 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. For frameobject.h, it was this version: --------------------------------------- Revision : 2.23 Date : 1997/5/5 20:55:43 Author : 'guido' State : 'Exp' Lines : +3 -1 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Especially interesting: There is no single hint mentioning f_tstate. But, the diff shows PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; added at the same time. This was Python 1.5, I remember very well, the time where threading became the standard for Python. ceval.c was at this revision: ----------------------------- Revision : 2.118 Date : 1997/5/5 20:55:58 Author : 'guido' State : 'Exp' Lines : +263 -151 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. I fetched ceval 2.117 and 2.118 and compared them. This was exactly the time when the thread state was introduced. There were exactly two references to f_tstate: The most interesting diff for this question: PyObject * PyEval_SaveThread() { #ifdef WITH_THREAD if (interpreter_lock) { - PyObject *res; - res = (PyObject *)current_frame; - current_frame = NULL; + PyThreadState *tstate = PyThreadState_Swap(NULL); + PyObject *res = tstate ? (PyObject *) (tstate->frame) : NULL; release_lock(interpreter_lock); return res; } #endif return NULL; } void PyEval_RestoreThread(x) PyObject *x; { #ifdef WITH_THREAD if (interpreter_lock) { int err; err = errno; acquire_lock(interpreter_lock, 1); errno = err; - current_frame = (PyFrameObject *)x; + PyThreadState_Swap(x ? ((PyFrameObject *)x)->f_tstate : NULL); } #endif } PyEval_SaveThread has changed a bit meanwhile, no longer return the frame, but tstate. semantics are similar. The two functions clearly show hwo the transition from a global current_frame variable was made to the tstate mechanism. f_tstate was the source for switching the thread state. The current ceval version looks like this: void PyEval_RestoreThread(PyThreadState *tstate) { if (tstate == NULL) Py_FatalError("PyEval_RestoreThread: NULL tstate"); #ifdef WITH_THREAD if (interpreter_lock) { int err = errno; PyThread_acquire_lock(interpreter_lock, 1); errno = err; } #endif PyThreadState_Swap(tstate); } That means, tstate is not fetched from the frame, but given as a parameter. I checked all places where this function gets called, the tstate parameter never comes from a frame. As a conclusion, f_tstate appears to be an intermediate step during the move to threading. It was an easy place to keep the thread state. Its use was reduced to very few later on. call_trace introduced today's slightly false behavior exactly in that 2.118 version. It looks very much like an optimization, since there was no macro for fetching tstate at that time. This was perfectly fine, until generators were introduced. Until then, a frame was always bound to its thread. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated.
Right, f_tstate is never updated. I think there is another inconsistent situation, which can be created easily. If a generator is run by a different thread than it's creator, then the frame is run in that other thread. eval_fame correctly uses tstate, but if tracing is activated, call_trace uses frame->f_tstate for no obvious reason, which will probably mess up the tracing flags of the wrong thread.
Right.
Could you dig through CVS logs to find out when f_tstate was first introduced? Maybe there's a clue about why there.
Here is a head start on the research. The ceval.c use of tstate goes back to the introduction of generators in 2001. The use in traceback.c and sysmodule.c goes back to 1997 when per-thread globals were factored into a structure to support separate thread-state management. Prior to that, the history is more diffuse and harder to follow. Raymond Hettinger --------------------------------------------------------------------- Ceval.c -------- 2.252 (nascheme 21-Jun-01): PyThreadState *tstate = 2.252 (nascheme 21-Jun-01): * necessarily their creator. */ 2.255 (tim_one 23-Jun-01): Py_XINCREF(tstate->frame); 2.252 (nascheme 21-Jun-01): assert(f->f_back == NULL); 2.252 (nascheme 21-Jun-01): f->f_back = tstate->frame; 2.252 (nascheme 21-Jun-01): revision 2.255 date: 2001/06/23 05:47:56; author: tim_one; state: Exp; lines: +2 -2 gen_iternext(): Don't assume that the current thread state's frame is not NULL. I don't think it can be NULL from Python code, but if using generators via the C API I expect a NULL frame is possible. revision 2.252 date: 2001/06/21 02:41:10; author: nascheme; state: Exp; lines: +27 -14 Try to avoid creating reference cycles involving generators. Only keep a reference to f_back when its really needed. Do a little whitespace normalization as well. This whole file is a big war between tabs and spaces but now is probably not the time to reindent everything. traceback.c ------------ 2.22 (guido 05-May-97): PyThreadState *tstate = frame->f_tstate; 2.22 (guido 05-May-97): tracebackobject *oldtb = (tracebackobjec revision 2.22 date: 1997/05/05 20:56:15; author: guido; state: Exp; lines: +6 -30 Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. The story for their life a globals pre-dates what I can sysmodule.c ------------ 2.45 (guido 02-Aug-97): PyThreadState *tstate = PyThreadState_Ge 2.45 (guido 02-Aug-97): PyObject *sd = tstate->interp->sysdict; 2.57 (guido 05-Oct-99): if (sd == NULL) 2.41 (guido 05-May-97): PyThreadState *tstate; 2.41 (guido 05-May-97): tstate = PyThreadState_Get(); 2.41 (guido 05-May-97): return Py_BuildValue( 2.41 (guido 05-May-97): "(OOO)", 2.41 (guido 05-May-97): tstate->exc_type != NULL ? tstat 2.41 (guido 05-May-97): tstate->exc_value != NULL ? tsta 2.41 (guido 05-May-97): tstate->exc_traceback != NULL ? 2.41 (guido 05-May-97): tstate->exc_traceback : 2.41 (guido 05-May-97): } 2.41 (guido 05-May-97): revision 2.41 date: 1997/05/05 20:56:11; author: guido; state: Exp; lines: +30 -9 Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Details of the 2.41 checkin ---------------------------- Index: sysmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/sysmodule.c,v retrieving revision 2.40 retrieving revision 2.41 diff -c -r2.40 -r2.41 *** sysmodule.c 29 Apr 1997 20:42:30 -0000 2.40 --- sysmodule.c 5 May 1997 20:56:11 -0000 2.41 *************** *** 48,56 **** #include "osdefs.h" - PyObject *_PySys_TraceFunc, *_PySys_ProfileFunc; - int _PySys_CheckInterval = 10; - #if HAVE_UNISTD_H #include <unistd.h> #endif --- 48,53 ---- *************** *** 98,103 **** --- 95,119 ---- } static PyObject * + sys_exc_info(self, args) + PyObject *self; + PyObject *args; + { + PyThreadState *tstate; + if (!PyArg_Parse(args, "")) + return NULL; + tstate = PyThreadState_Get(); + if (tstate == NULL) + Py_FatalError("sys.exc_info(): no thread state"); + return Py_BuildValue( + "(OOO)", + tstate->exc_type != NULL ? tstate->exc_type : Py_None, + tstate->exc_value != NULL ? tstate->exc_value : Py_None, + tstate->exc_traceback != NULL ? + tstate->exc_traceback : Py_None); + } + + static PyObject * sys_exit(self, args) PyObject *self; PyObject *args; *************** *** 112,123 **** PyObject *self; PyObject *args; { if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(_PySys_TraceFunc); ! _PySys_TraceFunc = args; Py_INCREF(Py_None); return Py_None; } --- 128,140 ---- PyObject *self; PyObject *args; { + PyThreadState *tstate = PyThreadState_Get(); if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(tstate->sys_tracefunc); ! tstate->sys_tracefunc = args; Py_INCREF(Py_None); return Py_None; } *************** *** 127,138 **** PyObject *self; PyObject *args; { if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(_PySys_ProfileFunc); ! _PySys_ProfileFunc = args; Py_INCREF(Py_None); return Py_None; } --- 144,156 ---- PyObject *self; PyObject *args; { + PyThreadState *tstate = PyThreadState_Get(); if (args == Py_None) args = NULL; else Py_XINCREF(args); ! Py_XDECREF(tstate->sys_profilefunc); ! tstate->sys_profilefunc = args; Py_INCREF(Py_None); return Py_None; } *************** *** 142,148 **** PyObject *self; PyObject *args; { ! if (!PyArg_ParseTuple(args, "i", &_PySys_CheckInterval)) return NULL; Py_INCREF(Py_None); return Py_None; --- 160,167 ---- PyObject *self; PyObject *args; { ! PyThreadState *tstate = PyThreadState_Get(); ! if (!PyArg_ParseTuple(args, "i", &tstate->sys_checkinterval)) return NULL; Py_INCREF(Py_None); return Py_None; *************** *** 202,207 **** --- 221,227 ---- static PyMethodDef sys_methods[] = { /* Might as well keep this in alphabetic order */ + {"exc_info", sys_exc_info, 0}, {"exit", sys_exit, 0}, #ifdef COUNT_ALLOCS {"getcounts", sys_getcounts, 0}, *************** *** 232,238 **** if (list == NULL) return NULL; for (i = 0; _PyImport_Inittab[i].name != NULL; i++) { ! PyObject *name = PyString_FromString(_PyImport_Inittab[i].name); if (name == NULL) break; PyList_Append(list, name); --- 252,259 ---- if (list == NULL) return NULL; for (i = 0; _PyImport_Inittab[i].name != NULL; i++) { ! PyObject *name = PyString_FromString( ! _PyImport_Inittab[i].name); if (name == NULL) break; PyList_Append(list, name);
Raymond Hettinger wrote: ...
Here is a head start on the research.
The ceval.c use of tstate goes back to the introduction of generators in 2001.
The use in traceback.c and sysmodule.c goes back to 1997 when per-thread globals were factored into a structure to support separate thread-state management.
Prior to that, the history is more diffuse and harder to follow.
Thanks a lot. I did a similar research on Friday, but for some reason it was not accepted by python.org, as it appears. (why, only 4 k, yours was 10?) I will try to send it again. My summary: f_tstate not needed! ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
[Christian Tismer]
I did a similar research on Friday, but for some reason it was not accepted by python.org, as it appears. (why, only 4 k, yours was 10?)
Your original was accepted: http://mail.python.org/pipermail/python-dev/2003-December/041287.html I think there may have been a long delay in sending it out, though, perhaps related to the python.org domain transfer, or to hardware fiddling recently done at mail.python.org's physical site. Whichever, it's in the archive, and I got it via email too.
My summary: f_tstate not needed!
More, it seems conceptually flawed, albeit subtly. Python's runtime is, in effect, simulating thread-local storage by hand, and the current value of _PyThreadState_Current always points to a PyThreadState struct holding the currently-executing thread's *conceptual* thread-local storage. If we were able to use honest-to-goodness TLS instead, it wouldn't have been possible (well, not w/o ugly hacks) for a frame to use the values of these guys associated with any thread other than the one currently executing. Or, IOW, the existence of f_tstate creates a possibility for inter-thread mixups. Still, the possibility to switch threads across generator resumptions seems darned hard to view as "a feature". I'd advise people not to rely on it <wink>.
Tim Peters wrote: ...
Your original was accepted:
http://mail.python.org/pipermail/python-dev/2003-December/041287.html
Yup, I wasn't aware of the DNS change.
My summary: f_tstate not needed!
More, it seems conceptually flawed, albeit subtly.
... Very much agreed. I always felt that f_tstate was not quite ok. You spelled it and supplied more reason.
Still, the possibility to switch threads across generator resumptions seems darned hard to view as "a feature". I'd advise people not to rely on it <wink>.
Well, this might only work for generators if it is guaranteed that the thread switch doesn't happen while the generator is executing. I think this will give an exception. But most probably the result will be hard to predict. Other for my "tasklet" tiny threads. Their first purpose is to have concurrent small tasks without interaction (or if with interaction, then via channels and blocking). A tasklet being interrupted by a thread switch will not be auto-schedulable until the thread is resumed, but other tasklets might be, and they don't need to care about which real thread is actually in charge. This is one of my reasons why I want to get rid of f_tstate -- it does not really apply to most of my frames. Being tied to a thread is a property of the tasklet, and it does not apply all the time, only when I have to keep C state. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
[Tim]
Still, the possibility to switch threads across generator resumptions seems darned hard to view as "a feature". I'd advise people not to rely on it <wink>.
[Christian Tismer]
Well, this might only work for generators if it is guaranteed that the thread switch doesn't happen while the generator is executing. I think this will give an exception. But most probably the result will be hard to predict.
I'm not sure I'm following, but nothing in Python now *stops* you from resuming a suspended generator from another thread. Here's an extreme example of that: """ import threading def myid(): while True: yield threading.currentThread() next = myid().next mut = threading.Lock() class Worker(threading.Thread): def run(self): while True: mut.acquire() print next() mut.release() for i in range(3): Worker().start() """ Then the body of the single generator is executed by 3 different threads, more or less randomly, although never by the thread that created the generator. If the body of the generator were relying on thread-local state (like, e.g., the mass of settings in the decimal arithmetic package's per-thread default "numeric context" object), the result would sure be unpredictable. It wouldn't crash Python, it would just be a bitch to debug. Then again, lots of bad threading practices lead to bitch-to-debug code, so I'm not really inclined to offer extra protection here.
Other for my "tasklet" tiny threads. Their first purpose is to have concurrent small tasks without interaction (or if with interaction, then via channels and blocking). A tasklet being interrupted by a thread switch will not be auto-schedulable until the thread is resumed, but other tasklets might be, and they don't need to care about which real thread is actually in charge. This is one of my reasons why I want to get rid of f_tstate -- it does not really apply to most of my frames. Being tied to a thread is a property of the tasklet, and it does not apply all the time, only when I have to keep C state.
If a tasklet can be affected by thread-local state (like numeric context -- Python has little of this nature now, but it's bound to increase), then a tasklet will also start to care "who's in charge". So I expect a patch to reintroduce f_tstate in about a year <wink>.
Tim Peters wrote:
[Tim]
Still, the possibility to switch threads across generator resumptions seems darned hard to view as "a feature". I'd advise people not to rely on it <wink>.
[Christian Tismer]
Well, this might only work for generators if it is guaranteed that the thread switch doesn't happen while the generator is executing. I think this will give an exception. But most probably the result will be hard to predict.
Explaining that: I was talking about freely using the generator in another thread, which would most likely resume it while it is active, and that's an exception.
I'm not sure I'm following, but nothing in Python now *stops* you from resuming a suspended generator from another thread. Here's an extreme example of that:
[XTreme example]
Then the body of the single generator is executed by 3 different threads, more or less randomly, although never by the thread that created the generator.
If the body of the generator were relying on thread-local state (like, e.g., the mass of settings in the decimal arithmetic package's per-thread default "numeric context" object), the result would sure be unpredictable. It wouldn't crash Python, it would just be a bitch to debug. Then again, lots of bad threading practices lead to bitch-to-debug code, so I'm not really inclined to offer extra protection here.
I agree, sure, cross-thread generators need to be designed for that. [tasklets]
If a tasklet can be affected by thread-local state (like numeric context -- Python has little of this nature now, but it's bound to increase), then a tasklet will also start to care "who's in charge". So I expect a patch to reintroduce f_tstate in about a year <wink>.
It is already there, not in the frames, but in the tasklet. A tasklet can be thread-bound or not. In that case, it keeps a reference to the thread state. But only once, since that information is not local for frames. Being thead bound is either enforced in the case that the tasklet is carrying non-trivial C state, otherwise it can be set as a user option (conservative default, of course). "Nomad threads" might be nice for some background monitoring. This is all theory, I haven't tried any real threads with Stackless so far. But people are mixing tasklets with threads already, so I have to care *somehow*. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
<this message didn't make it> Guido van Rossum wrote: ...
Could you dig through CVS logs to find out when f_tstate was first introduced? Maybe there's a clue about why there.
Yeehaa! Quite some ride... Well, f_tstate was introduced here, in frameobject.c: ----------------------------------------------------- Revision : 2.29 Date : 1997/5/5 20:55:52 Author : 'guido' State : 'Exp' Lines : +13 -2 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. For frameobject.h, it was this version: --------------------------------------- Revision : 2.23 Date : 1997/5/5 20:55:43 Author : 'guido' State : 'Exp' Lines : +3 -1 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. Especially interesting: There is no single hint mentioning f_tstate. But, the diff shows PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; added at the same time. This was Python 1.5, I remember very well, the time where threading became the standard for Python. ceval.c was at this revision: ----------------------------- Revision : 2.118 Date : 1997/5/5 20:55:58 Author : 'guido' State : 'Exp' Lines : +263 -151 Description : Massive changes for separate thread state management. All per-thread globals are moved into a struct which is manipulated separately. I fetched ceval 2.117 and 2.118 and compared them. This was exactly the time when the thread state was introduced. There were exactly two references to f_tstate: The most interesting diff for this question: PyObject * PyEval_SaveThread() { #ifdef WITH_THREAD if (interpreter_lock) { - PyObject *res; - res = (PyObject *)current_frame; - current_frame = NULL; + PyThreadState *tstate = PyThreadState_Swap(NULL); + PyObject *res = tstate ? (PyObject *) (tstate->frame) : NULL; release_lock(interpreter_lock); return res; } #endif return NULL; } void PyEval_RestoreThread(x) PyObject *x; { #ifdef WITH_THREAD if (interpreter_lock) { int err; err = errno; acquire_lock(interpreter_lock, 1); errno = err; - current_frame = (PyFrameObject *)x; + PyThreadState_Swap(x ? ((PyFrameObject *)x)->f_tstate : NULL); } #endif } PyEval_SaveThread has changed a bit meanwhile, no longer return the frame, but tstate. semantics are similar. The two functions clearly show hwo the transition from a global current_frame variable was made to the tstate mechanism. f_tstate was the source for switching the thread state. The current ceval version looks like this: void PyEval_RestoreThread(PyThreadState *tstate) { if (tstate == NULL) Py_FatalError("PyEval_RestoreThread: NULL tstate"); #ifdef WITH_THREAD if (interpreter_lock) { int err = errno; PyThread_acquire_lock(interpreter_lock, 1); errno = err; } #endif PyThreadState_Swap(tstate); } That means, tstate is not fetched from the frame, but given as a parameter. I checked all places where this function gets called, the tstate parameter never comes from a frame. As a conclusion, f_tstate appears to be an intermediate step during the move to threading. It was an easy place to keep the thread state. Its use was reduced to very few later on. call_trace introduced today's slightly false behavior exactly in that 2.118 version. It looks very much like an optimization, since there was no macro for fetching tstate at that time. This was perfectly fine, until generators were introduced. Until then, a frame was always bound to its thread. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Guido van Rossum wrote:
since I didn't get *any* reply to this request, either the request was bad or there is really nobody using f_tstate in a way that makes it urgent to keep. I will wait a few hours and then make the change to Stackless, and I'd like to propose to do the same to the Python core.
I saved the message, but haven't had the time yet to think things through.
I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated.
I've been running Stackless Python without f_tstate for more than three months now, in various applications. May I check in a patch to evict f_tstate? ciao - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
[Christian]
since I didn't get *any* reply to this request, either the request was bad or there is really nobody using f_tstate in a way that makes it urgent to keep. I will wait a few hours and then make the change to Stackless, and I'd like to propose to do the same to the Python core.
[Guido]
I saved the message, but haven't had the time yet to think things through.
I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated.
[Christian]
I've been running Stackless Python without f_tstate for more than three months now, in various applications. May I check in a patch to evict f_tstate?
Sure! Let stackless lead the way. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mar 28, 2004, at 5:05 AM, Guido van Rossum wrote:
I saved the message, but haven't had the time yet to think things through.
I *did* notice at least one case where using f_tstate might actually be a mistake: theoretically it's possible that two or more threads alternate calling next() on a generator (if they wrap it in a critical section); AFAICT the f_tstate is never updated.
[Christian]
I've been running Stackless Python without f_tstate for more than three months now, in various applications. May I check in a patch to evict f_tstate?
Sure! Let stackless lead the way. :-)
This may screw up the work I'm doing to get the profiler to work transparently with threads. Since I can't promise that the profiler will be in the same thread as the code being profiled, I can't guarantee that PyThreadState_GET() will give the correct thread state, so I grab the thread state from the frame object. Of course, this work is also in the super-early stages of development, so I may go some other direction in the future when I find out that this doesn't work correctly...just pointing out a potential user (victim). -- Nick
This may screw up the work I'm doing to get the profiler to work transparently with threads. Since I can't promise that the profiler will be in the same thread as the code being profiled, I can't guarantee that PyThreadState_GET() will give the correct thread state, so I grab the thread state from the frame object. Of course, this work is also in the super-early stages of development, so I may go some other direction in the future when I find out that this doesn't work correctly...just pointing out a potential user (victim).
Since the profiler is being invoked from the thread being profiled, how could it end up not being in the same thread? (If I am missing something, I must be missing a *lot* about your design, so you might want to say quite a bit more on how it works.) --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mar 29, 2004, at 10:23 AM, Guido van Rossum wrote:
Since the profiler is being invoked from the thread being profiled, how could it end up not being in the same thread?
(If I am missing something, I must be missing a *lot* about your design, so you might want to say quite a bit more on how it works.)
...previous reply to this email deleted after some testing... Nevermind, the specific situation that I was concerned about actually can't occur. I wasn't sure that I could make a guarantee that the profiler would be invoked from the same thread as the code being profiled in all cases of thread destruction, but that appears to be an unwarranted concern. -- Nick
Nick Bastin wrote:
On Mar 29, 2004, at 10:23 AM, Guido van Rossum wrote:
Since the profiler is being invoked from the thread being profiled, how could it end up not being in the same thread?
(If I am missing something, I must be missing a *lot* about your design, so you might want to say quite a bit more on how it works.)
...previous reply to this email deleted after some testing...
Nevermind, the specific situation that I was concerned about actually can't occur. I wasn't sure that I could make a guarantee that the profiler would be invoked from the same thread as the code being profiled in all cases of thread destruction, but that appears to be an unwarranted concern.
You are the first person since months who claimed a possible use of f_tstate. Please make sure that you really don't need it. It needs to be changed, anyway: Either it must be updated for generators for every run, or it should be dropped. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Mar 29, 2004, at 1:15 PM, Christian Tismer wrote:
Nick Bastin wrote:
Since the profiler is being invoked from the thread being profiled, how could it end up not being in the same thread?
(If I am missing something, I must be missing a *lot* about your design, so you might want to say quite a bit more on how it works.) ...previous reply to this email deleted after some testing... Nevermind, the specific situation that I was concerned about actually can't occur. I wasn't sure that I could make a guarantee that the
On Mar 29, 2004, at 10:23 AM, Guido van Rossum wrote: profiler would be invoked from the same thread as the code being profiled in all cases of thread destruction, but that appears to be an unwarranted concern.
You are the first person since months who claimed a possible use of f_tstate. Please make sure that you really don't need it.
I'll do some more testing, but it could take me a couple of days. I do remember Martin Löwis mentioning to me that he thought it was possible to temporarily get two threadstate objects for the same thread, maybe when calling into a C function, and if that were the case, that could potentially cause problems for the profiler. Somebody else might have some more knowledgeable input there. One of the problems with this discussion occurring now is that while I'm intimately familiar with the profiler and existing tracing functionality, I'm not all that familiar with Python threads, so I need to learn a few more things before I can really jump into this work, and I'm concerned that something I don't know about is going to cause me to need the threadstate object in the frame. My current plan is to pass a python interface to the threadstate object to the trace function, so it can build up it's own table of stats data per thread (or in the debugger's case, a thread-specific context). I had a concern about the threadstate not being relevant because of the profiler being called in a different thread than the traced code, which I now think is unlikely to occur in a way that I can't catch. However, if Martin is right and there can effectively be multiple threadstate objects for the current thread, that may present a problem for the profiler, since it's indexing its' stats data based on the previous threadstate object. My thought was that if the global threadstate object changes, hopefully at least the surrounding frame tstate member that I pass to the profiler would still be intact, which would be a reason for keeping the threadstate reference in the frame, although perhaps my assumption is false. -- Nick
Hello Christian, On Fri, Dec 19, 2003 at 10:32:37AM +0100, Christian Tismer wrote:
since I didn't get *any* reply to this request, either the request was bad or there is really nobody using f_tstate in a way that makes it urgent to keep.
Sorry for the delay, we were kind of busy this week here... There are references to f_tstate in Psyco but nothing critical. Using PyThreadState_GET() instead should be fine. A bientot, Armin.
participants (7)
-
Armin Rigo -
Christian Tismer -
Christian Tismer -
Guido van Rossum -
Nick Bastin -
Raymond Hettinger -
Tim Peters