From AndiDog at web.de Sun Nov 1 11:42:37 2009 From: AndiDog at web.de (Andreas Sommer) Date: Sun, 01 Nov 2009 11:42:37 +0100 Subject: [capi-sig] C integration with multithreaded Python script In-Reply-To: <4AEC99BC.4080304@web.de> References: <4AD62EF0.2040602@web.de> <4AEC99BC.4080304@web.de> Message-ID: <4AED661D.1080603@web.de> Okay, I found out that the function WaitForThreadShutdown in modules/main.c is responsible for calling threading._shutdown which will automatically join all non-daemonic threads. Hope this will help somebody else... Andreas Sommer wrote: > I still don't have an answer to this question. Meanwhile I've taken a > look at the Python interpreter source code and couldn't find out how > it manages to keep running while other threads exist. Taking the > /RunModule/ function from the code doesn't help me - same problem: > only the first few instructions of the background thread are executed > and then Python execution ends. > > > Andreas Sommer wrote: >> Hope this is the right mailing list, please complain if not ;) >> >> I'm currently experimenting with the integration of Python (2.6) >> scripts into my application, and am using the following script for >> testing: >> >> *import* threading, time >> >> *def* handler(): >> print("a") >> time.sleep(0.95) >> print("b") >> >> t = threading.Thread(target = handler) >> t.start() >> >> >> And in C++, it's this code: >> >> Py_SetProgramName("test"); >> Py_Initialize(); >> *if*(!Py_IsInitialized()) *throw* std::exception(); >> >> >> PyObject *module = PyImport_AddModule("__main__"); >> PyObject *dict = PyModule_GetDict(module); >> >> printf("1\n"); >> >> strcpy(script, "import threading, time\n" >> "def handler():\n" >> " print(\"a\")\n" >> " time.sleep(0.95)\n" >> " print(\"b\")\n" >> "t = threading.Thread(target = handler)\n" >> "t.start()\n"); >> >> PyObject *obj = PyRun_String(script, Py_file_input, dict, dict); >> printf("2\n"); >> >> >> PyErr_Print(); >> Py_Finalize(); >> >> >> When executing, the output is >> >> 1 >> a2 >> >> as opposed to the expected >> >> 1 >> a >> b >> 2 >> >> It seems to me that when the main execution path (after t.start()) >> finishes, the other thread (called t) is canceled?! Adding t.join() >> to the end of the script gives the expected output, but isn't it >> possible to let PyRun_xxx wait for all remaining threads to end? I >> know the Python interpreter does this, but I didn't get much >> information from its source code (it seems to run scripts with the >> runpy module). >> >> >> Best regards > _______________________________________________ > capi-sig mailing list > capi-sig at python.org > http://mail.python.org/mailman/listinfo/capi-sig > From jeff at taupro.com Sun Nov 1 22:37:17 2009 From: jeff at taupro.com (Jeff Rush) Date: Sun, 01 Nov 2009 15:37:17 -0600 Subject: [capi-sig] C integration with multithreaded Python script In-Reply-To: <4AED661D.1080603@web.de> References: <4AD62EF0.2040602@web.de> <4AEC99BC.4080304@web.de> <4AED661D.1080603@web.de> Message-ID: <4AEDFF8D.8030909@taupro.com> Andreas Sommer wrote: > Okay, I found out that the function WaitForThreadShutdown in > modules/main.c is responsible for calling threading._shutdown which will > automatically join all non-daemonic threads. > > Hope this will help somebody else... Thank you for posting that follow up to the issue. -Jeff From joost at h-labahn.de Sat Nov 7 22:21:55 2009 From: joost at h-labahn.de (Joost) Date: Sat, 07 Nov 2009 22:21:55 +0100 Subject: [capi-sig] PyObject_Type with no tp_new - any dangers ? Message-ID: <4AF5E4F3.3080703@h-labahn.de> Hello, these days i made depikt, a new minimalistic gtk-interface, published on sourceforge.net. depikt realizes the abstract base classes of gtk like GObject, Widget or Container as extension classes defining no tp_new and is using this preprocessor template for it: #define DEPIKT_ABSTRACT_PYOB_HEADER(XXX)\ typedef struct {\ PyObject_HEAD\ Gtk##XXX *gwidget;\ } XXX;\ \ static PyTypeObject XXX##_Type; #define DEPIKT_ABSTRACT_PY_TYPE(x) \ static PyTypeObject XXX##_Type = {\ PyObject_HEAD_INIT(NULL)\ 0,\ "depiktmodule."#XXX,\ sizeof(XXX),\ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\ PyObject_GenericGetAttr,\ PyObject_GenericSetAttr,\ 0,\ Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE,\ "Python type object for "#XXX,\ 0, 0, 0, 0, 0, 0,\ XXX##_methods\ }; and that is nearly all depikt does for these types (except the eventual tp_base-supplement in the PyMODINIT_FUNC), no tp_new, no tp_dealloc - but a canonical method table with the needed methods is added of course. Inheritance works, because for example a GtkContainer* is also a GtkWidget* - that is how gtk handles inheritance in C. Thus the Container_Type has a GtkWidget* in his core struct indeed. So far this is working, gtk_container_set_focus_chain for example is doing well. However in classes inheriting from depikt.Window there is bad behavior in the second generation. (It is not the first Python extension i write, but the first more ambitious, and also the first more ambitious C project. Thus i am glad to have come so far (basically depikt works fine) - but with the complex type system of gtk and my inexperience with Python extensions i need help at this point. I am looking for a decent co-author.) Here my question: Are their problems with abstract classes like this in Python ? I supposed not - otherwise i had not continued with the development of depikt to this point -, but i want to exclude error sources. Can i do anything to "harden" the concept ? Thanks for reading, Joost From joost at h-labahn.de Wed Nov 11 09:55:40 2009 From: joost at h-labahn.de (Joost) Date: Wed, 11 Nov 2009 09:55:40 +0100 Subject: [capi-sig] depikt - no urgent need for help anymore Message-ID: <4AFA7C0C.9010701@h-labahn.de> Hello, as announced some days before i published depikt on sourceforge. Depikt is a bundle of python wrappers for gtk in one file, used by me as a replacement for pygtk. I announced depikt together with a cry for help. This isn't necessary anymore. I've realized a version without abstract classes in the hierarchy, which works again for GUI-building. Inheritance from depikt classes has even got worse now (what will ease debugging) - nevertheless this is the final architecture. And help couldn't be provided at all this week (til Nov.15) - i have deleted the code at sourceforge for some days, til the new version is a bit better. Depikt will then only support Python 3.1.1 upwards - the new concept makes massive use of an innovation in Python's C-API (PyCapsule). Of course any feedback and amendments of depikt will always be welcome. Thanks for reading, Joost Behrends From stefan_ml at behnel.de Wed Nov 11 10:20:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Nov 2009 10:20:00 +0100 Subject: [capi-sig] depikt - no urgent need for help anymore In-Reply-To: <4AFA7C0C.9010701@h-labahn.de> References: <4AFA7C0C.9010701@h-labahn.de> Message-ID: <4AFA81C0.6060109@behnel.de> Joost, 11.11.2009 09:55: > as announced some days before i published > depikt on sourceforge. Depikt is a bundle of > python wrappers for gtk in one file, used by > me as a replacement for pygtk. It seems to me that it's written in plain C. Have you heard about Cython? Assuming that this project is intended to grow larger, it would allow you to cut down the code size by a factor of 10 or something in that order, while actually generating faster and safer glue code for you. > Depikt will then only support Python 3.1.1 > upwards - the new concept makes massive use > of an innovation in Python's C-API (PyCapsule). Note that earlier Python versions had a PyCObject type, which was essentially the same as PyCapsule is now, except for a bit of non-technical overhead. So you could still support Python 2.x quite easily by exchanging the underlying type object. Stefan From george_edison55 at hotmail.com Sun Nov 15 08:30:13 2009 From: george_edison55 at hotmail.com (Nathan Osman) Date: Sat, 14 Nov 2009 23:30:13 -0800 Subject: [capi-sig] How to bundle tkinter? Message-ID: I've really got a strange problem. I've got a C++ app that uses the Python/C API. All of the python standard modules are stored in a file python31.zip. Most modules import fine except tkinter... and of course it also happens to be the one I need. I get a message "could not find module _tkinter". How can I bundle tkinter with my app? - George _________________________________________________________________ Eligible CDN College & University students can upgrade to Windows 7 before Jan 3 for only $39.99. Upgrade now! http://go.microsoft.com/?linkid=9691819 From george_edison55 at hotmail.com Mon Nov 16 03:04:56 2009 From: george_edison55 at hotmail.com (Nathan Osman) Date: Sun, 15 Nov 2009 18:04:56 -0800 Subject: [capi-sig] How to bundle tkinter? In-Reply-To: References: Message-ID: Never mind. I figured it out. I needed to include _tkinter.pyd and the tcl/tk DLLs. - George > From: george_edison55 at hotmail.com > To: capi-sig at python.org > Date: Sat, 14 Nov 2009 23:30:13 -0800 > Subject: [capi-sig] How to bundle tkinter? > > I've really got a strange problem. I've got a C++ app that uses the Python/C API. All of the python standard modules are stored in a file python31.zip. Most modules import fine except tkinter... and of course it also happens to be the one I need. I get a message "could not find module _tkinter". How can I bundle tkinter with my app? > > - George > > _________________________________________________________________ > Eligible CDN College & University students can upgrade to Windows 7 before Jan 3 for only $39.99. Upgrade now! > http://go.microsoft.com/?linkid=9691819 > _______________________________________________ > capi-sig mailing list > capi-sig at python.org > http://mail.python.org/mailman/listinfo/capi-sig _________________________________________________________________ Windows Live: Keep your friends up to date with what you do online. http://go.microsoft.com/?linkid=9691815 From joost at h-labahn.de Mon Nov 16 12:04:45 2009 From: joost at h-labahn.de (Joost) Date: Mon, 16 Nov 2009 12:04:45 +0100 Subject: [capi-sig] Obstacles for subclassing extension types - need hints Message-ID: <4B0131CD.1080806@h-labahn.de> Hello, these days i make depikt, a new minimalistic gtk-interface, published on sourceforge.net. For Python3, what all here is referring to. First there were abstract classes for Container and so on - they have been replaced by method bundles added by the preprocessor to the individual derived widgets, which of course are extension classes. Using one baseclass for gobjects, called Pikt. I knew, this would not be simple, but what i hadn't any idea of are the problems Python causes here. From cStringIO i knew, that there were some not subclassable Python types, but i took that for an exotic exception. I also had no idea, that dynamic attributes are no standard behavior (having never tried [1,2,3].a = "new attribute" at a python prompt). I've got dynamic attributes now, and inheritance from the lower level classes in pango and gdk. But inheritance for gtk.Window or gtk.VBox is much more important in practice and after about 3 days of trying (with TPFLAGS_HEAPTYPE for example) and reading in object.c, objectimpl.c or typeobject.c depikt's widget classes are still "idempotent": class myEntry(depikt.Entry): ... doesn't crash but its __init__ creates - perfectly usable - objects of the type depikt.Entry (what can be iterated). A funny behavior, but seen from a strict perspective a bug in the C-API (should crash. Or are all such classes "idempotent" and is this needed internally ?) This here is from the preprocessor factory for the widget types (initialized for Button for readability) and from here the problem might stem: 378 PyObject *Button_new(PyTypeObject *type, PyObject *args, PyObject *kwds) {\ 379 Button *self;\ 380 self = (Button *)Pikt_new(&Button_Type, args, kwds); \ 381 self->Q.gobj = PyCapsule_New(newfunc, "Button", NULL); \ 382 if (! self->Q.gobj) { Py_DECREF(self); KTERR("Cannot create Button");};\ 383 return (PyObject *)self;\ That means, from line 380 (which follows a recommendation of the tutorial and is vital. It also took nearly a day to find out, that Pikt_new can accept a Button_Type*, that is not documented and far from obvious). Could somebody give me a complete list of obstacles for inheritance from the C-API ? Including the nexus to allocation ? I do not mean explicit explanation of each obstacle. Hints for each item are fully enough. Thanks for reading, Joost From joost at h-labahn.de Tue Nov 17 00:21:40 2009 From: joost at h-labahn.de (Joost) Date: Tue, 17 Nov 2009 00:21:40 +0100 Subject: [capi-sig] Subclassing extension types - sorry Message-ID: <4B01DE84.1080107@h-labahn.de> Hello, another cry for help i can withdraw. Yes, it was line 380 in " 378 PyObject *Button_new(PyTypeObject *type, PyObject *args, PyObject *kwds) {\ 379 Button *self;\ 380 self = (Button *)Pikt_new(&Button_Type, args, kwds); \ 381 self->Q.gobj = PyCapsule_New(newfunc, "Button", NULL); \ 382 if (! self->Q.gobj) { Py_DECREF(self); KTERR("Cannot create Button");};\ 383 return (PyObject *)self;\ " (newfunc came - as the backslashes - from the original preprocessor macro (the second argument), where i had filled Button in for the first argument). With the normal self = (Button *)type->tp_alloc(type, 0) things work now. Still not all is fine, but now these widgets can be subclassed. Unfortunately i cannot remember exactly, what lines in what part of the tutorial or references (or a comment in the sources ?) brought me to the idea to need the wrong code as above. Thanks for reading, Joost From joost at h-labahn.de Sat Nov 21 09:46:38 2009 From: joost at h-labahn.de (Joost) Date: Sat, 21 Nov 2009 09:46:38 +0100 Subject: [capi-sig] A special kind of abstract base class - will that work ? Message-ID: <4B07A8EE.1070904@h-labahn.de> Hello, With depikt, my python wrappers for gtk i am working at now, i posed a cry of help here, which wasn't needed anymore some days later (but perhaps any kind of utterance was indispensable for continuing). The problem is to represent gtk's kind of "class hierarchy" in Python. This here is my solution, it seems to work and given this is stable it will be the final version. depikt uses one abstract base class Pikt. Pikt is nothing than its instance and its type struct - and that is the point of my question. As follows: typedef struct { PyObject_HEAD PyObject *gobj; } Pikt; static PyTypeObject Pikt_Type = { PyVarObject_HEAD_INIT(NULL, 0) "depikt.Pikt", 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, }; That is all concerning Pikt. gobj is there to contain the PyCapsules, which itself contain the gtk-pointers to the widgets, created by the following preprocessor factory: #define KT__GOB_HEADER(YYY, newfunc)\ typedef struct { Pikt Q; } YYY;\ static PyTypeObject YYY##_Type;\ \ PyObject *YYY##_new(PyTypeObject *type, PyObject *args, PyObject *kwds) {\ YYY *self; self = (YYY *)type->tp_alloc(type, 0); \ self->Q.gobj = PyCapsule_New(newfunc, #YYY, NULL); \ if (! self->Q.gobj) { Py_DECREF(self); KTERR("Cannot create "#YYY);};\ Py_INCREF(self); return (PyObject *)self;\ }; #define KT__GOB_TYPE(XXX) \ \ void XXX##_free(XXX *self) \ { PyObject_Del(self); };\ void XXX##_dealloc(XXX *self)\ { Py_XDECREF(self); g_free((gpointer)self->Q.gobj); XXX##_free(self); };\ \ static PyTypeObject XXX##_Type = {\ PyVarObject_HEAD_INIT(NULL, 0)\ "depikt."#XXX,\ sizeof(XXX)+4*sizeof(NULL),\ 0,\ (destructor)XXX##_dealloc,\ 0,\ 0, \ 0, 0, 0, 0, 0, 0, 0, 0, 0,\ 0,\ 0,\ 0,\ Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE,\ "Python type object for "#XXX,\ 0, 0, 0, 0, 0, 0,\ XXX##_methods,\ 0, 0,\ &Pikt_Type,\ 0,\ 0, 0,\ -4*sizeof(NULL),\ 0, 0,\ XXX##_new,\ (freefunc)XXX##_free,\ }; The headers are only for those widgets, for which literally constant newfuncs can be provided (like gtk_window_new(GTK_WINDOW_TOPLEVEL). (Besides: I am still unsure, whether the dealloc/free couple is right here). All widgets shall also consist only of the gtk-pointers returned from those newfuncs and method tables - no attributes. The intermediate abstract widgets as GtkContainer are realized as method bundles getting added to each concrete widget's method table by the preprocessor. Pikt is indsipensable as the argument type for depikt_Container_add(Pikt *self, Pikt *to_add) for example. Thus this abstract class is not there for providing a protocol, but as kind of stub for the type system. It is not exposed to Python and actually no proper class. This looks very clean and petty to me - still i am amazed to have found this. It might even be useful as a design pattern for other extensions (thus i hope, you do not feel like wasting your time reading this). But this goes far beyond of what is teached in the Python tutorials and for me a painful doubt is remaining, whether Pikt as this will really always work. C combined with inheritance by casting itself and the Python C-API even more have so terrible side effects of allocation strategies. All the more, as i still produce bad programming errors - this solution has by no means solved all my problems with depikt (this is my first ambitious C-project). But already depikt has features pygtk has not. In this form my question is probably not decidable, but still i'd like to have an estimation of the involved risks doing it like this. Thanks for reading, Joost From joost at h-labahn.de Thu Nov 26 13:03:01 2009 From: joost at h-labahn.de (Joost) Date: Thu, 26 Nov 2009 13:03:01 +0100 Subject: [capi-sig] PyObject_Type with no tp_new - really no surveyor around ? Message-ID: <4B0E6E75.5050303@h-labahn.de> Hello, i won't repeat my last message - it was long enough. Just wanted to stay, that i have understood that the Py_INCREF(self) in tp_new for the gobject types was wrong. Setting tp_basicsize to (default + 4*sizeof(NULL)) is defensive, "1" instead of "4" - that is for the pointer to tp->dict - seems to be enough too. Still all this leaves my fairly uncomfortable. Nobody there to risk an estimation ? Thanks for reading, Joost