Mailman 3 September 2003 - Python-Dev

Re: [Python-Dev] Making python C-API thread safe (try 2)
by Harri Pesonen 11 Sep '03

11 Sep '03

Phillip J. Eby wrote: > Please do not CC: my mail to Python-Dev again; I intentionally did not > include python-dev on my CC: because it was asked that we move this > thread elsewhere. > > At 10:16 PM 9/11/03 +0300, Harri Pesonen wrote: > >> Phillip J. Eby wrote: >> >>> At 08:47 PM 9/11/03 +0300, Harri Pesonen wrote: >>> >>>> But my basic message is this: Python needs to be made thread safe. >>>> Making the individual interpreters thread safe is trivial, and >>>> benefits many people, and is a necessary first step; >>> >>> >>> >>> It's far from trivial - you're talking about invalidating every >>> piece of C code written for Python over a multi-year people by >>> dozens upon dozens of extension authors. >> >> >> The change is trivial in Python C API. I already said that it would >> break everything outside the Python distribution, but the change in >> other applications is also trivial. > > > How do you propose that C code called *from* Python *receive* the > threadstate pointer? Exactly like that. Is there a problem? I'm suggesting that every function call gets that pointer, unless the function can get it from some other argument, that contains a pointer to it. > >>> It doesn't benefit many people: only those using isolated >>> interpreters embedded in a multithreaded C program. >> >> >> I don't know how many people are writing threads in Python, either. I >> guess that not so many. In my case I only need a thread safe >> interpreter, I don't create threads in Python code. So just having >> what I described would be enough for me: no need for global >> interpreter lock, and Python would be really multithreading. It would >> benefit many people, I'm sure. > > > Obviously, it's enough for you, or you wouldn't be proposing it. What > does it do for me? Nothing whatsoever, except add needless overhead > and make me rewrite every C extension I've ever written for Python. > So, by and large, you're not going to get much support for your change > from Python developers, especially those who write C extensions, or > depend on extensions written by others. Probably. That's why I'm thinking now that the language should be called something else, like MPython for "multi-threading Python". It would be 99% compatible with the existing Python syntax, but have different internals. > >> Yes, I'm aware of the None problem at least (only one instance of >> it). Please enlighten me about the other critical sections? Object >> allocation/freeing? > > > Data structure manipulations, e.g. all use of dictionaries. Python > spends most of its time doing dictionary lookups or modifications, all > of which need to be protected. After sleeping over night, I think that I got it. :-) The simple solution is, that each thread created in Python gets its own independent interpreter state as well. And there could be a separate thread-global interpreter state for shared memory access. Access to this global state would always be synchronized. There could even be multiple named global states, so that the thread interlocking could be minimized. The python syntax for creating objects in this global state should be invented: synchronize a = "abcd" Also when creating the new thread, it's arguments would be copied from the creating state to the new state. What does it sound? Of course it would be incompatible with the current threading system in Python, but it would be totally multithreading, no global interpreter lock needed. It would be faster than current Python, there would be no need to free or acquire the lock when calling OS functions, and no need to check how many byte codes have been processed, etc. > >>> I'm guessing you haven't done much writing of C extensions for >>> Python (or Python core C), or else you'd realize why trying to make >>> INCREF/DECREF threadsafe would absolutely decimate performance. >>> Reference count updates happen *way* too often in normal code flow. >> >> I also knew that already. But how else can you do it? > > > The way it's done now! :) I understand why the current Python works like it does. But I think that it's time for the next generation. If you don't do it, and I have no time now to do it, I'm still sure that this is done at some point, rather sooner than later. > >> Of course, changing Python to not have a single None would help a >> lot. Or, perhaps it could have a single None, but in case of None, >> the reference count would have no meaning, it would never be >> deallocated, because it would be checked in code. Maybe it does it >> already, I don't know. > > > I really don't mean to be rude (another reason I'm writing this to you > privately), but this paragraph shows you are *really* new to Python > both at the level of coding in Python and coding with Python's C API. > I wish I could explain in detail why, but there's really far too much > that you don't understand and it would take me too long. I will > attempt to summarize a very few points, however: first, identity > (pointer comparison) is a part of the Python language, so you can't > have multiple None instances any more than you can have more than one > value be NULL in C. Second, at the C level, all Python objects > (including None) have an absolutely uniform API, so having refcount > behavior be different for different kinds of objects is not at all > practical. Third, if you had more than one Py_None at the C level, > you'd either have to make Py_None a macro, or rewrite all the C. If > you don't think that's a problem, you have absolutely no idea how much > C code out there is written to the Python API. Yes, Py_None would be a macro. All access to interpreter state would go through the interpreter state pointer that is always in stack, the first argument each C API function gets. That pointer should be named so that the macros will always work ("tState", for example, so that Py_None macro would expand to tState->mPy_None, for example). > >> I'm also wondering why this problem has not been addressed before? > > > It has; the cure is worse than the disease. A few years ago, somebody > wrote a "free-threading" version of Python, which locked individual > data objects rather than use the global interpreter lock. The > performance for single-threaded programs was abominable, and the > performance gain even on multiprocessor machines was not thought worth > the cost. So the project was scrapped. There would be no locking in my proposal, except when accessing the shared memory global thread state. I don't know, I got mail about writing a PEP. It is clear that it would not be accepted, because it would break the existing API. The change is so big that I think that it has to be called a different language. This is the last message I will make about this matter (before actually starting to code it), so I'm posting this to python-list as well, because this is too important to be ignored. Python *needs* to be free-threading... Harri

1 0

Making python C-API thread safe (try 2)
by Harri Pesonen 11 Sep '03

11 Sep '03

Sorry about the previous post! :-) Background: I'm new to Python. I just embedded Python into a multi-threaded application, and extended Python by callbacks as well. During this process I realised that Python is not internally as beautiful as externally. It is not thread safe. This surprised me, because Python is a quite young programming language. I started thinking how to fix this, how to make Python thread safe. Currently Python does not support multiprocessor machines or hyperthreading processors. It is only a matter of time when this has to be fixed, better soon than later. How to make C-API thread safe: It's quite simple, in fact. Thread should first call PyThreadState* Py_NewInterpreter() and use this thread state pointer in every following API call. This thread state pointer should point to a memory structure, that holds everything that Python interpreter needs. There should be no global variables, not even None, except const char* Py_GetVersion() and such. So every Python function should have this thread state as the first argument: int PyRun_SimpleString(PyThreadState*tstate, char *command) and so on. Also callbacks should have it, about every API function. Advantages: The following functions would not be needed any more, it would make the API much simpler: void PyEval_InitThreads( ) void PyEval_AcquireLock( ) void PyEval_ReleaseLock( ) void PyEval_AcquireThread( PyThreadState *tstate) void PyEval_ReleaseThread( PyThreadState *tstate) PyThreadState* PyEval_SaveThread( ) void PyEval_RestoreThread( PyThreadState *tstate) Py_BEGIN_ALLOW_THREADS Py_END_ALLOW_THREADS Py_BLOCK_THREADS Py_UNBLOCK_THREADS PyInterpreterState* PyInterpreterState_New( ) void PyInterpreterState_Clear( PyInterpreterState *interp) void PyInterpreterState_Delete( PyInterpreterState *interp) PyThreadState* PyThreadState_New( PyInterpreterState *interp) void PyThreadState_Clear( PyThreadState *tstate) void PyThreadState_Delete( PyThreadState *tstate) PyThreadState* PyThreadState_Get( ) PyThreadState* PyThreadState_Swap( PyThreadState *tstate) The current documentation says: "In order to support multi-threaded Python programs, the interpreter regularly releases and reacquires the lock -- by default, every 100 bytecode instructions". This would not be needed anymore. It would make the interpreter faster. Making this change would take a couple of weeks, if the job would be divided between modules to different persons. The change is trivial, but it would have to be made everywhere. Disadvantages: Everything old would break. Thread safe Python should have a version number significantly higher, like Python 3. Old applications would work with old Python versions, with old Python DLLs, but they would need to be rewritten for the new API. It would be a trivial task, but still. Disclaimer: I already got mail saying: " This issue gets discussed regularly on comp.lang.python; python-dev is for discussion of specific changes to the Python language and libraries. Until you are familiar with the way threading works in Python and have some concrete proposal to make, please keep the discussion on c.l.py." The current situation is very unfortunate, having a single threaded implementation in the modern computer age. I have a concrete proposal, which would break everything :-) but I think that is the only solution. Thanks for reading, Harri

4 4

Making python C-api thread safe
by Harri Pesonen 11 Sep '03

11 Sep '03

Background: I'm new to Python. I just embedded Python into a multi-threaded application, and extended Python by callbacks as well. During this process I realised that Python is not internally as beautiful as externally. It is not thread safe. This surprised me, because Python is a quite young programming language. I started thinking how to fix this, how to make Python thread safe. Currently Python does not support multiprocessor machines or hyperthreading processors. It is only a matter of time when this has to be fixed, better soon than later. How to make C-api thread safe: It's quite simple, in fact. Thread should first call

2 1

RE: [Python-Dev] Consistent logging in the standard library
by Michael Chermside 10 Sep '03

10 Sep '03

Quoth Matthew F. Barnes: > An application may want to organize things differently. For example, it > may want to redirect asyncore's log messages to a logger named > "myapp.network.asyncore" Seems like an uncommon request, but one which shouldn't be made impossible for those who want it. But couldn't it be achieved by writing a handler which simply re-logged the message at the new spot in the hierarchy?

3 2

Luscious, Seductive L|ps can be yours
by Women Helpline 10 Sep '03

10 Sep '03

1 0

Be Honest about LC_NUMERIC [REPOST]
by Christian Reis 10 Sep '03

10 Sep '03

So, in an attempt to garner comments (now that we have 2.3 off the chopping block) I'm reposting my PEP proposal (with minor updates). Comments would be appreciated, of course (nudges Barry slightly after him getting me to write this on my only free Sunday in months ;) PEP: XXX Title: Be Honest about LC_NUMERIC (to the C library) Version: $Revision: 1.9 $ Last-Modified: $Date: 2002/08/26 16:29:31 $ Author: Christian R. Reis <kiko at async.com.br> Status: Draft Type: Standards Track Content-Type: text/plain <pep-xxxx.html> Created: 19-July-2003 Post-History: ------------------------------------------------------------------------ Abstract Support in Python for the LC_NUMERIC locale category is currently implemented only in Python-space, which causes inconsistent behavior and thread-safety issues for applications that use extension modules and libraries implemented in C. This document proposes a plan for removing this inconsistency by providing and using substitute locale-agnostic functions as necessary. Introduction Python currently provides generic localization services through the locale module, which among other things allows localizing the display and conversion process of numeric types. Locale categories, such as LC_TIME and LC_COLLATE, allow configuring precisely what aspects of the application are to be localized. The LC_NUMERIC category specifies formatting for non-monetary numeric information, such as the decimal separator in float and fixed-precision numbers. Localization of the LC_NUMERIC category is currently implemented in only in Python-space; the C libraries are unaware of the application's LC_NUMERIC setting. This is done to avoid changing the behavior of certain low-level functions that are used by the Python parser and related code [2]. However, this presents a problem for extension modules that wrap C libraries; applications that use these extension modules will inconsistently display and convert numeric values. James Henstridge, the author of PyGTK [3], has additionally pointed out that the setlocale() function also presents thread-safety issues, since a thread may call the C library setlocale() outside of the GIL, and cause Python to function incorrectly. Rationale The inconsistency between Python and C library localization for LC_NUMERIC is a problem for any localized application using C extensions. The exact nature of the problem will vary depending on the application, but it will most likely occur when parsing or formatting a numeric value. Example Problem The initial problem that motivated this PEP is related to the GtkSpinButton [4] widget in the GTK+ UI toolkit, wrapped by PyGTK. The widget can be set to numeric mode, and when this occurs, characters typed into it are evaluated as a number. Because LC_NUMERIC is not set in libc, float values are displayed incorrectly, and it is impossible to enter values using the localized decimal separator (for instance, `,' for the Brazilian locale pt_BR). This small example demonstrates reduced usability for localized applications using this toolkit when coded in Python. Proposal Martin v. Löwis commented on the initial constraints for an acceptable solution to the problem on python-dev: - LC_NUMERIC can be set at the C library level without breaking the parser. - float() and str() stay locale-unaware. The following seems to be the current practice: - locale-aware str() and float() [XXX: atof(), currently?] stay in the locale module. An analysis of the Python source suggests that the following functions currently depend on LC_NUMERIC being set to the C locale: - Python/compile.c:parsenumber() - Python/marshal.c:r_object() - Objects/complexobject.c:complex_to_buf() - Objects/complexobject.c:complex_subtype_from_string() - Objects/floatobject.c:PyFloat_FromString() - Objects/floatobject.c:format_float() - Modules/stropmodule.c:strop_atof() - Modules/cPickle.c:load_float() [XXX: still need to check if any other occurrences exist] The proposed approach is to implement LC_NUMERIC-agnostic functions for converting from (strtod()/atof()) and to (snprintf()) float formats, using these functions where the formatting should not vary according to the user-specified locale. This change should also solve the aforementioned thread-safety problems. Potential Code Contributions This problem was initially reported as a problem in the GTK+ libraries [5]; since then it has been correctly diagnosed as an inconsistency in Python's implementation. However, in a fortunate coincidence, the glib library implements a number of LC_NUMERIC-agnostic functions (for an example, see [6]) for reasons similar to those presented in this paper. In the same GTK+ problem report, Havoc Pennington has suggested that the glib authors would be willing to contribute this code to the PSF, which would simplify implementation of this PEP considerably. [I'm checking if Alex Larsson is willing to sign the PSF contributor agreement [7] to make sure the code is safe to integrate; XXX: what would be necessary to sign here?] Risks There may be cross-platform issues with the provided locale-agnostic functions. This needs to be tested further. Martin has pointed out potential copyright problems with the contributed code. I believe we will have no problems in this area as members of the GTK+ and glib teams have said they are fine with relicensing the code. Code An implementation is being developed by Gustavo Carneiro <gjc at inescporto.pt>. It is currently attached to Sourceforge.net bug 744665 [8] [XXX: The SF.net tracker is horrible 8(] References [1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton http://www.python.org/peps/pep-0001.html [2] Python locale documentation for embedding, http://www.python.org/doc/current/lib/embedding-locale.html [3] PyGTK homepage, http://www.daa.com.au/~james/pygtk/ [4] GtkSpinButton screenshot (demonstrating problem), http://www.async.com.br/~kiko/spin.png [5] GNOME bug report, http://bugzilla.gnome.org/show_bug.cgi?id=114132 [6] Code submission of g_ascii_strtod and g_ascii_dtostr (later renamed g_ascii_formatd) by Alex Larsson, http://mail.gnome.org/archives/gtk-devel-list/2001-October/msg00114.html [7] PSF Contributor Agreement, http://www.python.org/psf/psf-contributor-agreement.html [8] Python bug report, http://www.python.org/sf/774665 Copyright This document has been placed in the public domain. Take care, -- Christian Reis, Senior Engineer, Async Open Source, Brazil. http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL

9 32

RE: [python-win32] pb with string conversion to float and GetObject call
by Moore, Paul 10 Sep '03

10 Sep '03

From: popov [mailto:outlook@evpopov.com] > Here's a strange one: > > print float('8.4') > > Works as expected. > But: > > import win32com.client > obj = win32com.client.GetObject ("winmgmts:") > print float('8.4') > > Throws a 'ValueError: invalid literal for float(): 8.4' exception on the > print line ! > If you use '8,4' instead of '8.4', then it works (!). So it seems that some > locals have been modified here by the call to GetObject (without this call - > leaving only the import statement - the problem does not show up). > Note that using 'LDAP:' instead of 'winmgmts:', everything is ok. So maybe > the culprit is the winmgmts provider... > If someone could test this and let me know the results. Thx. There's a discussion going on on python-dev at the moment (the messages mention LC_NUMERIC in the subject) on basically this issue. Python's float conversion routines rely on the underlying C locale remaining unchanged. It's known that 3rd party extensions can violate this, and also that Microsoft MAPI mucks up the locale behind the scenes. From what you are reporting, it looks like the WMI stuff also does this. I don't know of any fix for sure - you might be able to use the functions from the locale module to set the locale value back to "C". Assuming that an acceptable fix is found, something should be in Python 2.4 (I don't know if the fix will be deemed appropriate for a maintenance release of 2.3). Hope this helps, Paul. PS I'm copying this to python-dev, as evidence of another area where this issue comes up - I hope that's acceptable.

1 0

Base-n integer formatting
by Yuri Vilmanis 10 Sep '03

10 Sep '03

I have three feature requests for you: no. 1 is quite fundamental, no. 2 would be a 'nice' useful feature and no. 3 is a bit more esoteric, but very useful in certain applications 1) print formatting for integers is currently only available for octal, hex, and decimal. This seems rather odd, as string to integer conversion is available for bases 2 through 36. End result: when I want to do things in base 6 (rare, but it has happened twice) or much more commonly in binary, I can read the numbers in, but cannot print the results - ie quite useless. 2) Input and output formatting for unary (which is commonly referred to as base 1, yes, I know it's not 'really' base 1 by the general deinition of base, but "base 1" is the convention) would also be good, as would roman numeral support, especially in word-processing applications or IO from any formatted document with roman page/section numbers 3) negative bases: I have needed this more times than I think some people would care to imagine. These representations have several very useful properties such as implementation-independent signdness (hence arbirtrary word length), and some more interesting properties I won't expand on here. Context: I find python very handy not only as a programming/scripting language but also as a scratchpad for various number theory investigations. Unfortunately the output formatting leaves all but the most basic investigations out in the cold, and lack of negative base input/output makes it harder to use python to look for certain patterns in result sets, not to mention getting the data in... Thanks, Yuri

2 1

RE: [Python-Dev] Base-n integer formatting
by Chermside, Michael 10 Sep '03

10 Sep '03

Yuri: I think you have some interesting ideas, but this is really not the best place to present them. The newsgroup comp.lang.python, also a mailing list at python-list(a)python.org, is really the best place to discuss these ideas. Python-dev is intended for discussion about the development of Python. New ideas or language features are best introduced first on c.l.py, then based on the feedback there, written up as a PEP, *THEN* folks will start discussing it on python-dev as they figure out HOW (and perhaps whether) to implement it. There's an exception -- regular Python developers sometimes introduce ideas on this list, but that's a very short list of people, and it's a bit different, because if they bring it up it's because they're already volunteering to implement it (and probably have a prototype up and running already). If you'll contact me _separately_, I'll be glad to give you my thoughts on your suggestions. -- Michael Chermside -----Original Message----- From: Yuri Vilmanis [mailto:vilmanis@internode.on.net] Sent: Wednesday, September 10, 2003 8:39 AM To: python-dev(a)python.org Subject: [Python-Dev] Base-n integer formatting I have three feature requests for you: no. 1 is quite fundamental, no. 2 would be a 'nice' useful feature and no. 3 is a bit more esoteric, but very useful in certain applications 1) print formatting for integers is currently only available for octal, hex, and decimal. This seems rather odd, as string to integer conversion is available for bases 2 through 36. End result: when I want to do things in base 6 (rare, but it has happened twice) or much more commonly in binary, I can read the numbers in, but cannot print the results - ie quite useless. 2) Input and output formatting for unary (which is commonly referred to as base 1, yes, I know it's not 'really' base 1 by the general deinition of base, but "base 1" is the convention) would also be good, as would roman numeral support, especially in word-processing applications or IO from any formatted document with roman page/section numbers 3) negative bases: I have needed this more times than I think some people would care to imagine. These representations have several very useful properties such as implementation-independent signdness (hence arbirtrary word length), and some more interesting properties I won't expand on here. Context: I find python very handy not only as a programming/scripting language but also as a scratchpad for various number theory investigations. Unfortunately the output formatting leaves all but the most basic investigations out in the cold, and lack of negative base input/output makes it harder to use python to look for certain patterns in result sets, not to mention getting the data in... Thanks, Yuri _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org http://mail.python.org/mailman/listinfo/python-dev This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it.

1 0

Consistent logging in the standard library
by Matthew F. Barnes 10 Sep '03

10 Sep '03

Having just read the "Use of the logging package in the standard library" section of Brett's latest python-dev summary [1], I thought I might try to generate some discussion on this topic since I brought it up in the first place. There was actually a bit of a longer discussion of this topic on comp.lang.python [2], where A.M. Kuchling pointed out a number of good issues that will need to be considered and suggested that I write a PEP on the topic. In fact I've been working on one since, but it's not yet ready to be submitted (it's my first, so it's taking awhile). The basic idea in the PEP is this: maintain a level of indirection between the logger objects and the modules or class instances invoking them, so that custom application-specific loggers can be substituted for the "standard" ones. The method for doing so is modelled after the socket module's new timeout functionality [3], where each module that performs logging defines a pair of module-level functions named getdefaultlogger() and setdefaultlogger() that return and accept (respectively) a logging.Logger instance. The initial default logger for each of these modules is named after the module itself (i.e. logging.getLogger(__name__)) and its behavior mimics that of the Python 2.3 version of the module in terms of format and destination of the logging messages. So an application that wants to override the logging behavior of a standard library module can do so simply by passing a customized logger object to the module's setdefaultlogger() function. This works out even better for modules where all the logging is encapsulated in a class. The PEP calls for each class that performs logging to define an attribute called self.logger, and upon instantiation initialize self.logger to the module's *current* default logger (again, modelled after how the timeout value for a socket object is initialized). This allows each instance of a class to have its own unique logging behavior. The first item for discussion is whether this seems like a reasonable approach. The second is identifying which standard library modules are candidates for modification using this approach. I've identified the following by simply searching for the word "logging" in the source code, but I'm not sure whether this list is complete: asyncore BaseHTTPServer cgi doctest imaplib unittest the distutils package I have patches ready for asyncore and BaseHTTPServer and doctest, which fit the model I described above quite nicely. I determined that no changes are necessary for the cgi module, since it doesn't even use the logging functions it defines (initlog, dolog, etc.). I'm currently working on a way to replicate imaplib's command history using LogRecords, which may call for a new type of logging.BufferingHandler subclass that maintains a circular queue of LogRecords. And the distutils package seems prepped for using the logging package already. The oddball is unittest. Unittest is what motivated me to bring up the logging issue in the first place, but I can't (or more precisely - am not in a position to) decide what direction that module should take w.r.t. if and how it should use the logging package. Should the TextTestRunner class be modified to use a logging.Logger instead of a _WritelnDecorator, or should a separate TestRunner subclass (a LoggingTestRunner perhaps) be defined instead? I'm hoping that Steve Purcell or others may have some thoughts on this. I would also appreciate hearing any alternatives to the approach I've suggested, as well as comments on the strengths and weaknesses of each. Matthew Barnes [1] <http://mail.python.org/pipermail/python-dev/2003-September/037938.html> [2] <http://groups.google.com/groups?hl=en&lr=lang_en&ie=UTF-8&oe=UTF-8&threadm=…> [3] <http://www.python.org/doc/2.3/lib/module-socket.html>

2 2