Mailman 3 April 2000 - Python-Dev

RE: [Thread-SIG] Re: marking shared-ness
by Bill Tutt April 19, 2000

April 19, 2000

> From: Christian Tismer [mailto:tismer@tismer.com] > Are atomic decrements really so cheap, meaning "are they mapped > to the atomic dec opcode"? > Then this is all ok IMHO. > On x86en they are mapped to an "atomic assembly fragment" i.e. params to some registers and then stick in a "lock add" instruction or something. (please forgive me if I've botched the details). So in that respect they are cheap, its a hardware level feature. On the otherhand though, given the effect that these instructions have on the CPU (its caches, buses, and so forth) it is by for no means free. My recollection vaguely recalls someone saying that all the platforms NT has supported so far has had at the minimum an InterlockedInc/Dec. InterlockCompareExchange() is where I think not all of the Intel family (386) and some of the other platforms may not have had the appropriate instructions. InterlockCompareExchange() is useful for creating your own spinlocks. The GCC list might be a good place for enquiring about the feasability of InterlockedInc/Dec on various platforms. Bill

1 0

RE: [Thread-SIG] marking shared-ness (was: baby steps for free-th reading)
by Salz, Rich April 19, 2000

April 19, 2000

>Consider the case where a programmer forgets to note the sharedness. He >passes the object to another thread. At certain points: BAM! The >interpreter dumps core. No. Using the "owning thread" idea prevents coredumps and allows the interpreter to throw an exception. Perhaps my note wasn't clear enough? /r$

2 1

Encoding of code in XML
by David Ascher April 19, 2000

April 19, 2000

Lots of projects embed scripting & other code in XML, typically as CDATA elements. For example, XBL in Mozilla. As far as I know, no one ever bothers to define how one should _encode_ code in a CDATA segment, and it appears that at least in the Mozilla world the 'encoding' used is 'cut & paste', and it's the XBL author's responsibility to make sure that ]]> is nowhere in the JavaScript code. That seems suboptimal to me, and likely to lead to disasters down the line. The only clean solution I can think of is to define a standard encoding/decoding process for storing program code (which may very well contain occurences of ]]> in CDATA, which effectively hides that triplet from the parser. While I'm dreaming, it would be nice if all of the relevant language communities (JS, Python, Perl, etc.) could agree on what that encoding is. I'd love to hear of a recommendation on the topic by the XML folks, but I haven't been able to find any such document. Any thoughts? --david ascher

5 10

baby steps for free-threading
by Greg Stein April 19, 2000

April 19, 2000

A couple months ago, I exchanged a few emails with Guido about doing the free-threading work. In particular, for the 1.6 release. At that point (and now), I said that I wouldn't be starting on it until this summer, which means it would miss the 1.6 release. However, there are some items that could go into 1.6 *today* that would make it easier down the road to add free-threading to Python. I said that I'd post those in the hope that somebody might want to look at developing the necessary patches. It fell off my plate, so I'm getting back to that now... Python needs a number of basic things to support free threading. None of these should impact its performance or reliability. For the most part, they just provide a platform for the later addition. 1) Create a portable abstraction for using the platform's per-thread state mechanism. On Win32, this is TLS. On pthreads, this is pthread_key_*. This mechanism will be used to store PyThreadState structure pointers, rather than _PyThreadState_Current. The latter variable must go away. Rationale: two threads will be operating simultaneously. An inherent conflict arises if _PyThreadState_Current is used. The TLS-like mechanism is used by the threads to look up "their" state. There will be a ripple effect on PyThreadState_Swap(); dunno offhand what. It may become empty. 2) Python needs a lightweight, short-duration, internally-used critical section type. The current lock type is used at the Python level and internally. For internal operations, it is rather heavyweight, has unnecessary semantics, and is slower than a plain crit section. Specifically, I'm looking at Win32's CRITICAL_SECTION and pthread's mutex type. A spinlock mechanism would be coolness. Rationale: Python needs critical sections to protect data from being trashed by multiple, simultaneous access. These crit sections need to be as fast as possible since they'll execute at all key points where data is manipulated. 3) Python needs an atomic increment/decrement (internal) operation. Rationale: these are used in INCREF/DECREF to correctly increment or decrement the refcount in the face of multiple threads trying to do this. Win32: InterlockedIncrement/Decrement. pthreads would use the lightweight crit section above (on every INC/DEC!!). Some other platforms may have specific capabilities to keep this fast. Note that platforms (outside of their threading libraries) may have functions to do this. 4) Python's configuration system needs to be updated to include a --with-free-thread option since this will not be enabled by default. Related changes to acconfig.h would be needed. Compiling in the above pieces based on the flag would be nice (although Python could switch to the crit section in some cases where it uses the heavy lock today) Rationale: duh 5) An analysis of Python's globals needs to be performed. Any global that can safely be made "const" should. If a global is write-once (such as classobject.c::getattrstr), then these are marginally okay (there is a race condition, with an acceptable outcome, but a mem leak occurs). Personally, I would prefer a general mechanism in Python for creating "constants" which can be tracked by the runtime and freed. I would also like to see a generalized "object pool" mechanism be built and used for tuples, ints, floats, frames, etc. Rationale: any globals which are mutable must be made thread-safe. The fewer non-const globals to examine, the fewer to analyze for race conditions and thread-safety requirements. Note: making some globals "const" has a ripple effect through Python. This is sometimes known as "const poisoning". Guido has stated an acceptance to adding "const" throughout the interpreter, but would prefer a complete (rather than ripple-based, partial) overhaul. I think that is all for now. Achieving these five steps within the 1.6 timeframe means that the free-threading patches will be *much* smaller. It also creates much more visibility and testing for these sections. Post 1.6, a patch set to add critical sections to lists and dicts would be built. In addition, a new analysis would be done to examine the globals that are available along with possible race conditions in other mutable types and structures. Not all structures will be made thread-safe; for example, frame objects are used by a single thread at a time (I'm sure somebody could find a way to have multiple threads use or look at them, but that person can take a leap, too :-) Depending upon Guido's desire, the various schedules, and how well the development goes, Python 1.6.1 could incorporate the free-threading option in the base distribution. Cheers, -g -- Greg Stein, http://www.lyra.org/

2 2

switch to ANSI C (was: baby steps for free-threading)
by Greg Stein April 19, 2000

April 19, 2000

On Tue, 18 Apr 2000, Barry A. Warsaw wrote: > >>>>> "GvR" == Guido van Rossum <guido(a)python.org> writes: > > GvR> Actually, it's okay to do this on an "as-neeed" basis. I'm > GvR> also in favor of changing all the K&R code to ANSI, and > GvR> getting rid of Py_PROTO and friends. Cleaner code! > > I agree, and here's yet another plea for moving to 4-space indents in > the C code. For justification, look at the extended call syntax hacks > in ceval.c. They essentially /use/ 4si because they have no choice! > > Let's clean it up in one fell swoop! Obviously not for 1.6. I > volunteer to do all three mutations. Why not for 1.6? These changes are pretty brain-dead ("does it compile?") and can easily be reviewed. If somebody out there happens to have the time to work up ANSI C patches, then why refuse them? Cheers, -g -- Greg Stein, http://www.lyra.org/

1 0

Pasting interpreter prompts
by Ka-Ping Yee April 18, 2000

April 18, 2000

One gripe that i hear a lot is that it's really difficult to cut and paste chunks of Python code when you're working with the interpreter because the ">>> " and "... " prompts keep getting in the way. Does anyone else often have or hear of this problem? Here is a suggested solution: for interactive mode only, the console maintains a flag "dropdots", initially false. After line = raw_input(">>> "): if line[:4] in [">>> ", "... "]: dropdots = 1 line = line[4:] else: dropdots = 0 interpret(line) After line = raw_input("... "): if dropdots and line[:4] == "... ": line = line[4:] interpret(line) The above solution depends on the fact that ">>> " and "... " are always invalid at the beginning of a bit of Python. So, if sys.ps1 is not ">>> " or sys.ps2 is not "... ", all dropdots behaviour is disabled. I realize it's not going to handle all cases (in particular mixing pasted text with typed-in text), but at least it makes it *possible* to paste code, and it's quite a simple rule. I suppose it all depends on whether or not you guys often experience this particular little irritation. Any thoughts on this? -- ?!ng

5 9

RE: [Python-Dev] #pragmas in Python source code
by Da Silva, Mike April 18, 2000

April 18, 2000

Java uses ResourceBundles, which are identified by basename + 2 character locale id (eg "en", "fr" etc). The content of the resource bundle is essentially a dictionary of name value pairs. MS Visual C++ uses pragma code_page(windows_code_page_id) in resource files to indicate what code page was used to generate the subsequent text. In both cases, an application would rely on a fixed (7 bit ASCII) subset to give the well-known key to find the localized text for the current locale. Any "hardcoded" string literals would be mangled when attempting to display them using an alternate locale. So essentially, one could take the view that correct support for localization is a runtime issue affecting the user of an application, not the developer. Hence, myfile.py may contain 8 bit string literals encoded in my current windows encoding (1252) but my user may be using Japanese Windows in code page 932. All I can guarantee is that the first 128 characters (notwithstanding BACKSLASH) will be rendered correctly - other characters will be interpreted as half width Katakana or worse. Any literal strings one embeds in code should be purely for the benefit of the code, not for the end user, who should be seeing properly localized text, pulled back from a localized text resource file _NOT_ python code, and automatically pumped through the appropriate native <--> unicode translations as required by the code. So to sum up, 1 Hardcoded strings are evil in source code unless they use the invariant ASCII (and by extension UTF8) character set. 2 A proper localized resource loading mechanism is required to fetch genuine localized text from a static resource file (ie not myfile.py). 3 All transformations of 8 bit strings to and from unicode should explicitly specify the 8 bit encoding for the source/target of the conversion, as appropriate. 4 Assume that a Japanese / Chinese programmer will find it easier to code using the invariant ASCII subset than a Western European / American will be able to read hanzi in source code. Regards, Mike da Silva -----Original Message----- From: Ka-Ping Yee [mailto:ping@lfw.org] Sent: Wednesday, April 12, 2000 6:45 PM To: Fred L. Drake, Jr. Cc: Python Developers @ python.org Subject: Re: [Python-Dev] #pragmas in Python source code On Wed, 12 Apr 2000, Fred L. Drake, Jr. wrote: > > Or do we need to separate out two categories of pragmas -- > > pre-parse and post-parse pragmas? > > Eeeks! We don't need too many special forms! That's ugly! Eek indeed. I'm tempted to suggest we drop the multiple-encoding issue (i can hear the screams now). But you're right, i've never heard of another language that can handle configurable encodings right in the source code. Is it really necessary to tackle that here? Gak, what do Japanese programmers do? Has anyone seen any of that kind of source code? -- ?!ng _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org http://www.python.org/mailman/listinfo/python-dev

13 43

Escaping CDATA
by Ka-Ping Yee April 17, 2000

April 17, 2000

Here's what i'm playing with, if you want to mess with it too: import string def replace(text, old, new, join=string.join, split=string.split): return join(split(text, old), new) la, ra = "@@", "@@]" lb, rb = "]]>", "@@>" la, ra = "<]", "<]>" lb, rb = "]]>", "<][" def escape(text): cdata = replace(text, la, ra) cdata = replace(cdata, lb, rb) return cdata def unescape(cdata): text = replace(cdata, rb, lb) text = replace(text, ra, la) return text chars = "" for ch in la + ra + lb + rb: if ch not in chars: chars = chars + ch if __name__ == "__main__": class Tester: def __init__(self): self.failed = [] self.count = 0 def test(self, s, find=string.find): cdata = escape(s) text = unescape(cdata) print "%s -e-> %s -u-> %s" % (s, cdata, text) if find(cdata, "]]>") >= 0: print "EXPOSURE!" self.failed.append(s) elif s != text: print "MISMATCH!" self.failed.append(s) self.count = self.count + 1 tester = Tester() test = tester.test for a in chars: for b in chars: for c in chars: for d in chars: for e in chars: for f in chars: for g in chars: for h in chars: test(a+b+c+d+e+f+g+h) print if tester.failed == []: print "All tests succeeded." else: print "Failed %d of %d tests." % (len(tester.failed), tester.count) for t in tester.failed: tester.test(t)

1 0

Object customization (was: Arbitrary attributes on funcs and methods)
by Vladimir.Marangozov＠inrialpes.fr April 17, 2000

April 17, 2000

Obviously, the user-attr proposal is not so "simple" as it looks like. I wish we all realize what's really going on here. In all cited use cases for this proposal, functions are no more perceived as functions per se, but as data structures (objects) which are the target of the computation. IOW, functions are just considered as *instances* of a class (inheriting from the builtin "PyFunction" class) with user-attributes, having a state, and eventually a set of operations bound to them. I guess that everybody realized that with this proposal, one could bind not only doc strings, but also functions to the function. def func(): pass def serialize(): ... func.pack = serialize func.pack() What is this? This is manual instance customization. Since nobody said that this would be done 'exceptionally', but rather on a regular basis for all functions (and generally, for all objects) in a program, the way to customize instances after the fact, makes those instances singletons of user-defined classes. You may say "so what?". Well, this is fine if it were part of the object model from the start. And there's no reason why only functions and methods can have this functionality. Stick the __dict__ slot in the object header and let me bind user-attributes to all objects. I have a favorite number, 7, so I want to label it Vlad's number. seven = 7; seven.fanclub = ['Vlad']. I want to add a boolean func to all numbers, n.is_prime(). I want to have a s.zip() method for a set of strings in my particular application, not only the builtin ones. Why is it not allowed to have this today? Think about it! How would you solve your app needs today? Through classes and instances. That's the prescribed `legal' way to do customized objects; no shortcuts. Saying that mucking with functions' __doc__ strings is the only way to implement some functionality is simply not true. In short, there's no way I can accept this proposal in its current state and make the distingo between functions/methods and other kinds of objects (including 3rd party ones). If we're to go down this road, treat all objects as equal citizens in this regard. None or all. The object model must remain consistent. This proposal opens a breach in it. And not the lightest! And this is only part of the reasons why I'm still firmly -1 until P3K. Glad to see that Barry exposed some of the truth about it, after preserving our throats, i.e. he understood that we understood that he fully understood the power of namespaces, but eventually decided to propose a fraction of a significant change reserved for the next major Python release... wink <wink object at 80c5f30> >>> wink.fraction = 1e+-1 >>> wink.fraction.precision = 1e-+1 >>> wink.compute() 0.0 -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

17 80

trashcan and PR#7
by Vladimir.Marangozov＠inrialpes.fr April 17, 2000

April 17, 2000

While I'm at it, maybe the same recursion control logic could be used to remedy (most probably in PyObject_Compare) PR#7: "comparisons of recursive objects" reported by David Asher? -- Vladimir MARANGOZOV | Vladimir.Marangozov(a)inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

6 15