Mailman 3 June 2004 - Python-Dev

RE: [Python-Dev] Proposal: thread.get_dict
by Chermside, Michael 29 Jun '04

29 Jun '04

I wrote: > > We're all consenting adults here. Tampering with the private > > data of an extension module is a well-known way to crash > > Python. Armin replies: > Is it? I am not aware of any such example in the standard > library. I am sure > that it would be considered as a bug. The only (internal and > expected) ways > to crash the Python interpreter from Python are currently to > abuse new.code() > or gc.getreferents(). If an extension module allows this > kind of tampering > from Python code I'd consider it poorly written. As I had always understood it, we go to great lengths to ensure that Python itself and the standard library are as robust as possible and nearly impossible to crash. However, extension authors, by definition, are writing *extensions* and are not under the supervision of the Python development team. It is, of course, trivial to write an extension which will (intentionally) crash Python, and just as easy to (unintentionally) write one with a flaw which allows crashes. Thus, I have always understood that as soon as you use an extension, Python no longer offers any guaranties. Of course, the author of the EXTENSION may offer such a guarantee. Depending on who it is, and how widely-used and well-tested the extension is, you may choose to trust such a guarantee. But I've always thought that was separate from Python's promise of "no crashes". As for handling this particular issue, I agree with your suggestion: > I'd advocate using fresh instances of > 'object' as keys instead of strings, which is even safer. Good advice for extension authors AND for pure-python writers whenever they don't want their values tramped on. Of course, one can still call keys() on the dictionary to obtain the key if one is *trying* to break (or monkeypatch) the extension, but that's where I think the "consenting adults" rule applies. -- Michael Chermside This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it.

1 0

Re: [Python-Dev] Allowing u.encode() to return non-strings
by Tim Peters 29 Jun '04

29 Jun '04

[Bill Janssen] > Tim, do I understand then that Unicode strings have an implicit > character encoding, but non-Unicode strings do not? An 8-bit string is a sequence of 8-bit bytes. If those bytes are to "mean something", you have to supply the meaning, or use them in a context that supplies a specific meaning for you. This seems nearly impossible for an American to understand, but non-Americans appear to know it at birth (if not earlier). A Unicode string is, at least in theory, a sequence of Unicode characters, the latter defined in excruciating detail by the Unicode Consortium. There's no conventional sense in which a Unicode string is an encoding of something other than exactly itself, but you could certainly make one up.

1 0

Re: [Python-Dev] Allowing u.encode() to return non-strings
by Tim Peters 29 Jun '04

29 Jun '04

[Bill Janssen] > While we're talking about this, Martin, what is the encoding of the > "string" returned by > > struct.pack("bbb", 0xFF, 0x00, 0x83) That raises an exception. Did you intend "BBB" as the format string? > And what should it be? Whatever you explicitly say it is, when you feed the string to a context that requires knowing the intended encoding. It has no intrinsic encoding -- encoding is in your head (or your app's requirements), not in the string itself. It's like asking which numeric base the sequence of digits 1111 is written in. You can't tell by staring it at. Maybe that was decimal, maybe it was binary. But no -- I actually had base 2004 in mind <wink>. Here are two distinct possibilities for your specific string: >>> s = struct.pack("BBB", 0xFF, 0x00, 0x83) >>> unicode(s, 'latin1') u'\xff\x00\x83' >>> unicode(s, 'cp1252') u'\xff\x00\u0192' >>> Python can't guess the intent. Neither could a highly knowledgeable but non-telepathic human, for that matter.

2 1

Re: [Python-Dev] Re: decorators and 2.4
by Jewett, Jim J 28 Jun '04

28 Jun '04

Jeff Bone > (1) Does a function have access to its own decorators? > I expect the answer here should be "yes." I don't expect it to have access to the decorators, or even to the results of running them -- at least, not any more than I expect a function to have access to its own name. Conceptually, def funcname(args): ... is (almost) just a nicer way to write funcname=lambda(args) ... In either case, a function object is created, and then "decorated" by binding a name to it. By analogy, [wrap1()] def funcname(args): ... means (almost) the same thing as def funcname(args): ... funcname=wrap1(funcname) Functions have no direct access to their own name, or their decorators, or even the affects of these decorators. Recursion works anyhow, because it looks the name up in the enclosing namespace. [The actual lookup may be short-circuited for efficiency.] Decorators may add attributes, or even replace a function object entirely with something else. (In Barry's example, that something else then delegates back to the original object.) This doesn't mean that the function itself needs direct access to those attributes (that name), let alone to the decorators that created them. > (2) Do decorators define literal constant values > at design-time or do they define a scope of mutable > bindings? Not either really. They do post-processing on a function (or class?) object before binding a name to it. This may or may not add attributes which may or may not be constants. It may or may not create a new object with new scopes. > That seems fine, but what about this: > x = "Guido van Rossum" > def mymethod(f) [attrs(versionadded="2.2", author=x)]: attrs is evaluated before the body of mymethod. Part of that evaluation is to properly bind its own author parameter. This resolves to a constant string, but I agree that using a dictionary could allow confusion -- in exactly the same way that mutable default arguments do today. > (3) Should it be possible to conditionally evaluate decorators > based on run-time state? At first I misunderstood. Conditional upon state when the function *definition* is evaluated, yes (though maybe not in the first version). But that is compile-time. Conditional upon the call-time state -- no. The decorators themselves may well have been garbage collected by then. Note that this does not forbid a decorator that *replaces* the original function with another callable object that happens to include conditional logic. The decorator itself would still be evaluated only once. -jJ

1 0

RE: [Python-Dev] Re: decorators and 2.4
by Chermside, Michael 28 Jun '04

28 Jun '04

> I'm really curious what these uses might be. Folks keep saying this > kind of thing, but when you dig in IMHO most of these > actually seem to > be design-time things, statements *about* the intent or use of the > function (relative to the language runtime itself, e.g. > classmethod and > so forth, or relative to a framework, such as registering a function, > or relative to some tool, e.g. for extended documentation purposes or > type-checking, etc.) rather than e.g. runtime modifications to its > behavior. Well, perhaps you and I mean slightly different things when we talk about "declarative" functionality. Here are some uses, you let me know whether they are "declarative" or not: 1. Registering with a framework: [FooFramework] def some_func(): body() FooFramework() "registers" the function with a framework by simply adding it to a list of known functions. This allows the GUI interface for FooFramework to display the function. 2. Take actions before or after a call: [traceable] def some_func(): body() traceable() replaces some_func() with a function which may write a log message before and after actually invoking the original some_func(). Whether the log message is actually written is controlled by a list of function names (compared to some_func.__name__) and that list may change dynamically while the program is executing. 3. Enforce pre- and post- conditions: [precondition(lambda x,y: x<y), postcondition(lambda x,y,result: x<result<y)] def some_func(x, y): body() precondition() and postcondition() document what we expect to be true before and after a function is invoked. Furthermore, if the constant Contract.PROD in the Contract module is False, then they replace some_func() with a wrapped version which tests the conditions and raises a ContractException if they fail. (Note: I don't think this is the best design for a design-by-contract library, just one approach which I think decorators SHOULD make possible.) 4. Storing Private Data: class Some_Class: [private] def some_method(self, private, x): body(self, private, x) This is kind of far out, but private() replaces the function some_method(self,private,x) with a wrapper function some_method(self,x) which takes just TWO arguments. Then the original some_method() is invoked with all three arguments, including "private" which is a dict that the wrapper conjures up from someplace well hidden. This means that the method some_method() can have access to the values in the "private" dict (and can write values into it also), but someone inspecting the __dict__ of the Some_Class instance won't be able to find (or change) "private". There are lots of other good ideas out there (many better than these), but I'm wondering whether you consider these to be "declarative". You follow with a few questions: > (1) Does a function have access to its own decorators? I wouldn't think so. They are NOT "metadata", that's just one possible use. The function no more has access to its decorators than it has access to its source code! If we just needed a place to stick metadata we'd use the function's __dict__. > (2) Do decorators define literal constant values at > design-time or do > they define a scope of mutable bindings? The decorators are simply functions that get executed. > x = "Guido van Rossum" > def mymethod(f) [attrs(versionadded="2.2", author=x)]: That's fine. author is set to Guido. > x = "Guido van Rossum" > def mymethod(f) [attrs(versionadded="2.2", author=x)]: > oldAuthor = someHandleToMyDecorators.atts.author > someHandleToMyDecorators.atts.author = "Eric Idle" That's meaningless unless I know what "someHandleToMyDecorators" means. > IMHO, the RHS of an "assignment" in a decorator should only be some > value that can be known at "compile" time I disagree. I think you and I just have different ideas about what we'd use decorators for. (Actually, I think that perhaps your set of intended uses is a subset of mine.) > (3) Should it be possible to conditionally evaluate decorators based > on run-time state? > > The following, IMHO, is truly scary and inhibits many of the intended > uses of decorators. This shouldn't be legal, ever: > > if foo = 3: > [someDecoratorThatModifiesBarsRuntimeBehavior] > else: > [someOtherDecorator] > def bar(...): > ... Hmm... I hadn't considered that one. I'll reserve judgment. I wouldn't mind if it worked, but I'd certainly try never to use it. I *WOULD* however make use of things like this: if foo = 3: standard_decoration = someDecoratorWithALongName else: standard_decoration = someOtherDecorator # ... [standard_decoration] def bar(...): ... You write "The following, IMHO, is truly scary and inhibits many of the intended uses of decorators". I'm not sure that intended uses it is inhibiting. The only one I can imagine is using decorators to create a type-aware compiler that generates type- specific optimized code. I don't think this IS a possible use for decorators. So... what use *would* be inhibited if decorators were used in this manner? -- Michael Chermside This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it.

2 1

RE: [Python-Dev] Re: decorators and 2.4
by Jewett, Jim J 28 Jun '04

28 Jun '04

> This works with the proof-of-concept code I posted Apparently I clipped too much. With your proof-of-concept, one of your suggestions was using names like classmethod_follows() My intent was to say: "Whatever method does become standard -- either in C, or through adding your code to stdlib -- I want some way for a reader to know quickly that a function is being decorated." Therefore I do not want "better names" like classmethod() def foo(): pass because then various decorators will have nothing in common, except for the invisible fact that they they share a common base class. decorators(classmethod()) def foo(): pass is OK, and [classmethod()] def foo(): pass might be, because the "[]\n" or "decorators(...)" warn the reader about what is happening. -jJ

1 0

Re: [Python-Dev] Re: decorators and 2.4
by Phillip J. Eby 28 Jun '04

28 Jun '04

At 12:28 PM 6/28/04 -0400, Jewett, Jim J wrote: >But you would also see things like > > dec1() # No good way to chain decorators. > def g(): > pass > g=dec2(g) This works with the proof-of-concept code I posted: [dec2(), dec1()] def g(): pass and so does this: dec2() dec1() def g(): pass So, I don't know what you mean about no good way to chain them. The chaining is implemented in the tracing hook. >Personally, I want decorators tied together with a word (such >as "decorate" or "using"), but I admit that list brackets + >line-break *might* be enough. You can always do this: def using(*args): pass <wink>

1 0

Re: decorators and 2.4
by Jewett, Jim J 28 Jun '04

28 Jun '04

Phillip Eby: > Is it the name you dislike? Or the idea of simply > invoking decorating functions directly? If decorate is a callable class rather than a function, it can be subclassed for common idioms. The question is whether or not this is a good idea. Any specific use will be more clear with a usage-specific name, but the construct as a whole may be muddied if nothing ties these names together. For instance, it classmethod were subclassed from decorate (but used current usage when it got an argument), you could write classmethod() def f(): pass But you would also see things like dec1() # No good way to chain decorators. def g(): pass g=dec2(g) Personally, I want decorators tied together with a word (such as "decorate" or "using"), but I admit that list brackets + line-break *might* be enough. decorators(classmethod()) def f(): pass decorators(dec1(), dec2()) def g(): pass vs [classmethod()] def f(): pass [dec1(), dec2()] def g(): pass -jJ

1 0

Wild Idea for the Year
by Raymond Hettinger 28 Jun '04

28 Jun '04

What is wrong with the following code? def join(*iterable): s = "" for e in iterable: s = s + e return s What if that code could run as fast as ''.join(iterable)? How much existing code would start to perform better? Would Python be easier to teach and write? Would the resulting code look more natural and obvious? Without expanding the size of the string structure, I think it is possible to implement something equivalent to the following proposal (which for clarity, does add two words to the structure). Roughly, the idea is: Add two PyObject pointers to the structure, component0 and component1, and initialize them to NULL. Alter string_concat(a, b) to return a new string object with: ob_sval = '\0' ob_size = len(a) + len(b) component0 = a // be sure to INCREF(a) component1 = b // be sure to INCREF(b) To the start of every string method except string_len() and string_concat(), add a macro that expands to: if (self->component0 != NULL) _autojoin(self); Write an _autojoin(self) method who's goal in life is to recursively fill-out the string from the components: * resize self to be big enough for an ob_sized string * treat component0 and component1 as a binary tree (with only the leaves holding real strings) and recursively (in-order) find the leaves and copy their string data into self->ob_sval. DECREF each component after traversing it. * set self->component0 to NULL to mark self as being fully joined. The recursion depth should not be worrisome. In the typical case, one side of the tree will be a leaf and will not recurse further. It gets more interesting if the user writes things like b=a+a or has a nested series like c=a+b; f=d+e; g=c+f. The recursion and reference counting trivializes the handling of those cases. An actual implementation striving to save the two additional structure fields could use the ob_sstate field to indicate a deferred string (my name for one that has not yet had ob_sval filled out). The ob_sval array itself would be given room to hold two object pointers for the components and access to those pointers would be cast to (PyObject *). If this works out, it would be an innovation. The join([a,b]) versus a+=b problem also exists in several other languages. AFAICT, Python would have been the first to solve it. As a nice side benefit, __builtin__.sum() would no longer need to reject the case where the inputs were strings. Raymond Hettinger

9 11

Re-raise in absence of an "active" exception
by "Martin v. Löwis" 28 Jun '04

28 Jun '04

http://python.org/sf/973103 points to two interesting bugs in Python: First, using a re-raise after the except-block has completed will still re-raise the last exception. Even though the language spec is ambiguous (what is the "last expression that was active in the current scope" (*)), I doubt this is intended. It then also shows that the error you "normally" get for a reraise in absence of an exception is TypeError: exceptions must be classes, instances, or strings (deprecated), not NoneType I think this is in violation of the language description, which says "If no exception is active in the current scope, an exception is raised indicating this error." "This" error probably being "no active exception", not "exception must not be NoneType". What do you think? Regards, Martin

9 14