Mailman 3 October 2005 - Python-Dev

PEP 3000 and exec
by Jim Jewett Oct. 17, 2005

Oct. 17, 2005

For communicating with an exec/eval child, once exec cannot run in the current namespace, I asked that it be possible to pass a read-only "current view" and to see a return value. (Guido): >... it's probably better to create an empty namespace and > explicitly copy into it ... > ... just pick certain things out of the namespace [afterwards] Yes and no. If the exec'ed code is well defined (and it needs to be if security is a concern), then that works well. For more exploratory code, it can be hard to know what in advance what the code will need, or to agree on the names of return variables. The simplest general API that I can come up with is "You're allowed to see anything I can" (even if it is in a nested scope or base class, and I realize that you *probably* won't need it). "Return value is whatever you explicitly choose to return" (Lisp's "last result" might be even simpler, but would probably lead to confusion other places.) -jJ

1 0

AST branch update
by Jeremy Hylton Oct. 17, 2005

Oct. 17, 2005

Neil and I have been working on the AST branch for the last week. We're nearly ready to merge the changes to the head. I imagine we'll do it this weekend, barring last minute glitches. There are a few open issues that remain. I'd like to merge the branch before resolving them. Please let me know if you disagree. The current status of the AST branch is that only two tests fail: test_trace and test_symtable. The causes for these failures are described below. We did not merge the current head to the branch again, but I diffed the test suite between head and branch and did not see any substantive changes since the last merge. Some of the finer points of generating the line number table (lnotab) are wrong. There is some very delicate code to support single stepping with the debugger. We'll get that fixed soon, but we'd like to temporarily disable the failing tests in test_trace. The symtable module exposed parts of the internal representation of the old symbol table. The representation changed, and the module is going to need to change. The old module was poorly documented and tested, so I'd like to start over. Again, I'd like to disable a couple of failing tests until after the merge occurs. I don't think the current test suite covers all of the possible syntax errors that can be raised. I'd like to add a new test suite that covers all of the remaining cases, perhaps moving some existing tests into this module as well. I'd like to do that after the merge, which means there may be some loose ends where syntax errors aren't handled gracefully. For those of you familiar with the ast work, I'll summarize the recent changes: We added line numbers to expressions in the AST. There are many cases where a statement spans multiple lines. We couldn't generate a correct lnotab without knowing the lines that expressions occur on. We merged the peephole optimizer into the new compiler and restored PyNode_Compile() so that the parser module works again. The parser module will still expose the old parse trees (just what it's users want). We should probably provide a similar AST module, but I'm not sure if we'll get to that. We fixed some flawed logic in the symbol table for handling nested scopes. Luckily, the test cases for nested scopes are pretty thorough. They all pass now. Jeremy

12 24

PEP 3000 and exec
by Jim Jewett Oct. 17, 2005

Oct. 17, 2005

Guido van Rossum wrote: > Another idea might be to change the exec() spec so that you are > required to pass in a namespace (and you can't use locals() either!). > Then the whole point becomes moot. I think of exec as having two major uses: (1) A run-time compiler (2) A way to change the local namespace, based on run-time information (such as a config file). By turning exec into a function with its own namespace (and enforcing a readonly locals()), the second use is eliminated. Is this intentional for security/style/efficiency/predictability? If so, could exec/eval at least (1) Be treatable as nested functions, so that they can *read* the current namespace. (2) Grow a return value, so that they can more easily pass information back to at least a (tuple of) known variable name(s). -jJ

2 1

Re: [Python-Dev] Autoloading? (Making Queue.Queue easier to use)
by Michael Chermside Oct. 17, 2005

Oct. 17, 2005

John Camera writes: > It sounds like he feels Queue should just be part of threading but queues > can be used in other contexts besides threading. So having separate > modules is a good thing. Perhaps I am wrong here, but the Queue.Queue class is designed specifically for synchronization, and I have always been under the impression that it was probably NOT the best tool for normal queues that have nothing to do with threading. Why incur the overhead of synchronization locks when you don't intend to use them. I would advise against using Queue.Queue in any context besides threading. continued... > I guess from Gregs comments Im not sure if he wants to [...] I'm going to stop trying to channel Greg here, he can speak for himself. But I will be quite surprised if _anyone_ supports the idea of having an module modify the local namespace importing it when it is imported. and later... > Here are 2 possible suggestions for the import statements > > import Queue asneeded > delayedimport Queue # can't think of a better name at this time Woah! There is no need for new syntax here! If you want to import Queue only when needed use this (currently legal) syntax: if queueIsNeeded: import Queue If you want to add a module (call it "Queue") to the namespace, but delay executing some of the code for now, then just use "import Queue" and modify the module so that it doesn't do all its work at import time, but delays some of it until needed. That too is possible today: # start of module initialized = False def doSomething(): if not initialized: initialize() # ... Python today is incredibly dynamic and flexible... despite the usual tenor of conversations on python-dev, it is very rare to encounter a problem that cannot be solved (and readably so) using the existing tools and constructs. -- Michael Chermside

5 5

PEP 343 updated
by Nick Coghlan Oct. 17, 2005

Oct. 17, 2005

PEP 343 has been updated on python.org. Highlights of the changes: - changed the name of the PEP to be simply "The 'with' Statement" - added __with__() method - added section on standard terminology (that is, contexts/context managers) - changed generator context decorator name to "context" - Updated "Resolved Issues" section - updated decimal.Context() example - updated closing() example so it works for objects without close methods I also added a new Open Issues section with the questions: - should the decorator be called "context" or something else, such as the old "contextmanager"? (The PEP currently says "context") - should the decorator be a builtin? (The PEP currently says yes) - should the decorator be applied automatically to generators used to write __with__ methods? (The PEP currently says yes) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

4 4

Divorcing str and unicode (no more implicit conversions).
by Martin Blais Oct. 17, 2005

Oct. 17, 2005

Hi. Like a lot of people (or so I hear in the blogosphere...), I've been experiencing some friction in my code with unicode conversion problems. Even when being super extra careful with the types of str's or unicode objects that my variables can contain, there is always some case or oversight where something unexpected happens which results in a conversion which triggers a decode error. str.join() of a list of strs, where one unicode object appears unexpectedly, and voila! exception galore. Sometimes the problem shows up late because your test code doesn't always contain accented characters. I'm sure many of you experienced that or some variant at some point. I came to realize recently that this problem shares strong similarity with the problem of implicit type conversions in C++, or at least it feels the same: Stuff just happens implicitly, and it's hard to track down where and when it happens by just looking at the code. Part of the problem is that the unicode object acts a lot like a str, which is convenient, but... What if we could completely disable the implicit conversions between unicode and str? In other words, if you would ALWAYS be forced to call either .encode() or .decode() to convert between one and the other... wouldn't that help a lot deal with that issue? How hard would that be to implement? Would it break a lot of code? Would some people want that? (I know I would, at least for some of my code.) It seems to me that this would make the code more explicit and force the programmer to become more aware of those conversions. Any opinions welcome. cheers,

14 34

Re: [Python-Dev] Definining properties - a use case for classdecorators?
by Delaney, Timothy (Tim) Oct. 17, 2005

Oct. 17, 2005

Guido van Rossum wrote: > To which Tim Delaney responded, "have a look at my response here:" > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/408713 > > I looked at that, and now I believe it's actually *better* to mention > the property name twice, at least compared to Tim' s approach. I never said it was a *good* approach - just *an* approach ;) Tim Delaney

1 0

Re: [Python-Dev] Definining properties - a use case for class decorators?
by Delaney, Timothy (Tim) Oct. 17, 2005

Oct. 17, 2005

Greg Ewing wrote: > class C: > > foo = overridable_property('foo', "The foo property") > > def get_foo(self): > ... > > def set_foo(self, x): > ... > > This has the advantage that the accessor methods can be > overridden in subclasses with the expected effect. This is a point I was going to bring up. > The only wart is the necessity of mentioning the property > name twice, once on the lhs and once as an argument. > I haven't thought of a good solution to that, yet. Have a look at my comment to Steven Bethard's recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/408713 Tim Delaney

1 0

Early PEP draft (For Python 3000?)
by Eyal Lotem Oct. 16, 2005

Oct. 16, 2005

I would like to re-suggest a suggestion I have made in the past, but with a mild difference, and a narrower scope. Name: Attribute access for all namespaces Rationale: globals() access is conceptually the same as setting the module's attributes but uses a different idiom (access of the dict directly). Also, locals() returns a dict, which implies it can affect the local scope, but quietly ignores changes. Using attribute access to access the local/global namespaces just as that is used in instance namespaces and other modules' namespaces, could reduce the mental footprint of Python. Method: All namespace accesses are attribute accesses, and not direct __dict__ accesses. Thus globals() is replaced by a "module" keyword (or "magic variable"?) that evaluates to the module object. Thus, reading/writing globals in module X, uses getattr/setattr on the module object, just like doing so in module Y would be. locals() would return be replaced by a function that returns the frame object (or a weaker equivalent of a frame object) of the currently running function. This object will represent the local namespace and will allow attribute getting/setting to read/write attributes. Or it can disallow attribute setting. Examples: global x ; x = 1 Replaced by: module.x = 1 or: globals()[x] = 1 Replaced by: setattr(module, x, 1) locals()['x'] = 1 # Quietly fails! Replaced by: frame.x = 1 # Raises error x = locals()[varname] Replaced by: x = getattr(frame, varname) Advantages: - Python becomes more consistent w.r.t namespacing and scopes. Disadvantages: - "module" is already possible by importing one's own module, but that is: * Confusing and unnecessarily requires naming one's self redundantly (Making renaming of the module a bit more difficult). * Not easily possible in a __main__/importable module. * No equivalent for locals() - Automatic script conversion may be difficult in some use cases of globals()/locals()

4 5

Re: [Python-Dev] Unicode charmap decoders slow
by Tony Nelson Oct. 16, 2005

Oct. 16, 2005

I have written my fastcharmap decoder and encoder. It's not meant to be better than the patch and other changes to come in a future version of Python, but it does work now with the current codecs. Using Hye-Shik Chang's benchmark, decoding is about 4.3x faster than the base, and encoding is about 2x faster than the base (that's comparing the base and the fast versions on my machine). If fastcharmap would be useful, please tell me where I should make it available, and any changes that are needed. I would also need to write an installer (distutils I guess). <http://georgeanelson.com/fastcharmap.d.tar.gz> Fastcharmap is written in Python and Pyrex 0.9.3, and the .pyx file will need to be compiled before use. I used: pyrexc _fastcharmap.pyx gcc -c -fPIC -I/usr/include/python2.3/ _fastcharmap.c gcc -shared _fastcharmap.o -o _fastcharmap.so To use, hook each codec to be speed up: import fastcharmap help(fastcharmap) fastcharmap.hook('name_of_codec') u = unicode('some text', 'name_of_codec') s = u.encode('name_of_codec') No codecs were rewritten. It took me a while to learn enough to do this (Pyrex, more Python, some Python C API), and there were some surprises. Hooking in is grosser than I would have liked. I've only used it on Python 2.3 on FC3. Still, it should work going forward, and, if the dicts are replaced by something else, fastcharmap will know to leave everything alone. There's still a tiny bit of debugging print statements in it. >At 8:36 AM +0200 10/5/05, Martin v. Löwis wrote: >>Tony Nelson wrote: > ... >>> Encoding can be made fast using a simple hash table with external chaining. >>> There are max 256 codepoints to encode, and they will normally be well >>> distributed in their lower 8 bits. Hash on the low 8 bits (just mask), and >>> chain to an area with 256 entries. Modest storage, normally short chains, >>> therefore fast encoding. >> >>This is what is currently done: a hash map with 256 keys. You are >>complaining about the performance of that algorithm. The issue of >>external chaining is likely irrelevant: there likely are no collisions, >>even though Python uses open addressing. > >I think I'm complaining about the implementation, though on decode, not >encode. > >In any case, there are likely to be collisions in my scheme. Over the >next few days I will try to do it myself, but I will need to learn Pyrex, >some of the Python C API, and more about Python to do it. > > >>>>...I suggest instead just /caching/ the translation in C arrays stored >>>>with the codec object. The cache would be invalidated on any write to the >>>>codec's mapping dictionary, and rebuilt the next time anything was >>>>translated. This would maintain the present semantics, work with current >>>>codecs, and still provide the desired speed improvement. >> >>That is not implementable. You cannot catch writes to the dictionary. > >I should have been more clear. I am thinking about using a proxy object >in the codec's 'encoding_map' and 'decoding_map' slots, that will forward >all the dictionary stuff. The proxy will delete the cache on any call >which changes the dictionary contents. There are proxy classed and >dictproxy (don't know how its implemented yet) so it seems doable, at >least as far as I've gotten so far. > > >>> Note that this caching is done by new code added to the existing C >>> functions (which, if I have it right, are in unicodeobject.c). No >>> architectural changes are made; no existing codecs need to be changed; >>> everything will just work >> >>Please try to implement it. You will find that you cannot. I don't >>see how regenerating/editing the codecs could be avoided. > >Will do! ____________________________________________________________________ TonyN.:' <mailto:tonynelson@georgeanelson.com> ' <http://www.georgeanelson.com/>

4 8